pip wheel -r ... does not pass config_settings

[belm0]

Description

pip wheel correctly passes config_settings when given a requirement specifier directly, but not when given a requirements file.

Works:

pip wheel --config-settings "setup-args=-Dblas=blas" scipy==1.10.1

Doesn't work:

echo 'scipy==1.10.1' > r.txt
pip wheel --config-settings "setup-args=-Dblas=blas" -r r.txt

Expected behavior

config_settings are passed when using pip wheel with a requirements file

pip version

23.2.1 (also head and older versions)

Python version

3.8

OS

Linux

How to Reproduce

1. in an environment with gfortran intentionally missing, try to build scipy with pip wheel -r ... and setting setup-args
2. in the error output (due to missing gfortran), confirm whether setup-args propagated to the meson command line

-r case (broken)

pip install --upgrade pip wheel
echo 'scipy==1.10.1' > r.txt
pip wheel --config-settings "setup-args=-Dblas=blas" -r r.txt

direct specifier case

pip install --upgrade pip wheel
pip wheel --config-settings "setup-args=-Dblas=blas" scipy==1.10.1

Output

-r case (broken)

-Dblas is not passed to meson build

+ meson setup --prefix=/usr /tmp/pip-wheel-6j8l4ith/scipy_206aaca2ef1b4369946502a91692a4b9 /tmp/pip-wheel-6j8l4ith/scipy_206aaca2ef1b4369946502a91692a4b9/.mesonpy-oc75antm/build --native-file=/tmp/pip-wheel-6j8l4ith/scipy_206aaca2ef1b4369946502a91692a4b9/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2

direct specifier case

+ meson setup --prefix=/usr /tmp/pip-wheel-jf4yjx_6/scipy_5f743564b93f4f4eaadba75a455c21b7 /tmp/pip-wheel-jf4yjx_6/scipy_5f743564b93f4f4eaadba75a455c21b7/.mesonpy-l963lj1w/build --native-file=/tmp/pip-wheel-jf4yjx_6/scipy_5f743564b93f4f4eaadba75a455c21b7/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 -Dblas=blas

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[qci-amos]

I just discovered this problem too. In my case, I'm using --config-settings editable_mode=compat in order for pylance to find my editable installations.

[qci-amos]

And it's not trivial to just convert my reqs into command line args. For example, I'd have to work out a new mechanism for environment variable interpolation and/or deal with changes in req sort order.

[belm0]

This seems to be a regression, because it was working for me last year.

I wonder if the culprit is pyproject-hooks and/or the pyproject.toml transition.

[belm0]

I've confirmed the discrepancy of -r vs. direct specifier this far:

• in both cases, ConfiguredBuildBackendHookCaller.prepare_metadata_for_build_wheel() is called with config_settings=None
• prepare_metadata_for_build_wheel() ignores config_settings anyway, and defers to self.config_holder.config_settings
• for -r case, self.config_holder.config_settings is None
• for direct specifier, self.config_holder.config_settings reflects the command line config_settings (e.g. {'setup-args': '-Dblas=blas'})

aside: it's odd that ConfiguredBuildBackendHookCaller.prepare_metadata_for_build_wheel() ignores the config_settings arg

pip/src/pip/_internal/utils/misc.py

Lines 757 to 768 in 496b268

	def prepare_metadata_for_build_wheel(
	self,
	metadata_directory: str,
	config_settings: Optional[Dict[str, Union[str, List[str]]]] = None,
	_allow_fallback: bool = True,
	) -> str:
	cs = self.config_holder.config_settings
	return super().prepare_metadata_for_build_wheel(
	metadata_directory=metadata_directory,
	config_settings=cs,
	_allow_fallback=_allow_fallback,
	)

... tracing higher up, in bad case install_req_from_line() is called with config_settings=None

[belm0]

here are backtraces for bad (passing config_settings=None) and good cases:

bad (-r)

Traceback (most recent call last):
  File "/.../pip/src/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
  File "/.../pip/src/pip/_internal/cli/req_command.py", line 245, in wrapper
    return func(self, options, args)
  File "/.../pip/src/pip/_internal/commands/wheel.py", line 120, in run
    reqs = self.get_requirements(args, options, finder, session)
  File "/.../pip/src/pip/_internal/cli/req_command.py", line 436, in get_requirements
    req_to_add = install_req_from_parsed_requirement(
  File "/.../pip/src/pip/_internal/req/constructors.py", line 500, in install_req_from_parsed_requirement
    req = install_req_from_line(
  File "/.../pip/src/pip/_internal/req/constructors.py", line 422, in install_req_from_line
    assert False, f'{config_settings=}'
AssertionError: config_settings=None

good

Traceback (most recent call last):
  File "/.../pip/src/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
  File "/.../pip/src/pip/_internal/cli/req_command.py", line 245, in wrapper
    return func(self, options, args)
  File "/.../pip/src/pip/_internal/commands/wheel.py", line 120, in run
    reqs = self.get_requirements(args, options, finder, session)
  File "/.../pip/src/pip/_internal/cli/req_command.py", line 411, in get_requirements
    req_to_add = install_req_from_line(
  File "/.../pip/src/pip/_internal/req/constructors.py", line 422, in install_req_from_line
    assert False, f'{config_settings=}'
AssertionError: config_settings={'setup-args': '-Dblas=blas'}

[belm0]

in CLI get_requirements():

install_req_from_parsed_requirement() case:

config_settings=parsed_req.options.get("config_settings") if parsed_req.options else None

install_req_from_line() case:

config_settings=getattr(options, "config_settings", None)

So the code is clearly only expecting config_settings from the requirements entries, and ignoring the command line.

I suppose the settings should be merged, with command line taking precedence...

pip/src/pip/_internal/cli/req_command.py

Lines 431 to 445 in 496b268

	# NOTE: options.require_hashes may be set if --require-hashes is True
	for filename in options.requirements:
	for parsed_req in parse_requirements(
	filename, finder=finder, options=options, session=session
	):
	req_to_add = install_req_from_parsed_requirement(
	parsed_req,
	isolated=options.isolated_mode,
	use_pep517=options.use_pep517,
	user_supplied=True,
	config_settings=parsed_req.options.get("config_settings")
	if parsed_req.options
	else None,
	)
	requirements.append(req_to_add)

[belm0]

I suppose the settings should be merged, with command line taking precedence...

Apparently, when #11634 added per-requirement config_settings, the PR initially tried to merge with the command line settings, but that was removed before the final version.

It appears intentional: #11915 (comment) and reinforced: #11941

My use case for setting config_settings on the command line, and expecting it to propagate to dependency builds, is valid:

• using pip wheel -r ... to build all dependencies of an application to target a less-popular Python implementation (Pyston)
• need to control the blas library selected by scipy via config_settings

I'll investigate per-requirement config_settings for my use case. But at the least, it may be good for pip to issue a warning that the command-line config settings are being ignored. @sbidoul @pfmoore

Configuration settings provided via --config-settings command line options (or the
equivalent environment variables or configuration file entries) are passed to the build
of requirements explicitly provided as pip command line arguments. They are not passed
to the build of dependencies, or to the build of requirements provided in requirement
files.

[belm0]

I'll investigate per-requirement config_settings for my use case.

It's not practical, because pip-compile doesn't appear to propagate settings from .in to .txt.

$ echo 'scipy==1.10.1 --config-settings="setup-args=-Dblas=blas"' > r2.in
$ pip-compile -r r2.in
numpy==1.24.4
    # via scipy
scipy==1.10.1
    # via -r r2.in

filed jazzband/pip-tools#2000

[uranusjr]

I feel that config settings should not be merged (perferrably with a warning), but if config settings are only provided from one source that should be respected instead of being “cancelled”.

[sbidoul]

So as noted in #12310 (comment) the behaviour is intentional and was changed to resolve an inconsistency with passing config settings to the build of dependencies.

That does not mean it cannot be changed again. I think this would require a detailed specification of the desired behaviour, though. For instance I'm not sure what build backends should do with config settings they don't understand - is that defined in a standard? How should per-requirements config settings behave when there are "global" config settings? Should per-requirements config settings propagate to their dependencies or not?

[qci-amos]

I'll note that my use case is, as far as I can tell, due to some kind of disagreement between pylance and pip and I feel kinda "caught in the middle". Ultimately, I believe I shouldn't need to pass any cli arg for vscode to be able to find my editable installations, but I'm willing do so if pip/pylance say that's what I need to do.

[pfmoore]

@qci-amos Yes, it's a disagreement between two tools, if you want to look at it that way. The tools in question are setuptools and pylance. The problem is that setuptools implements editable installs by default in a way that pylance can't detect. The compat mode uses a different implementation that is recognised by pylance.

I'd suggest bringing this up with setuptools. They may have a way of setting a global (or per-project) configuration which chooses a different default implementation. Or they may have another way of working around this - I'd be surprised if you are the only person affected by this.

I know setuptools and the typing community have had discussions about this issue - the technical problem behind it is currently unresolved (it needs people to work on, and agree, a standard way of publishing the data that type checkers need from an editable install) as far as I know, but I don't know if it's something they are actively working on, or if there are acceptable workarounds for now (if there are, they would presumably help you as well).

[qci-amos]

Ok, thank you. Digging around in here I found many potentially related issues so I just picked one to cross-link.

I'd be surprised if you are the only person affected by this.

Yea, I agree! I and my users are not doing anything complicated as far as the build system itself is concerned. All I'm trying to do is install multiple repos in editable mode and not get orange squiggles all over my vscode tabs! My guess is that many devs out there assume it's their bug somehow but give up trying to figure it out and decide instead to just live with the squiggles.

[pfmoore]

Digging around in here I found many potentially related issues so I just picked one to cross-link.

The specific one you want is pypa/setuptools#3518

[qci-amos]

Thank you! One suggestion made there that might be relevant here is to define an environment variable as an alternative to a cli arg. That might be easier to get working with -r and I could see this being useful beyond only the PEP-660 use case.

[pfmoore]

All of pip's command line options can be specified via environment variables - see the documentation

[qci-amos]

Ah, ok, it looks like I can get this to work:

export PIP_CONFIG_SETTINGS="editable_mode=compat"
pip install -e .

and this is a win for me because usually I don't install from a requirements file (and the cli arg is difficult for me in my day-to-day ad hoc work). However, this doesn't seem to see the env var:

export PIP_CONFIG_SETTINGS="editable_mode=compat"
echo "-e ." > reqs.txt
pip install -r reqs.txt

[sbidoul]

However, this doesn't seem to see the env var

Yes, that is because pip processes PIP_CONFIG_SETTINGS identically as the --config-settings CLI option.
And these are applied to requirements passed as arguments only and do not currently propagate to requirements passed inside -r requirement files.

Note that with #12494, you should be able to add --config-settings to -e lines in requirement files. Would that help in your use case?

[qci-amos]

Yes, @sbidoul I think that would work for me. Thank you!

[qci-amos]

For what it's worth, the env var solution I'd been using before, described above #12310 (comment), seems to have stopped working with pip 24, but I haven't had time to debug this.