Isolated builds when both pyproject.toml and setup.py co-exist

[adithyabsk]

Environment

• pip version: 20.1.1
• Python version: 3.7.7
• OS: macOS 10.14.6 (18G103)

Description
The maintainers of flake8 are suggesting that the adoption of pyproject.toml will not move forward until pip's behavior is amended. This seems to be due to the backend build system changing in the presence of a pyproject.toml file. Popular projects such as black have centralized their configuration using pyproject.toml and hence there may be many systems with both setup.py and a pyproject.toml configuration files during the transition period to pyproject.toml for build specifications.

I wanted to open a dialogue here to try and understand better understand pip's behavior in this regard and the maintainer's claim of "forced isolated builds with no recourse".

How to Reproduce
I was able to reproduce their issue with pip using the latest version.

$ pip install six
$ pip --version
pip 20.1.1 from /Users/adithyabalaji/.pyenv/versions/3.7.7/envs/pip_test/lib/python3.7/site-packages/pip (python 3.7)
$ pip freeze
six==1.15.0
$ cat setup.py
import setuptools
import six

setuptools.setup(
    name="pip_test", # Replace with your own username
    version="0.0.1",
    author="Example Author",
    author_email="author@example.com",
    description="A small example package",
    long_description="long_description",
    long_description_content_type="text/markdown",
    url="https://github.com/pypa/sampleproject",
    packages=setuptools.find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    python_requires='>=3.6',
)
$ pip install .
Processing /Users/adithyabalaji/Coding/pip_test
Building wheels for collected packages: pip-test
  Building wheel for pip-test (setup.py) ... done
  Created wheel for pip-test: filename=pip_test-0.0.1-py3-none-any.whl size=1563 sha256=71b54964c94d7ebbc065f2a3afb2cb05ba23ae23974a58e1eefbeb91c8c8e493
  Stored in directory: /private/var/folders/ky/xpvlh4052fb99mp1n12s4pzc0000gn/T/pip-ephem-wheel-cache-uv97sy5f/wheels/5c/62/db/42908c304e6d9828375edcd5c4347c2d13df469ca005a1e575
Successfully built pip-test
Installing collected packages: pip-test
Successfully installed pip-test-0.0.1
$ touch pyproject.toml
$ pip install .
Processing /Users/adithyabalaji/Coding/pip_test
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/adithyabalaji/.pyenv/versions/3.7.7/envs/pip_test/bin/python /Users/adithyabalaji/.pyenv/versions/3.7.7/envs/pip_test/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/ky/xpvlh4052fb99mp1n12s4pzc0000gn/T/tmpw33op9k3
       cwd: /private/var/folders/ky/xpvlh4052fb99mp1n12s4pzc0000gn/T/pip-req-build-scma8bnc
  Complete output (18 lines):
  Traceback (most recent call last):
    File "/Users/adithyabalaji/.pyenv/versions/3.7.7/envs/pip_test/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/Users/adithyabalaji/.pyenv/versions/3.7.7/envs/pip_test/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/adithyabalaji/.pyenv/versions/3.7.7/envs/pip_test/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 114, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/ky/xpvlh4052fb99mp1n12s4pzc0000gn/T/pip-build-env-zv257o29/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 148, in get_requires_for_build_wheel
      config_settings, requirements=['wheel'])
    File "/private/var/folders/ky/xpvlh4052fb99mp1n12s4pzc0000gn/T/pip-build-env-zv257o29/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 128, in _get_build_requires
      self.run_setup()
    File "/private/var/folders/ky/xpvlh4052fb99mp1n12s4pzc0000gn/T/pip-build-env-zv257o29/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 250, in run_setup
      self).run_setup(setup_script=setup_script)
    File "/private/var/folders/ky/xpvlh4052fb99mp1n12s4pzc0000gn/T/pip-build-env-zv257o29/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 143, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 2, in <module>
      import six
  ModuleNotFoundError: No module named 'six'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/adithyabalaji/.pyenv/versions/3.7.7/envs/pip_test/bin/python /Users/adithyabalaji/.pyenv/versions/3.7.7/envs/pip_test/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/ky/xpvlh4052fb99mp1n12s4pzc0000gn/T/tmpw33op9k3 Check the logs for full command output.
$ pip install . --no-use-pep517
Processing /Users/adithyabalaji/Coding/pip_test
Building wheels for collected packages: pip-test
  Building wheel for pip-test (setup.py) ... done
  Created wheel for pip-test: filename=pip_test-0.0.1-py3-none-any.whl size=1563 sha256=91a191eac2dcb7cc85c10c27d1cb603bc1ba98c1091cc0f177dfd644e18b87dc
  Stored in directory: /private/var/folders/ky/xpvlh4052fb99mp1n12s4pzc0000gn/T/pip-ephem-wheel-cache-o2hrr93v/wheels/5c/62/db/42908c304e6d9828375edcd5c4347c2d13df469ca005a1e575
Successfully built pip-test
Installing collected packages: pip-test
  Attempting uninstall: pip-test
    Found existing installation: pip-test 0.0.1
    Uninstalling pip-test-0.0.1:
      Successfully uninstalled pip-test-0.0.1
Successfully installed pip-test-0.0.1

[adithyabsk]

Tagging the aforementioned maintainer of flake8 here in case they have anything to add to the discussion here.

@asottile

[McSinyx]

Hi, IIUC isolated build will get triggered when pyproject.toml exists, and I don't understand how the linked thread is relevant (which to my understanding, it's about flake8 supporting configuration from the TOML file, not on the build of the project). With build isolation, aside from the fallback setuptools and wheel, you'll need to add six to the build requirements in this case.

[adithyabsk]

@McSinyx I do not want to speak on behalf of the flake8 maintainers but my understanding of their concern is that if pyproject.toml is supported, some projects could become broken due to the pip installation backend changing. As mentioned in #2381, I am not aware of a way to specify build-time requirements in setup.py without switching to pyproject.toml as the build system.

My (unfounded, data here would be good) claim would be that users who have setup.py scripts that depend on non-isolated builds (ie their setup.py script has build-time dependencies) are few and far between and could easily be pointed to an FAQ page as to how to address that issue (migrating to pyproject.toml). But, I can see the burden that might pose to developers since flake8 is such a widely used package.

[McSinyx]

I think there's some misunderstanding here (either from my side or your side):

• The flake8's ticket is about delay support flake8 configuration (which usually resides in tox.ini, setup.cfg, etc.), not delay the usage of isolation of the build process of flake8 itself.
• I think (I could be wrong though) that the final goal of Python packaging is to have build-isolation for every package (i.e. PEP 517/518). The isolation can be used by pip regardless pyproject.toml exists or not. I don't understand how flake8 is relevant here since it's for linting, not building and thus is not relevant to the PEPs anyhow. It is entirely possible to have build isolation and have flake8 running, e.g. pip itself.

It is possible that you're confused about the meaning of build-isolation. If one's

setup.py script has build-time dependencies

perse should have the dependency specified in pyproject.toml. The mentioned PEPs are to solve such situation, so that a front-end (like pip) may be aware of the back-end needed (setuptools, wheel and six in your case) and install them in a virtual environment during build, regardless of the outer environment. To my understanding

setup.py scripts that depend on non-isolatedable builds

are obsolete stuff build via e.g. GNU make and should be converted to or wrapped by Python tooling somehow.

[adithyabsk]

there's some misunderstanding here

It's likely on my end. I've read the above-linked issue thread in the flake8 repo multiple times and I'm still not fully following. Maybe you can help me figure out what I am missing.

The flake8's ticket is about delay support flake8 configuration [...] not delay the usage of isolation of the build process of flake8 itself.

I think we both agree here. It's a long thread over there, but my understanding of the argument against supporting pyproject.toml in flake8 is due to the default behavior of pip when a pyproject.toml exists in a repository.

I don't understand how flake8 is relevant here since it's for linting, not building and thus is not relevant to the PEPs anyhow.

flake8 is not specifically relevant here. The reason I linked their thread was to provide context for my question. The key reason cited against supporting pyproject.toml as a flake8 configuration option was due to the backend switching behavior of pip:

Anthony Sottile · 6 months ago
If you want this, I'd suggest lobbying for the following:

• pip to change its behaviour so mere presence of the file does not change functionality
  • this probably involves a PEP, or at least a discussion on the various pypa mailing lists

Based on this:

the final goal of Python packaging is to have build-isolation for every package... [scripts that depend on non-isolatable builds] are obsolete stuff build via e.g. GNU make

I guess my question would then be, is the old pip build backend on deprecation track? If it is, there would be a stronger argument to suggest the "forced isolated builds" are not a concern since that would be the direction of the Python packaging ecosystem, and solutions are provided in the form of PEP 517/518.

[pfmoore]

I agree with @McSinyx here. The example setup.py provided is incorrect, in the sense that it will not work in an environment where six isn't present, but there is no indication of this fact in the project metadata.

To demonstrate this, create a new virtualenv without six installed, and with just the given setup.py (no pyproject.toml). pip install . will fail there, too. This is not about pyproject.toml, instead it's about projects having build requirements that are not available in machine readable form.

The fix for a project like this is to amend the documentation that says "you must manually install six before installing this project" to also say "... and you need to provide the --no-build-isolation flag to pip". Or better, just add six to pyproject.toml and then you need neither bit of documentation 🙂

[uranusjr]

Title and top post edited to fix typos.

I also removed #6163 from the title since it describes the use case of a setup.py importing a module from the to-be-installed-distribution, which is a supported (although discouraged) usage, while the setup.py in this case is referencing a module that is not a part of the to-be-installed distribution, which has a well-defined way to declare (listing it in pyproject.toml).

[pganssle]

My (unfounded, data here would be good) claim would be that users who have setup.py scripts that depend on non-isolated builds (ie their setup.py script has build-time dependencies) are few and far between and could easily be pointed to an FAQ page as to how to address that issue (migrating to pyproject.toml). But, I can see the burden that might pose to developers since flake8 is such a widely used package.

To be 100% clear here, because I don't think anyone has clarified it — pyproject.toml is not a replacement for setup.py, you don't have to do any migration. If you have a setup.py you should also have a pyproject.toml that describes that your build backend in setuptools. In almost all cases, even if the build-system table is unspecified in pyproject.toml, pip will still work just fine.

[pfmoore]

Also, this is not about pyproject.toml at all, but about build isolation. It's easy to confuse the two, because (for better or worse) pip uses the presence of pyproject.toml as a signal that projects had opted into newer standards, and were therefore prepared for build isolation. The unexpected speed with which tools like black and flake8 switched to storing configuration in pyproject.toml made that assumption seem a bit optimistic in hindsight.

Better documentation and blogs/articles/tutorials explaining how all of this hangs together would be very welcome. I don't know if anyone with the right combination of familiarity with the subject, good writing skills, and access to a suitable audience, would be able to help here.

[adithyabsk]

Hi @McSinyx @pfmoore @uranusjr and @pganssle thank you for taking the time to follow up on my question. I've been going back and forth with a flake8 developer via email (@asottile) based on your feedback. I asked if it was okay for me to share his concerns with this thread and he gave me the go-ahead.

Here is an (abbreviated) summary of the exchange:

---

On Jun 13, 2020, at 9:52 PM, @adithyabsk wrote:
[...]
I know it was mentioned in the thread that the presence of pyproject.toml forces isolated builds with no opt-out but my understanding is that you can use the “—no-use-pep517” to opt-out of the new pip build backend. I know that toml language support still isn’t in the core python lang, another concern that was raised. But, I was hoping this new information might persuade you to re-open the thread for discussion.

---

On Jun 22, 2020, at 12:08 AM, @asottile wrote:
yes, the problem is that the presence of the file fundamentally changes how pip installs things. --no-use-pep517 is a cop-out solution at best, pip install X should just work. additionally the file ends up bundled in distributions (since setuptools includes it by default) making --no-use-pep517 even more of a non-solution (since you cannot predict which of your dependencies use / don't use isolated builds).
because it causes pip to have strange behaviours, I can't in-earnest support pyproject.toml in flake8 and deprecate all of the other configuration approaches. and adding yet another way to configure flake8 when the configuration story is pretty terrible is not going to happen.

---

On Sun, Jun 21, 2020 at 9:17 PM @adithyabsk wrote:
Okay, that’s totally fair.

I can't in-earnest support pyproject.toml in flake8 and deprecate all of the other configuration approaches

I cannot speak to the other implementation but the PR that I had opened, was only around 100 lines of code to support pyproject.toml in addition to the other configuration approaches. Regardless, I understand your frustration with additional code to support yet another “standard” for configuration which increases the burden on the flake8 developers.

because it causes pip to have strange behaviours

With that said, would you willing to post your concerns to the thread that I started in pypa/pip - or - would you be okay if I shared your concerns with that thread? Since, I do not maintain nearly as many projects you do, I don’t have personal experience with strange pip behaviors that you mention.

---

On Jun 22, 2020, at 12:21 AM, @asottile wrote:
sure feel free to share

my thoughts on the sensible things pip should do is either:

• don't treat empty pyproject as isolated
• default isolated build on regardless of pyproject

[pfmoore]

I'm not sure where this discussion is going. We've covered pip's existing behaviour. Is there a specific request for a change in pip that we can consider here? The current behaviour is a compromise (between encouraging projects to specify build dependencies in pyproject.toml and not breaking projects who aren't ready to change how they are configured). The two proposals above:

• don't treat empty pyproject as isolated
• default isolated build on regardless of pyproject

are basically the two extremes that pip is compromising between. So which are you proposing we move to?

Note that as far as I can see, there's been no explanation given of why a project that changes to add flake8 configuration to pyproject.toml can't, at the same time add their build dependencies and get the whole transition done at once. Or conversely why, if they don't want to use pyproject.toml to specify their build dependencies yet, they can't just hold off putting flake8 config in there? "We didn't know that making this one change would also affect this other area" is a fair point, but it'll get caught in testing, and it's easy to fix (revert or add the dependencies, and you're done).

I'm genuinely baffled as to why people feel pip has to change, rather than projects make a simple build config update - so please, if there is a good reason, can someone explain it?

[bluemellophone]

Thank you all in this thread for pushing this discussion and development of pip forward. In regards to Black using project.toml files for configuration, it is a known issue that the original maintainer has refused to fix. For anybody else who has run into this issue with building wheels and setup.py throwing ModuleNotFound errors due to a Black configuration file, I'd suggest pivoting away from Black and using this instead: https://github.com/odwyersoftware/brunette. Luckily, Flake8 also allows setup.cfg files, so a great alternative for our project is to pivot to configuring Flake8 and Brunette in a single file and deprecating the toml file.

This fix works with pip 20.1.1

[pganssle]

Probably a better fix would be to just explicitly add a [build-system] table to your pyproject.toml file, and report any issues you have with it?

Eventually the behavior that you are switching away from black to avoid will become the default, and you'll have no recourse but to pin to an older pip. It would be better to opt in today and get your problems fixed before it causes problems.

[bluemellophone]

We will migrate to pyproject.toml once the standard is better documented, discussions and bugs around policies like this one have settled, and setup.py is officially deprecated. Our workflows rely on editable installs of local packages for development and, as far as I know, the new pyproject.toml + setup.cfg doesn't allow easily. We may eventually go back to Black and its configuration via toml but switching to Brunette for the interim has more than one upside for us.

Thanks for following up about there being multiple paths, though, but we may peg pip depending on how stable we want our immediate CI infrastructure to behave under sporadic development under COVID.

[pganssle]

We can close this issue now, right? There's nothing actionable, and it mainly seems like it's based on misunderstandings, which I'm not sure we can clarify better than we already have - both here and in many, many other venues.

[bluemellophone]

I would like to quote @pfmoore above:

"We didn't know that making this one change would also affect this other area" is a fair point, but it'll get caught in testing, and it's easy to fix (revert or add the dependencies, and you're done).

This will get caught in testing, but it breaks existing behavior and leaves the developer with not much in terms of hints on where to turn. For example, we use scikit-build to marshal our built C libraries from CMake. This is the error log when we do pip install -e .

Obtaining file:///Users/<username>/code/<package>
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/<username>/virtualenv/<env3.6>/bin/python3.6 /Users/<username>/virtualenv/<env3.6>/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/h2/wjcy5cpx3hd2zb3jss36lfnr0000gn/T/tmp27lw61jr
       cwd: /Users/<username>/code/<package>
  Complete output (18 lines):
  Traceback (most recent call last):
    File "/Users/<username>/virtualenv/<env3.6>/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/Users/<username>/virtualenv/<env3.6>/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/<username>/virtualenv/<env3.6>/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 114, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/h2/wjcy5cpx3hd2zb3jss36lfnr0000gn/T/pip-build-env-ghtpvvp6/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 147, in get_requires_for_build_wheel
      config_settings, requirements=['wheel'])
    File "/private/var/folders/h2/wjcy5cpx3hd2zb3jss36lfnr0000gn/T/pip-build-env-ghtpvvp6/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 127, in _get_build_requires
      self.run_setup()
    File "/private/var/folders/h2/wjcy5cpx3hd2zb3jss36lfnr0000gn/T/pip-build-env-ghtpvvp6/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 249, in run_setup
      self).run_setup(setup_script=setup_script)
    File "/private/var/folders/h2/wjcy5cpx3hd2zb3jss36lfnr0000gn/T/pip-build-env-ghtpvvp6/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 142, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 252, in <module>
      import skbuild
  ModuleNotFoundError: No module named 'skbuild'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/<username>/virtualenv/<env3.6>/bin/python3.6 /Users/<username>/virtualenv/<env3.6>/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/h2/wjcy5cpx3hd2zb3jss36lfnr0000gn/T/tmp27lw61jr Check the logs for full command output.

If I delete the pyproject.toml file as recommended by Black for configuration (https://github.com/psf/black/blob/master/pyproject.toml), this magically works. There isn't much to indicate that isolated builds or PEP 517/518 are related. At the very least, if pip determines that a build error is related to this change, a simple message would help. I'm not the only one who is confused or out of date: https://news.ycombinator.com/item?id=22746762 or https://www.reddit.com/r/learnpython/comments/biq99o/what_is_the_current_best_practice_for_specifying/.

I don't mind the change per se, I mind that the change is causing breaking behavior but that the developer is not provided a clear error message to prompt a migration.

Edit: Since it's not listed explicitly, here would be a correct pyproject.toml file that would be consistent with black, flake8, etc.

[tool.black]
line-length = 88
target-version = ['py36', 'py37', 'py38']
'''

[build-system]
requires = ["setuptools>=41.0", "setuptools-scm", "wheel"]
build-backend = "setuptools.build_meta"

[pfmoore]

At the very least, if pip determines that a build error is related to this change, a simple message would help.

As I already noted, better docs or other information is welcome. In particular, if anyone wants to submit a PR providing a better message covering this situation, it would be very much appreciated. However, I hope you understand that the pip developers aren't deliberately trying to obscure the issue here - the reason there's no such message at the moment is because it's hard to detect this situation, so anyone contemplating writing a PR should appreciate that.

Anyway, as @pganssle noted, there isn't really anything actionable here, so I'm going to close the issue now.

[bluemellophone]

Fair enough, stay safe!

[gbdlin]

Sorry for digging this topic up again, but I think this was closed prematurely and due to some misunderstanding. This issue is IMO still present, but it may be poorly described. So here is my (hopefully better) explanation:

Currently there is no option to use pyproject.toml with build isolation disabled for the project by default. There is only a workaround on the user side to add --no-build-isolation when installing the project, but that's not the solution for the problem as this:

• Disables build isolation for every package installed during this pip session, including all the requirements for the package in question
• Forces custom installation process when package using pyproject.toml but incompatible with build isolation is to be installed and those steps has to be provided separately.

Why there may be need for package to have pyproject.toml but opt out from build isolation?

There may be some dependencies required for the package build that simply cannot be installed using pip (but such dependencies may be easily skipped for the runtime), to name a reason for a 2nd part of this question. The example above with six not being specified is quite misleading, as six can be installed by pip, so this is clearly an issue on the side of package maintainer by not providing proper build dependencies when doing so. A proper example should require a build dependency that is impossible to install using pip. Unfortunately, I cannot provide an example as this issue doesn't affect me personally, but I know some people affected by some python bindings for Mac OS API that has to be installed system-wide and cannot be installed inside a virtualenv. Unfortunately, I don't have any details for that, as I don't have a Mac OS environment to confirm that issue and any of people affected that I've asked to specify any details simply didn't provide those.

As for the first part of that question above, there are some tools using pyproject.toml for their configuration. As the scope of the pyproject.toml changed to allow such usage, I think there should be a way to use this file without build isolation process being forced on.

[uranusjr]

There is not, because the plan is to make all (new) pip-installable projects to use PEP 517. The old non-pyproject.toml installation support is only left for legacy releases made prior to the file’s introduction. If your project for some reason can’t be built in an isolated environment, it means either one of the followings

1. pip’s isolated environment implementation has limitations.
2. You project is not pip-installable and should be managed otherwise.

It’s perfectly fine for a project to not work with isolated builds. But if you need the project to be pip-installable, we expect it to be able to be built within an isolated environment going forward. Please raise the issues you have, so we can work on it.

[gbdlin]

I understand that isolated builds will be mandatory at some point, but we are in the transition period right now and IMO this transition period should be done as smooth as possible. If there is a possibility to disable isolated builds by the end user, there should be one for package maintainers as well.

Isolated builds are opt-in right now (implicit, by the existence of pyproject.toml), next period of the transition (as this is a major change) should be opt-out IMO, but there is no mechanic that will allow for that right now. Creating this mechanic right now, that will co-exist with opting-in via the existence of the pyproject.toml should be considered.

[uranusjr]

I fail to see how the transition isn't smooth. You test your library works in isolation builds (and other PEP 517 context), and once it does, you include pyproject.toml in your .tar.gz to enable it for your users. There's not a switch for opt-in, because pyproject.toml is the switch.

[gbdlin]

As the scope of pyproject.toml is now wider than just enabling the isolated builds, as it's now officially allowed to store the configuration for various tools inside of it, this transition may be rough for a lot of projects.

Also, as opt-in stage finishes, we shouldn't just force every package to use the isolated builds without any alternative to disable that project-wise, as not every package creators may get the memo that this switch is happening. Adding an opt-out phase will probably get their attention but with the possibility to still disable the isolated builds if they cannot move forward with it because of some not yet addressed issues.

I'm not saying that the opt-out should be permanently possible, but it should be for sure possible for some time after the opt-in phase

[uranusjr]

It’s like upgrading to a new Python version. You either make your code compatibile and then switch to the new version, or you choose not to switch and don’t get the new features. You can’t have your cake and eat it too.

[gbdlin]

I don't fully understand what you mean... As a package maintainer I have no control of which pip version the end user will use to install (and some cases also build my package, if for some reason I can't provide a wheel for his platform) my package. It's not a matter of not upgrading something on the package side when the switch in pip will happen to force build isolation on all packages.

[pfmoore]

OK, I'm going to be blunt here, as far as I'm concerned (and yes, this is a personal opinion), tools that use pyproject.toml for their configuration, are IMO abusing the file, unless they are explicitly build tools.

If we ignore such tools for a moment, what @uranusjr says is quite correct - if your project isn't (yet) able to support build isolation, then you're not ready to opt into PEP 517/518 and you shouldn't have a pyproject.toml. If you want the benefits pyproject.toml brings, you need to look at getting build isolation sorted out.

So with that said, the only remaining issue is what if you want to use a tool that puts its config in pyproject.toml? In that case, you report to that tool's maintainers that they don't have a means of configuration for projects that can't use PEP 517/518 (because they don't support build isolation yet) and could they please offer an alternative? If they say no, you have to decide how to proceed, as you want to use two incompatible tools.

Anyway, this issue is closed, so nothing is going to change as a result of any further discussion here. I hope the above explains things for you.

[gbdlin]

OK, I'm going to be blunt here, as far as I'm concerned (and yes, this is a personal opinion), tools that use pyproject.toml for their configuration, are IMO abusing the file, unless they are explicitly build tools.

According to the PEP 518, especially:

The [tool] table is where any tool related to your Python project, not just build tools, can have users specify configuration data as long as they use a sub-table within [tool]

this is not an abuse, maybe it was in the future, but it is now allowed.

That being said, we shouldn't simply ignore those tools in pip, because now we are abusing this file when making assumptions that specific package wants to use build isolation when this file exists.

[pfmoore]

this is not an abuse, maybe it was in the future, but it is now allowed.

🤷 I said it was a personal opinion...

Anyway, if we're trading PEP quotes, PEP 517 says

A build frontend SHOULD, by default, create an isolated environment for each build, containing only the standard library and any explicitly requested build-dependencies

So pip is following the PEP recommendations.

Honestly, I don't think this discussion is going anywhere, so I'll leave it here.

[RonnyPfannschmidt]

i think the spec is pretty clear that having correct build metadata is not optional, and i like that it stays that way

[gbdlin]

Spec is also pretty clear that isolated builds should be enabled regardless of the pyproject.toml file presence, but pip is not enforcing this right now. And not following the PEP recommendations then. If we really want to have it as an opt in, it shouldn't be based on the presence of this file, but rather on it's contents. Alternatively, there should be a way to disable build isolation for specific package even if the pyproject.toml file is present.

Right now, pip is slowing down the transition to the pyproject.toml file for various tools and it shouldn't have. There should be no impact created by pip in that field, but it's clearly there. Saying "it's not our problem, it's their problem" and "they don't have to use the pyproject.toml file" is not a valid solution. pip doesn't own that configuration file. It even slows the transition of packages itself as now build isolation is enabled implicitly and it is not clear why exactly to some package maintainers. The only thing they know is that removing the pyproject.toml file magically "solves their issues".

i think the spec is pretty clear that having correct build metadata is not optional, and i like that it stays that way

I understand and accept that, but this is not what I'm disputing right now. If build isolation is an opt-in right now, it shouldn't be based on the presence of this file... Or there should be no opt-in at all.

[pganssle]

I think there's no appetite from pip's maintenance team to change the established method of opting-in to build isolation except in the direction of making more packages automatically use build isolation.

@gbdlin It seems like you have a theoretical use case rather than a practical impediment here. Generally speaking, the advice here is:

1. Make your dependencies pip installable or
2. Use something other than pip to manage your installations.

Both of these are well-trodden ground, and it seems unlikely that pip is going to adopt an approach other than "try these workarounds if you have a problem."

I am not sure that I can even come up with packages that cannot be pip installable without a little preparation. All that is required is to build wheels for them ahead of time and use either a local / caching PyPI server like devpi or even a wheelhouse (local directory as PyPI server). It is also possible to use the horribly-named and impossible to search for build (github.com/pypa/build) to build a wheel for your package in whatever environment you like, then pip install that wheel.

These approaches may not be convenient, but the standard here is "it should be possible", not "it should be easy". The majority of people shouldn't see any problems if they add a pyproject.toml, the majority of those who do see problems will be able to fix them easily by adding the build metadata. The very small remaining fraction of people doing unusual things will need to adjust their approach in some way.

[gbdlin]

All I can really observe right now is that there are some tools that are postponing the migration to the pyproject.toml file as a source of configuration, because pip enables isolated builds by default and there are some projects for which it is not trivial to migrate to isolated builds. This can be very easily fixed by adding support for:

[tools.pip]
disable-isolated-build = true

or equivalent. To not break any package that relies on isolated builds, this should default to false. It has additional benefits of making the transition period easier for some packages that are not aware of isolated builds being pushed forward and will wake up some day with being forced to use them which may require several hours of development to be possible.

[pradyunsg]

FWIW, one middle ground for pip is to not do build isolation if the pyproject.toml file doesn't contain a build-system table (this also means that those projects stay in a setuptools world, but that's likely OK).

I've tried pushing for this before, but given the renewed interest here, floating that idea again. :)

[uranusjr]

No build isolation, or no PEP 517 entirely? I think I'd be fine with the latter.

(BTW can someone remind me which case is setuptools.build_meta.__legacy__ used again? I can never remember all these variants, and this is not going to make things easier 😔)

[pfmoore]

tl; dr Build isolation is unrelated to PEP 517. It's triggered by the "build via wheel" code path. There are at least 3 different "build via wheel" paths, all of which use isolation.

---

A large history-lesson brain dump. To an extent just for the record, but also to give some context when we're debating what to do here.

What the standards say

PEP 517 is actually pretty precise about how we should behave.

The specification states that there are two source tree formats - legacy with setup.py and new with pyproject.toml. There's no vagueness here, other than the (necessary) one that the setup.py based format is left undefined for historical reasons.

In that section, the PEP says

If the pyproject.toml file is absent, or the build-backend key is missing, the source tree is not using this specification, and tools should revert to the legacy behaviour of running setup.py (either directly, or by implicitly invoking the setuptools.build_meta:legacy backend).

So that again is entirely clear - if there's no pyproject.toml, or no build-backend, we should be using legacy behaviour. No PEP 517 at all (we can invoke setuptools via its build backend rather than by running setup.py, but that's the only way PEP 517 applies).

So, to be clear, if there's a pyproject.toml with a build-backend key, we must conform to PEP 517.

Build isolation is optional but recommended behaviour for conforming front ends, so we could choose not to do isolation. But pip has made the choice that we implement isolation for PEP 517, and I'm against changing that.

Legacy behaviour

OK, so let's look at the case where we hit "legacy behaviour". That's basically any case where there's no pyproject.toml or there's no build-backend key. That's the only case where we have any flexibility, and it's here where there's the most complexity.

Some of this goes back 3 years, to when PEP 517 support was added (by me!) to pip. Other parts go back even further, to the PEP 518 support (which was not implemented by me). Build isolation was introduced with PEP 518, which is slightly odd, as PEP 518 makes no mention of build isolation, but without a "create a temporary build environment" mechanism, there's not much practical benefit to PEP 518.

So isolated builds were introduced for all legacy builds, long before there even was an idea of a "legacy" build path. The only opt-out provided was --no-build-isolation. None of that has changed since that time, as far as I know, we still isolate all legacy builds unless the user opts out. This is the first important bit of context - PEP 517, and pyproject.toml actually makes no difference regarding build isolation.

Direct installs

What does have an impact here is the even older ("legacy legacy", if you like 🙂) behaviour of installing directly with setup.py install. We've been trying to kill that behaviour for years now - it bypasses all modern standards and behaviour, and uses unsafe and generally deprecated mechanisms. Removal of that behaviour generally gets discussed under the term "install via wheel".

The thing is, if you are doing a direct install, build isolation is irrelevant, because we don't do a build! So anything that uses the direct-install code path doesn't use build isolation. This is (I believe) the code path that is being taken by projects that are complaining that adding pyproject.toml imposes build isolation on them.

Installing via wheels

PEP 517 has no mechanism for installing projects, only for building wheels. So anything that goes via PEP 517 must use install-via-wheel, simply because there's no alternative.

Legacy builds may use install via wheel or direct install. The setuptools legacy PEP 517 backend, like any other PEP 517 backend, has no direct install mechanism, so that installs via wheel. The "direct setup.py" route can either install via wheel, or do a direct install.

All installs via wheel (by whatever mechanism) use build isolation, I believe.

Which precise route used is complex, but basically down to judgement calls we made 3 years ago as to how fast we should push people towards PEP 517. This is where the mess of the --use-pep517 flag comes in. When a project has no pyproject.toml or no build-backend, PEP 517 says you use a legacy install (as noted above). We can do that via the legacy backend or setup.py, and the code in pip/_internal/pyproject.py implements this choice, And this condition is what decides that we use the backend if there's a pyproject.toml.

Phew 😅

Note, by the way, that all of the logic here was implemented long before anyone suggested that using pyproject.toml for general configuration should be allowed.

So where does that leave us?

First of all, any requests to allow projects to not use build isolation are essentially requests to allow them to stick with the old, insecure, "run setup.py directly" code path. They are not using any form of PEP 517 machinery, not even setuptools.build_meta:__legacy__.

Secondly, this is stuff that has been present for three years, so it's not a new problem. What's new is people introducing pyproject.toml into projects that don't use PEP 517. This is a scenario we anticipated 3 years ago, and switching on the new, "build-via-wheel" PEP 517 based behaviour when the project added pyproject.toml was a conscious, deliberate choice at that time. It's not an accident or mistake, it's the exact transition mechanism we designed at the time.

I'm not claiming that decisions made 3 years ago can't be reviewed in the light of new fact. Nor am I even claiming that decisions I made at that time were necessarily the best ideas - a lot of what we decided was guesswork, frankly.

But what I am saying is that every part of this behaviour is in code that we've been trying to deprecate and remove for at least 3 years, and probably longer. We're retaining the old behaviour because we don't have a good way of warning users that change is coming - so we have to make the transition extremely long. But "fixing" breakages where users have encountered the change and want some of the benefits but don't want to complete the work to switch, is deliberately extending the transition period, and does nothing to help us achieve the ultimate goal of removing all of that old code.

OK, that's it. This post took me 3 hours to write, so I apologise if there are any remaining inaccuracies, but I have no more energy to go checking 💤

[pfmoore]

one middle ground for pip is to not do build isolation if the pyproject.toml file doesn't contain a build-system table

No build isolation, or no PEP 517 entirely? I think I'd be fine with the latter.

No build isolation introduces a brand new combination of conditions, that would need testing, would no doubt have weird interactions with other features, and would be yet another thing we'd eventually have to deprecate. It adds yet more complexity to the already-complex "legacy install" set of code paths. -1 from me.

No PEP 517 (i.e., fall back to the setup.py install code path) is relatively easy (I pointed out the relevant code above) but might not even work, because the legacy wheel build uses isolation. We'd have to force the direct setup.py install path if we wanted it to actually disable isolation. -1 on this for so many reasons...

If we have to do something like this, I'd argue for "no PEP 517 if there's no build-system" but with the explicit clarification that this does not disable build isolation, it simply does what --no-use-pep517 does. If that's not enough, then tough, we're not reintroducing setup.py install in cases where we've stopped using it.

And of course when the "always build via wheel" changes are completed, there will be no setup.py install route anyway, so it's only a very short term fix anyway.