Project setup.cfg causes errors in cibuildwheel installing other packages #1487
Comments
|
That way that you're using setup.cfg is quite unusual. Normally setup.cfg is a static file that lives with the project and is committed to source control. If you want to provide build-time parameters to your package, I'd recommend using environment variables- you can set them for this build using CIBW_ENVIRONMENT. (There might also be a way to use CIBW_CONFIG_SETTINGS for this too but I'm not sure how that all works in setuptools. ) |
|
It is a static file and it is part of the project! This issue would still apply - the intention surely can't be that the project setup.cfg should affect other non-project packages cibuildwheel is installing! Some details about why and what is done: APSW wraps the most recent SQLite library (source or binary), which means the most recent SQLite needs to be present at compile time. It has always had an option to automatically fetch the SQLite source code for you. When someone is doing an install directly from the github source release they will need to choose how SQLite is found. For example they can supply the For a pypi release it isn't practical to have all those mechanisms, and compilers + header files are unlikely to be present. So the decision is to have SQLite automatically fetched and have binary builds on pypi. There is also a source distribution on pypi and it should produce exactly the same results as installing a binary release. To make a source pypi distribution I include a setup.cfg that does the appropriate options, because compilation is done on the users machine. For the binary builds I had been specifying stuff in environment variables etc for cibuildwheel. That then led to a bug (and doubled maintenance burden), because the two different mechanisms got out of sync. So I made the binary build also use the same single setup.cfg, which led me to creating this issue. |
|
I'd highly recommend a) adding a pyproject.toml, with I'm not sure how most users get your project, but often users are building from SDist, which does not even offer the opportunity to edit a project file like setup.cfg! If you really want to be able to configure the build, you need to switch to a backend that supports the modern By the way, I thought you said you have no dependencies, so what is failing to build? It has to be something that's not provided via wheel, why are you building someone else's wheels in your CI? |
|
You have this in your config, and it's the line that's failing, I think: CIBW_BEFORE_BUILD: python setup.py buildThis is very much not supported. You should not build before you build. You need to use the built-in project builder ( Also, calling |
|
Sorry, I am getting very confused by the responses. If you are saying that it is an error for a project to have a setup.cfg when using cibuildwheel, then close the issue, and ideally make cibuildwheel refuse to run if setup.cfg is detected. I am the project developer. I am the one supplying the setup.cfg. There is no user involved. The build process, setup.py, setup.cfg, etc all work and have all worked with cibuildwheel for ages. There is absolutely no problem there. I understand that this traditional distutils type approach is deprecated, but it still provides the most reliable path for building C extensions especially if some customisation is needed like me adding options to fetch SQLite, and configure what extensions in SQLite are enabled. Everything was good until the other day when I moved some of those options out of CIBW variables and put them into a setup.cfg. This bug report is that the options I put in my setup.cfg for my setup.py (which works perfectly) now fail under cibuildwheel because my setup.cfg is being used when cibuildwheel is installing other packages. The root cause of that is that cibuildwheel has the current working directory be my project when doing the other package installs. If you want proof that this stuff all works with my project then do this, feeling free to replace the last line with an alternate build command. You will see that it does the fetch as directed by setup.cfg. The line you quote does not fail, and in fact wasn't reached because the install of other packages by cibuildwheel before it was reached failed because they tried to use my setup.cfg! As mentioned I did a later update where CIBW_BEFORE_BUILD is used to copy the setup.cfg into place. This is the workflow which does the build step just fine. Here it is on ubuntu and you can see the build step took about 90 seconds which is about right, and would have failed if the fetch flag had not been detected or used. Please please note that by clicking on any of the build_binary steps that building works, and then cibuildwheel does the "Testing wheel..." bit and runs It is that command that is failing. The reason for the failiure is that |
|
Sorry, no, didn't mean From what I understood, you changed from the (working) design to setup.cfg so that users could build your package and only edit or replace setup.cfg. I (or one of us, probably will be able to look into it tomorrow) needs to understand what exactly is happening here, though. Since you say you have no dependencies, only our dependencies should be being installed, and they should all provide wheels, so you should never even trigger a build step, so nothing should even try to look at setup.cfg. And the local setup.cfg shouldn't affect other packages, except maybe in the case that a dependency is missing a You should never break your local directory so someone can't run |
|
FYI, I can only find 500 files in all GitHub with Also, making sure we install our requirements in a non-project directory sounds fine. But I would like to understand why it’s affected and not just pulling wheels. |
|
I meant that the existence of setup.cfg here is a form of build configuration, rather than static - at build time it is copied into place to configure the build. I only meant that env vars might be a better choice for build configuration.
I suppose I'd agree! But the issue is not with cibuildwheel. This config file affects distutils, a module that (until recently) is part of Python. I was curious so I added your config file to a project, and I couldn't get any of these to work on Python <=3.9: So, even if we did happen to install packages from outside your project root, you're gonna have a lot of problems with this approach. |
I too was curious - this is the backtrace from a pip install- It seems that pip still uses distutils, even when installing a wheel (speculating, but perhaps to get the install location?). Distutils loads the config file and raises the error. |
|
There seem to be two simultaneous responses going on here. 1: Don't use setup.cfg, use toml files, use environment variables, etc etc which will make this issue not be hit. I agree that would be the case. However please give me the benefit of the doubt, that there are good reasons why things are done the way they are and they do work. I would be delighted to continue this specific conversation somewhere more appropriate, but it is NOT relevant to this issue being reported here - it is an avoidance of the issue. 2: A setup.cfg in the project directory has no effect on cibuildwheel installing other software The second one is wrong - it does and what this issue is about. I have created a minimal standalone test case at https://github.com/rogerbinns/issue1487 It adds Here is the clickable build run https://github.com/rogerbinns/issue1487/actions/runs/4851042323/ Line 419 has cibuildwheel running Testing wheel... and doing Which fails because (backtrace ends line 448) This proves that the setup.cfg in the project directory is affecting the pip install of virtualenv. |
|
IMO, you should not break I didn't doubt that you are seeing this failure on adding I think changing the Also, the suggestion that you add the following pyproject.toml: [build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_api"Is a general one and not related to the files you have. It should be done on all projects that today assume setuptools and wheel will be pre-installed (this is changing, actually already has changed in virtualenv as of a few days ago), and it ensures your build procedure will not change depending on pip version. And it ensures you'll always have |
|
I am happy to test potential fixes like PR1488 - let me know when (and probably how). Some other potential (simpler?) fix approaches:
The real gotcha is pip install using setup.cfg in the current directory to install content not from the current directory. Partial thread 1 discussion:
Adding a pyproject.toml means addressing all these points! |
|
The test environment phase shouldn't happen inside the build environment - it's optional, and that's mixing steps. The most common line in CIBW_BEFORE_TEST is probably 1). Up until setuptools is removed from the default environment, which is already happening. The first step happened 4 days ago with Adding a pyproject.toml doesn't require addressing anything, it simply informs tools that were updated on or after 2016 that setuptools is used and lets them use modern isolated builds. That's it, it doesn't "change" setup.cfg, setup.py, or MANIFEST.in. When setuptools is removed from the default environments, projects with a (And I'm pretty sure it won't cause this to start working, as the isolation won't affect Footnotes |
|
You can test the PR now if you want. Take a line like |
|
How did you get that failure outside of cibuildwheel, @joerick? I am not seeing it in https://github.com/henryiii/issue1487. I'd have expected https://github.com/henryiii/issue1487/blob/2be7d04d9511ba24eeab4cd815b0135cc0044f6c/.github/workflows/build.yml#L39 to break. |
|
Ahh, never mind, missed the Python < 3.9 part. Got the failure! So this is only an issue with 3.6-3.8. So this is an example of a something you can't do (before Python 3.9) as soon as that setup.cfg is present: on:
workflow_dispatch:
pull_request:
push:
branches:
- main
jobs:
build_local:
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: "3.8"
- run: pip install virtualenv |
|
FYI, if you "fix" the test step by running it as in #1488, then the next line fails: Since pip can't do anything if this is set, even install a local wheel. So it's not really a fix. It might make the workaround a bit easier, since you can now delete the copied This design also stops you from adding a I believe the correct way to do set this, rather than coping around a setup.cfg file, would be to pass IMO, setuptools did not support this added build-options in setup.cfg based design before Python 3.9, and they have been moving away from these sort of options (which is also why you don't see it used in other projects). (@abravalheri can correct me if I'm wrong?) Can't you just make these three options environment variables that cibuildwheel sets? |
|
@henryiii some responses
That is ok and something I will deal with as it happens. Please note that cibuildwheel is not the only or even the most common way the project is built. To my knowledge the only person who uses cibuildwheel is me once per release in order to publish on pypi. Everybody else is invoking setup.py with appropriate options for their situation. There are many options covering how SQLite is found and its configuration, with no two using the same options, all of which matter at compile time.
And they can. Using standard releases from github do not have a setup.cfg at all. Doing a pip install of a source release from pypi does have the setup.cfg, but you aren't installing things into the current directory of the extracting source while pip does it build etc steps.
This has been an issue for years! Python has not yet provided a better means for building C extensions and letting the people who do the build provide options. Even the documentation still shows it done this way. setup.cfg is the only way to provide options to the various setup.py stages in the situation where no explicit command line parameters are provided. If there is a better I'm all ears!
Yes you can :-) There is no setup.cfg - look. If you grab a source release then there is a setup.cfg not made by me with two lines about eggs.
The only build that is broken is using cibuildwheel because when a setup.cfg is added just before build, the later pip installs done by cibuildwheel are done in my project directory.which picks up the setup.cfg not intended for them. The other builds involve all the Linux and BSD distros, various other developers who want different customisations and options, those doing a pip install from pypi where there isn't a binary for their platform, and my testing which involves all the combinations of SQLite versions, 32 & 64 bit, all the Python versions in debug and release, Windows, etc.
Just that the in development version of Python still installs setuptools - ie it hasn't actually been removed from "Python", yet.
It means I have to test it actually works across the matrix of all the releases, tools, versions etc. And will have no effect on this specific issue of setup.cfg. BTW PEP 518 has wording about the file and assuming what is there. It doesn't appear to be superseded. |
Thank you for making the changes. Here is the build doing it in the issue1487 repository I made earlier. Progress - the "Testing wheel" stuff now proceeds past the pip install virtualenv part. However things fall over when trying to install the built wheel. Again it is the same issue - the pip install is run in my project directory to grab the wheel from elsewhere and the setup.cfg is being applied to some other setup.py which falls over. Having the working directory be the venv or similar should address that. May I suggest also including the working directory each command cibuildwheel is run in is made part of the log. |
This is exactly the point of pyproject.toml and removing setuptools as the one and only way to make Python projects. Other tools have been popping up with much better support for compiled extensions. But we can't move forward if people will not use pyproject.toml and specify what backend they want to use.
My point is, if you did put this file in, it would break pyproject.toml builds, virtualenvs, and all pip install of anything, including your own package. The only reason it's working is because it's not in any build except cibuildwheel. You are copying a file that breaks all these things into cibuildwheel and then complaining that it breaks cibuildwheel.
It's not in Python. Python does not contain setuptools. It contains "ensurepip", which installs pip and setuptools. It also contains venv, which includes wheel and setuptools by default, but that can be deactivated with
It actually completely breaks it, because it requires that Pip may change to using modern builds by default as soon as 23.3. More likely probably a version or two after that, though. |
|
Your changes also mean CIBW_BEFORE_TEST can now remove the setup.cfg, and a build test in my issue1487 works perfectly. This means that your changes are sufficient for me to work around my setup.cfg being picked up by pip install of other stuff that it shouldn't apply to. I do still believe that commands like pip install should not be run with the cwd being the project because files (like setup.cfg) can affect that install in hard to detect and certainly unwanted ways. I will now do a build on my main project to do a full test. It takes many hours because of all the platforms. |
|
These are my needs for building my C extension
That is how I end up at the least worst solution of setup.py and setup.cfg. I would be delighted if something better exists ... |
|
I got bitten by the os.abspath thing too, but you'd fixed it as I looked. On retrying I got daring and added the pyproject.toml. That then caused a build failure because the pip installs are done with my project as the current working directory which picks up the setup.cfg which then gets applied to a package outside of my project which then fails. This is what you were describing above.
Yes I can easily do that, and it is what I was doing. However if someone does a pip install of the source distribution from pypi it needs to have the same configuration. ie a binary or source install from pypi should have the same options established. The only way I could make that the case for the source distribution is by including a setup.cfg inside which is what I did. That then resulted in having to maintain the exact same information twice in two different formats - once in the github workflow environment variables and again in the setup.cfg. Those ended up unintentionally diverging, so I went for only having it in the setup.cfg. Which then led me to discovering this issue. The root problem is really that pip install is using setup.cfg from the current directory when downloading an external package and installing it somewhere else - ie the current directory is not relevant and should have no effect. I tried to make a reproducer in a standard venv and at no point did setup.cfg in the cwd have any effect. So this pip install using setup.cfg thing seems to only happen when using cibuildwheel. |
|
I have a reproducer outside cibuildwheel at https://github.com/henryiii/issue1487, based on yours. You need Python <3.9 to see the issue. |
|
You shouldn't and can't get rid of setup.py (without completely writing a new build system, like CMake/Meson, which also would probably require Python 3.7+ as we don't support 3.6 and this is all fairly new development). But using setup.cfg for this is not the way setup.cfg is normally used, and breaks on Python 3.6, 3.7, and 3.8. What about adding a new section to setup.cfg, like |
|
Could you try #1488 again? |
|
A build I did yesterday afternoon (California time) ended up timing out. I started another two hours ago from this message and am having the similar problems. However the cibuildwheel bits relevant to this issue are working fine. What I was trying to make a reproducer for is pip install using a local setup.cfg when installing a remote package, since that is the root cause of this issue. In particular when I try from outside of cibuildwheel the local setup.cfg is ignored, so something somewhere is causing the behaviour difference. |
|
My build completed which includes all the platforms plus qemu ones, all the Python versions, running my tests, and publishing to (test) pypi. Looks good to me. |
|
The issue occurs in Python 3.9 and earlier and is triggered by code in pip. I have two possible fixes.
@henryiii I'd appreciate your advice on how to proceed Short description of the problem:pip wants to get the list of all possible install directories which ends up instantiating a distutils install object which instantiates an unused distutils build object which doesn't understand build options from the First fixThis creates a dummy distutils build command that accepts all options. Changes are to class dummy_build(DistutilsCommand):
def __getattr__(self, name):
return []
initialize_options = finalize_options = lambda *args: NoneThen in d = Distribution(dist_args)
d.cmdclass["build"] = dummy_build # override default build
if not ignore_config_files:This fix means the Second fixAppend this line to the end of the _PIP_USE_SYSCONFIG = TrueThis fix means that Long description of the problem:The exception is happening when pip wants to know if all package directories are writable, long before doing the actual install. Package directories are found by calling
If If it is
I don't think the |
The ship sailed on this long ago - it should have been fixed in pip before they dropped support for Python 3.6, and they are dropping support for 3.7 soon. Placing custom additions in The "fix" in #1488 doesn't seem too bad as it just isolates our extra package install steps and doesn't patch anything. I would highly, highly recommend you reconsider your design; it's not valid until you drop Python 3.9. If pip does accept a patch, then it might become valid for the latest pips + 3.7+ or 3.8+, depending on if there's a release with it in it for 3.7. Most packages have managed without adding extra |
|
Excellent, will watch that pip issue. FYI, setuptools was just removed from Python 3.12's ensurepip and venv two weeks ago in python/cpython#101039. So 3.12 will truly be setuptools and distutils free. :) |
|
How exactly are C extensions supposed to be built without setuptools and its embedded copy of distutils? Is this just a case of setuptools not being installed by default, or completely gone? The doc still has setuptools. My original problem is that I want to specify build parameters in one place only. I had it in a setup.cfg for the source distribution that goes to pypi and as command line flags in the github action, and it was a source of bugs because they easily get out of sync. I'm investigating another approach now, where the idea is to specify the command line flags in the github action, and then generate a setup.cfg from that, that is used when making the sdist. However I can't find any documentation or examples of how to specify options to cibuildwheel that are then seen by setup.py. Even my workflow above is doing it in the CIBW_BEFORE_BUILD command which is the wrong place. Still looking. |
setuptools will not be shipped by default. But it sill could be downloaded if needed. |
|
I am completely stalled. The only way to pass options to setup.py I couldn't find any way of passing the options on the command line via If pip is not fixed, my last resort will be monkey patching https://github.com/pypa/setuptools/blob/main/setuptools/command/setopt.py#L19 to make it look for a different file. Yuck. |
|
Let's put the setup.cfg stuff to one side. in terms of suggesting solutions here, can we go back to first principles? My understanding is that you want to have different build settings between source checkouts, wheels builds and sdists, with source checkouts the odd-one-out. What are the specific options that you're setting, and what is the software in the build process that is reading these options? |
|
@joerick I have implemented a workaround by monkey patching distutils, which does work going back to Python 3.7 (distutils still in stdlib) and Python 3.12a7 (distutils buried inside setuptools). That looks for a different filename but otherwise has the same semantics as To help with the understanding: My project consists primarily of C code gluing the C api of CPython with the C api of SQlite. In order to build the project, the standard pattern of a Various bits of information are needed at compile time, which includes compilation flags, debugging, whether optional functionality in SQlite is present etc. It also requires the current(ish) version of SQLite, which is unlikely to be present on the platform. To address that my The extension is built by the various Linux/BSD platforms, by developers who use it in their projects (sometimes including their own extra code at compilation time), and by me for publishing binary builds and source to pypi, and by me for source github releases. In the olden days they would invoke it like
For pypi builds (using cibuildwheel) I had to pick a specific set of options, such as embedding a specific version of SQLite inside the extension, and which extensions are enabled/disabled. It is also important to me that if you get a binary build from pypi, the source build produces the same result with the same configuration. ie if there wasn't a binary build for your platform, the source based build pip does gives you the same configuration and options. This is how configuration was done:
This issue arose because I had a bug in pypi installs where the configuration I specified in the workflow got out of sync with the |
|
The options that you set in setup.cfg, are they setuptools options or are they configuring your own code? |
|
Some are for my own code (eg fetch related stuff, omitting/including functionality) and others are generic distutils/setuptools options. This is the setup.py |
|
All I can see in the repo are is this config [build]
fetch = True
enable_all_extensions = True
enable = COLUMN_METADATAAll of those appear to be read using this code in a subclass of a build command in setup.py # We allow enable/omit to be specified to build and then pass them to build_ext
build_enable = None
build_omit = None
build_enable_all_extensions = False
bparent = build.build
class apsw_build(bparent):
user_options=bparent.user_options+\
[ ("enable=", None, "Enable SQLite options (comma separated list)"),
("omit=", None, "Omit SQLite functionality (comma separated list)"),
("enable-all-extensions", None, "Enable all SQLite extensions"),
("fetch", None, "Fetches SQLite for pypi based build"),
]
boolean_options = bparent.boolean_options + ["enable-all-extensions", "fetch"]
def __init__(self, dist):
self._saved_dist = dist
bparent.__init__(self, dist)
def initialize_options(self):
v = bparent.initialize_options(self)
self.enable = None
self.omit = None
self.enable_all_extensions = build_enable_all_extensions
self.fetch = False
return v
def finalize_options(self):
global build_enable, build_omit, build_enable_all_extensions
build_enable = self.enable
build_omit = self.omit
build_enable_all_extensions = self.enable_all_extensions
if self.fetch:
fc = fetch(self._saved_dist)
fc.initialize_options()
fc.all = True
fc.finalize_options()
fc.run()
return bparent.finalize_options(self)So I'm wondering - rather than configuring these using the setuptools options system, can you read defaults for these options from somewhere else? e.g. you could have a build-defaults.json file and read it in setup.py like import json
with open('build-defaults.json') as f:
build_defaults_json = json.load(f)
build_enable = build_defaults_json['enable']
build_omit = build_defaults_json['omit']
build_enable_all_extensions = build_defaults_json['enable_all_extensions']The reason I'm suggesting this is I think the core of your issue with setup.cfg is the creation of new options - putting the options elsewhere would stop distutils from choking on them. |
|
You can even put it into |
|
Though I think another file is better, as a setup.cfg is supposed to be static and contain permanent options (like most of the options you have in setup.py now), not something a user copies in from a tools dir. |
I understand where you are coming from and thank you for doing the investigation. My recent bug was due to having two different ways of configuring the same thing, and then getting out of sync. Adding another way, including debugging and testing of it is something I'm avoiding :)
I actually want the exact semantics that setup.cfg already provides, and don't want to have to reimplement the code in "distutils" that automatically loads the right options from the right sections and applies them to the right commands in the right way!
It does for the environment where a build is happening. Builds using cibuildwheel for pypi done by me use the file I have, while other developers who build for their own environments have their own. The defaults in Good newsThe monkey patching I did as rogerbinns/apsw@ee7a632 works perfectly. If a The project |
|
Well, glad you got something working! |
Description
My project is a C extension. Of note it has no dependencies on any other package for building or testing.
To customize the github actions binary build I added a
setup.cfgto it with some additional options defined.This is causing an error when cibuildwheel is installing other packages because the current working directory is my project and my
setup.cfgis affecting their installation. In particular thefetchoption causes their installation to fail because they have no knowledge of it (and it is my option not theirs).Here is an example of the problem occurring:
I did a second run where I changed the setup.cfg to not be present until just before the build so that it didn't affect the build packages being installed (using CIBW_BEFORE_BUILD). The test step then fell over:
I tried removing the setup.cfg in CIBW_BEFORE_TEST but that is run after the step above so it is too late.
This is the build log when CIBW_BEFORE_BUILD adds the setup.cfg so building works, but then falls over on the Testing wheel stage.
Suggested solutions:
Build log
https://github.com/rogerbinns/apsw/actions/runs/4844599878/jobs/8632966833
CI config
https://github.com/rogerbinns/apsw/actions/runs/4844599878/workflow
The text was updated successfully, but these errors were encountered: