Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for pyproject.toml in Pip 10 #312

Open
henryiii opened this issue Apr 2, 2018 · 10 comments
Open

Support for pyproject.toml in Pip 10 #312

henryiii opened this issue Apr 2, 2018 · 10 comments

Comments

@henryiii
Copy link
Contributor

@henryiii henryiii commented Apr 2, 2018

Pip 10.0.0b1 was released, with support for pyproject.toml. This allows a project to specify that it needs scikit-build in the setup phase, allowing for the first time a simple "pip install package" for a scikit-build package. pyproject.tomls should be added to the examples in tests and documentation.

On a related note, cmake could be added as a dependency as well, but since a user may have cmake, that's probably better in setup_requires or maybe not at all. If scikit-build could gain a new require_cmake_version argument, which would then check for the cmake command and ask it's version, then add the python 'cmake' as a requirement if this does not pass, that might be ideal.

@henryiii
Copy link
Contributor Author

@henryiii henryiii commented Apr 2, 2018

@jcfr jcfr self-assigned this May 28, 2018
@jcfr
Copy link
Contributor

@jcfr jcfr commented Jul 4, 2018

xref #313

jcfr added a commit that referenced this issue Jul 8, 2018
See #312
jcfr added a commit that referenced this issue Jul 8, 2018
See #312
@jcfr
Copy link
Contributor

@jcfr jcfr commented Jul 9, 2018

@thewtex
Copy link
Member

@thewtex thewtex commented Jul 16, 2018

@jcfr we can close this?

@jcfr
Copy link
Contributor

@jcfr jcfr commented Jul 16, 2018

Not yet:

  • add test using pyproject.toml
@stefansjs
Copy link

@stefansjs stefansjs commented Jan 15, 2021

FYI there's a relationship between build-system.requires and setup_requires that's a little less than obvious. For PEP517 builds I noticed that I needed to already have my dependencies installed by declaring them in pyproject.toml, and then setuptools will fail if the setup_requires criteria don't already match the environment.

More specifically, yes you need cmake in build-system.requires. Here's an example of my pyproject.toml and the relevant chunk of setup.py:

[build-system]
requires = [
    "setuptools>=50.0",
    "wheel",
    "scikit-build",
    "cmake==3.14.4",
    "Cython==0.25.1; python_version>='2.7' and python_version<='3.6'",
    "Cython==0.29.19; python_version>='3.7'",
    "cysignals==1.6.5",
    "pycodestyle==2.5.0"  # This is required in order to install avro-python3
]
        setup_requires=[
            'scikit-build',
            'cmake==3.14.4',
            # I tried specifying cython versions with environment markers, but setuptools doesn't respect them.
            # Relying on pyproject.toml to correctly install the right cython for the right python interpreter
            "Cython>=0.25.1",  # This version doesn't work on 3.7+
            "Cython<=0.29.19",
            'cysignals==1.6.5',
            'pycodestyle==2.5.0',  # This is required in order to install avro-python3
        ],

(I had issues with cython). It was interesting that I could use all the environment markers in pyproject.toml, but setup.py just needed me to list the same requirement multiple times.

Just wanted to share my experience so that others don't go through the same painful process I did to figure it out.

@henryiii
Copy link
Contributor Author

@henryiii henryiii commented Jan 15, 2021

setup_requires is problematic and if you can avoid supporting non PEP 517 builds, you can just delete it, it's deprecated and doesn't really work the way you think. Specifically, if you have something in setup_requires, it only gets downloaded, not installed, so you have to list it in install_requires, even if you don't need it for the installed package (like Cython). It also causes a double pass through the setup.py, and only the second time will you have the contents of setup_requires. The only reason to keep setup_requires is to support non-PEP 517 builds. If you do list both, it might certainly require they match, since I'm not sure it will do the download step if it's already in the PEP 517 environment.

Also, just in case, always list your build backend if you use build-system! The default one is the somewhat broken setuptools __legacy__ builder. See https://scikit-hep.org/developer/packaging#pep-517518-support-high-priority.

@stefansjs
Copy link

@stefansjs stefansjs commented Jan 15, 2021

Oh thanks for the feedback. Those are some good pointers I didn't know.

I have never seen setup_requires install any dependencies. It's a catch-22 that's solved by pep517. You can't successfully run setup.py without having those things installed, which only works if you have some way to declaratively install those dependencies.

That said, I've found that it is useful to fail if the wrong dependencies are installed. In my case cython will produce a binary that doesn't work but which doesn't fail until runtime. Hence failing at build time is an advantage in my case.

@henryiii
Copy link
Contributor Author

@henryiii henryiii commented Jan 15, 2021

If you are careful, yes you can. You just have to list the dependencies in both setup_requires, and install_requires, and then very carefully design it so that the first run of setup.py does not have any errors; it won't build anything the first time round. But it does work. See pybind/python_example@fc12654 - the "previous" version did work (although teaching people to use setup_requires is bad so I moved it to pyproject.toml quickly after setting it up.).

Cython happens to be especially easy, since setuptools years ago added native support for Cython files alongside the normal .cpp files and such for extensions, so you don't even have to bother with working around importing it.

I agree, it's a very good idea to verify that requirements are there in setup.py; for example, someone setting up a conda-forge recipe might disable build-isolation and specify the dependencies manually.

@henryiii
Copy link
Contributor Author

@henryiii henryiii commented Jan 15, 2021

The catch-22 is you can't update setuptools this way, that's what prompted PEP 518. You also are stuck with build requirements being installed even when not needed by the final package. And working around the double run is a pain.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants