-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support installing requirements from arbitrary keys in pyproject.toml #8049
Comments
I'd rather support 1 A quick search found https://github.com/jamesmunns/tomlq |
Personally I tend to think all |
|
I'm looking for something like this too. My use case is making sdists in CI. That's great if I'm making a wheel, but the same problems exist for sdists, which remain unsolved. Even though I'm not building anything for an sdist, some of my projects still import So what I would really like is a
pip install toml && python -c 'import toml; c = toml.load("pyproject.toml")
print("\n".join(c["build-system"]["requires"]))' | pip install -r /dev/stdin Which is what I'm currently considering putting in my CI. |
@chrisjbillington for building sdists, you could try |
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
Please do not use non-standard top-level keys in |
We need to build all protobuf files to be able to use sphinx autodoc (otherwise `metricq` is not importable). But we cannot run `setup.py` to build protobuf files before having all build dependencies installed (we require `mypy-protobuf` for example). These dependencies are specified in `pyproject.toml`. Currently, no tool supports simply installing dependencies from this file. Options are: 1. `pip wheel ...`: Does not work; resolves and installs build dependencies from `pyproject.toml` in an isolated directory. They are not accessible anymore after the wheel has been built. 2. `python -m pep517.build --source .`: Does not work for the same reasons as above. This would work if the source distribution this produces included the documentation source files (`./docs/*`). 3. Duplicate the list found in `pyproject.toml`: Rejected, DRY. 3. `pip install -r pyproject.toml`: Not supported by pip [1]. Option 4 leads us to this [2] *wonderful* solution: Parse `pyproject.toml` ourselves, tell `pip` to install the build dependencies we found. Further advise on how to deal with Python packaging issues can be found here [3]. [1] pypa/pip#8049 (comment) [2] pypa/pip#8049 (comment) [3] https://www.youtube.com/watch?v=vLaX8UvVUQw
We need to build all protobuf files to be able to use sphinx autodoc (otherwise `metricq` is not importable). But we cannot run `setup.py` to build protobuf files before having all build dependencies installed (we require `mypy-protobuf` for example). These dependencies are specified in `pyproject.toml`. Currently, no tool supports simply installing dependencies from this file. Options are: 1. `pip wheel ...`: Does not work; resolves and installs build dependencies from `pyproject.toml` in an isolated directory. They are not accessible anymore after the wheel has been built. 2. `python -m pep517.build --source .`: Does not work for the same reasons as above. This would work if the source distribution this produces included the documentation source files (`./docs/*`). 3. Duplicate the list found in `pyproject.toml`: Rejected, DRY. 3. `pip install -r pyproject.toml`: Not supported by pip [1]. Option 4 leads us to this [2] *wonderful* solution: Parse `pyproject.toml` ourselves, tell `pip` to install the build dependencies we found. Further advise on how to deal with Python packaging issues can be found here [3]. [1] pypa/pip#8049 (comment) [2] pypa/pip#8049 (comment) [3] https://www.youtube.com/watch?v=vLaX8UvVUQw
If building documentation is successful, the generated HTML is uploaded as a build artifact. On a matching rev, it is also deployed to GitHub Pages. We need to build all protobuf files to be able to use sphinx autodoc (otherwise `metricq` is not importable). But we cannot run `setup.py` to build protobuf files before having all build dependencies installed (e.g. `mypy-protobuf`). These are specified in `pyproject.toml`. Currently, no tool supports simply installing dependencies from this file. Options are: 1. `pip wheel ...`: Does not work; resolves and installs build dependencies from `pyproject.toml` in an isolated directory. They are not accessible anymore after the wheel has been built. 2. `python -m pep517.build --source .`: Does not work for the same reasons as above. This would work if the source distribution this produces included the documentation source files (`./docs/*`). 3. Duplicate the list found in `pyproject.toml`: Rejected, DRY. 4. `pip install -r pyproject.toml`: Not supported by pip [1]. Option 4 leads us to this [2] *wonderful* solution: Parse `pyproject.toml` ourselves, tell `pip` to install the build dependencies we found. Further advise on how to deal with Python packaging issues can be found here [3]. [1] pypa/pip#8049 (comment) [2] pypa/pip#8049 (comment) [3] https://www.youtube.com/watch?v=vLaX8UvVUQw
This comment was marked as off-topic.
This comment was marked as off-topic.
My problem is completely addressed by |
@chrisjbillington in the meantime, build became a thing so the awkward name issue is resolved too. |
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as resolved.
This comment was marked as resolved.
I wouldn’t call it a hack, it’s simply very manual. A hack is something reaching into APIs you’re not supposed to muck around with. This uses bog-standard, spec compliant ways to do what we want. But it is awkward. Slightly less is e.g. using rq to convert it to JSON and then jq to query and transform: rq -t <pyproject.toml | jq -r '.["build-system"]["requires"][]' | xargs -d '\n' pip install PS: |
This comment was marked as resolved.
This comment was marked as resolved.
The same can be achieved by |
@uranusjr - just to display the case in a different angle.
What I believe most community members here are asking is to be able to use pyproject.toml as a replacement for the Pipfile concept. This is not in conflict or diminishing the need for a Pipfile.lock mechanism / full protected installation through pyproject.toml, but this is just a different scenario which seems to be already covered in the proposed PEP 650 where if you to specify an installer backend will take care of that (Please correct me if wrong). However, for developers who are not asking for locking mechanism (for their own reasons) installing from pyproject.toml as alternative of requirements.txt seems very much reasonable. |
I don’t disagree with your analysis, but pipenv installs from
The general consensus of pip maintainers working on this issue is that people should ask for locking in all cases, so we’re going for a lock file solution. People are free to disagree with this decision, but they can always work on their own solution in their own tool as well. |
It wasn't clear from previous discussions but I get your point now, that makes more sense - thank you for the elaboration.
Yes - that was clear. |
Yeah, we’re working on a completing proposal to PEP 650 that does introduce the lock file. Stay tuned. |
For those in this same boat, here's the solution I ended up using. For devs, I just make a venv, then |
This comment was marked as off-topic.
This comment was marked as off-topic.
I've just gone through and hidden a bunch of comments that are about "how to build a pyproject.toml based project into a distribution" -- which is not what this issue is about. I've also retitled this issue to better reflect the original request from @dholth. Responding to the original request here: I don't think reading from pyproject.toml and installing arbitrary keys is a use case that I want pip to directly handle. It is already possible to do this, by using other tools to parse the pyproject.toml file and passing those values into pip -- as demonstrated in #8049 (comment). To that end, I don't think this functionality belongs in pip. I can see users (ab)using this to install from |
I filed #11440 to bring this forward. |
Waaait a sec.. Why is this discussion about pip reading And:
|
@ctapobep I think you're confused here. This has nothing to do with backends or Poetry, or lock files. Poetry isn't even a backend here, it's a frontend (it offer both and more, but we're talking about installing which is a frontend feature). If you want to install dependencies with Poetry, use |
@rgommers okay, I may have been late for the party, and this may not be the right issue to comment in (just the 1st one that google returned for "pip install pyproject.toml").. But the fact that pip is trying to read & install dependencies from Why do we use This whole So what we get is:
Seems 0:2 for "Python vs. build tools". |
I'm going to close this issue, as I don't believe it's something we should do in the form suggested here. Most of the actionable items either have their own issues, or can be done already.
|
What's the problem this feature will solve?
It can sometimes be useful to install from a
pyproject.toml
that has a list of install_requires = [] without having to build the whole package.pip install -r pyproject.toml --key tool.enscons.install_requires
would open pyproject.toml, look up['tool']['enscons']['install_requires']
, and install as if those items had been passed as arguments topip install
, or written out to a requirements.txt file and then installed.It wouldn't work correctly if the package referenced its own extras, unless a second key for the extras dict was passed.
If this feature was added it would be easy to support .json as well.
The text was updated successfully, but these errors were encountered: