Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support poetry.lock #835

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

Support poetry.lock #835

wants to merge 10 commits into from

Conversation

cjolowicz
Copy link
Contributor

@cjolowicz cjolowicz commented Aug 28, 2019

Export pinned requirements from poetry.lock to requirements.txt. This PR is a follow-up to #833 and #834.

If a poetry.lock file is present, use the Poetry CLI to export the pinned requirements to a requirements.txt file. If the project already contains a requirements.txt, use that and ignore poetry.lock.

Install pyproject.toml-based projects using pip install ., instead of creating a requirements.txt containing .. The requirements.txt file is needed to handle pinned versions extracted from the poetry.lock file.

Closes #796


CI is broken for PRs from another repository, see this comment. Please check the corresponding PR in my fork to verify CI:


Why is this necessary at all, after supporting PEP 517 installations?

Note that supporting pyproject.toml-based installations is not enough to handle pinned requirements: When pip installs a pyproject.toml-style project using the process described in PEP 517, it only uses Poetry to build a wheel. The wheel contains the version constraints from pyproject.toml, not the pinned versions from poetry.lock. See python-poetry/poetry#537

Why not use Poetry to install?

Poetry is not used to install the project because it does not clean up stale requirements. This means that requirements need to be exported anyway, for the pip-uninstall step. See python-poetry/poetry#648

Why use Poetry 1.0.0b1 instead of the version specified in pyproject.toml?

Since we only use Poetry to export a requirements.txt file, ignore the Poetry version specified in pyproject.toml. Install a pre-release of 1.0.0 because the export command is not available before 1.0.0a0.

Update: Poetry 1.0.0 has been released. Also, the version specified in pyproject.toml is a minimum version, so it would not make sense to install it anyway.

Why not append . to the exported requirements file?

Note that we cannot simply append . to the exported requirements.txt file. Currently pip requires that either all requirements have a hash or none. Including . would thus force us to omit hashes for all requirements. See pypa/pip#4995

Test case

Add a test to check that pinned requirements are honored when poetry.lock is present. Request an outdated version of marshmallow in poetry.lock (3.0.0). Check that the outdated version is installed, rather than a newer version also satisfying the version constraint in pyproject.toml (^3.0.0).

@cjolowicz cjolowicz requested a review from a team as a code owner August 28, 2019 11:41
@cjolowicz cjolowicz changed the title Support poetry.lock WIP: Support poetry.lock Aug 28, 2019
This was referenced Aug 28, 2019
@cjolowicz cjolowicz force-pushed the poetry-lock branch 2 times, most recently from 22005a2 to 143d0f8 Compare August 29, 2019 08:29
@cjolowicz cjolowicz changed the title WIP: Support poetry.lock Support poetry.lock Aug 29, 2019
@cjolowicz
Copy link
Contributor Author

Rebased onto master (v157).

  • Tests were moved from test/run to test/run-features

@cjolowicz
Copy link
Contributor Author

Rebased onto master (af860be)

@cjolowicz
Copy link
Contributor Author

Poetry 1.0.0b2 was released on 2019-10-11:

https://github.com/sdispater/poetry/releases/tag/1.0.0b2

@cjolowicz
Copy link
Contributor Author

Poetry 1.0.0b3 was released on 2019-10-25:

https://github.com/sdispater/poetry/releases/tag/1.0.0b3

@cjolowicz
Copy link
Contributor Author

Poetry 1.0.0b4 was released on 2019-11-08:

https://github.com/sdispater/poetry/releases/tag/1.0.0b4

@cjolowicz
Copy link
Contributor Author

  • Rebased onto master (93a620e)
  • Upgrade to Poetry 1.0.0b7 (03e27237193405f4ce017fe107b7ea7154aa5883)
  • Minor cleanup (3998f12508bfe2245da76e3b0b10b8b0421180b1)

@cjolowicz
Copy link
Contributor Author

cjolowicz commented Nov 24, 2019

Hey @CaseyFaist, are you still planning to merge this series of PRs? If so, do you have an approximate ETA? Thanks!

Add two test cases for projects using poetry as the build backend:

- testPoetry: project without dependencies

- testPoetryLock: project with dependencies, using a poetry.lock file
Add two test cases for projects using flit as the build backend:

- testFlit: project without dependencies

- testFlitRequires: project with dependencies
Check that pinned requirements are honored when poetry.lock is present.
Request an outdated version of marshmallow in poetry.lock (3.0.0). Check
that the outdated version is installed, rather than a newer version
also satisfying the version constraint in pyproject.toml (^3.0.0).
Install pyproject.toml-based projects using `pip install .`. Do not use a
requirements.txt for this. The requirements.txt file is needed to handle pinned
versions extracted from the poetry.lock file.

Note that we cannot simply append `.` to the exported requirements.txt file.
Currently pip requires that either all requirements have a hash or none.
Including `.` would thus force us to omit hashes for all requirements.
If a poetry.lock file is present, use the Poetry CLI to export the
pinned requirements to a requirements.txt file. If the project already
contains a requirements.txt, use that and ignore poetry.lock.

Poetry is not used to install the project because it does not clean up
stale requirements. This means that requirements need to be exported
anyway, for the `pip-uninstall` step.

Since we only use Poetry to export a requirements.txt file, ignore the
Poetry version specified in pyproject.toml. Install a pre-release of
1.0.0 because the export command is not available before 1.0.0a0.

Note that supporting pyproject.toml-based installations is not enough to
handle pinned requirements: When pip installs a pyproject.toml-style
project using the process described in PEP 517, it only uses Poetry to
build a wheel. The wheel contains the version constraints from
pyproject.toml, not the pinned versions from poetry.lock.
@orokusaki
Copy link

Poetry 1.0.0b8 was released a few days back (fixes at least one bug)

@adamkgoldfarb
Copy link

Poetry 1.0 was officially released today, formalizing the export command.

@cedricbonhomme
Copy link

Could someone give us an idea of when this PR will be merged ?
Thank you.

@zyv
Copy link
Contributor

zyv commented Mar 5, 2020

A stop-gap solution: https://elements.heroku.com/buildpacks/moneymeets/python-poetry-buildpack

@yunti
Copy link

yunti commented Apr 7, 2020

What does heroku currently default to (prior to this PR being merged) if their is a pyproject.toml file and a requirement.txt file (that has previously been generated with poetry export ..)? Does it currently pick up and use that hashed and fully resolved requirement.txt?

@zyv
Copy link
Contributor

zyv commented Apr 7, 2020

@yunti this PR is nowhere near to be merged, so if both files are present currently Heroku defaults to using requirements.txt - that’s what we did before writing our own overlay buildpack.

@edmorley edmorley changed the base branch from master to main August 3, 2020 13:29
Copy link
Member

@edmorley edmorley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi! Thank you for the PR. We're in the process of moving to Cloud Native Buildpacks, so all development effort is focused there at the moment, and eventually they will replace the current buildpacks. As such, we're only making minimal changes to the current buildpacks for now.

For more information, see:
#834 (review)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support for Poetry
8 participants