Build packages with pinned dependencies from .lock file

[artslob]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

As title says, this feature is about to poetry build build wheels and packages with pinned versions specified in poetry.lock file. Command can be something like poetry build --locked.

Its duplicate of closed #1307. Reason to close this issue was argument about libraries and applications (in short):

Libraries use wheels for distribution. Their dependencies should not be very tight so libraries can coexist with each other. Thats why poetry build dont use versions from lock file. There are also applications. Applications need pinned dependencies from lock file, so all environments (dev and prod) use same dependencies. Applications should use docker for distribution.

So I think its not actually true that applications should not use wheels and pypi for distribution, sometimes its reasonable to have package with pinned dependencies from lock file. Example of such use case can be some cli tools like https://github.com/ytdl-org/youtube-dl, so development team and end-user use same dependencies. Another example is some backend application that deployed on server by wheel package.

[tomzx]

I'd think it is reasonable for a tool such as poetry to generate Requires-Dist constraints in the wheel's METADATA, knowing that it is the user's responsibility to ensure that whatever they've built can be installed in the other projects that need them.

I assume that what you want as behavior is that each package listed in your pyproject.toml have the exact version defined in your poetry.lock. All the packages those packages depend on would not be part of the list of packages defined in the built wheel.

Is that an accurate assessment?

[artslob]

Well, no actually. What I want is that when package is builded it will require exact versions of all dependencies from lock file. These versions can be also extracted (vied) using poetry export -f requirements.

Right now when package is builded it requires versions specified in pyproject.toml file.

[Tobotimus]

I'm certainly a fan of this idea, and made a similar comment regarding that "Applications are not distributed via wheels" argument on the original issue.

[sinoroc]

@artslob @Tobotimus @tomzx A PR has been filed: #3341. Would you mind giving it a try, giving feedback on it, reviewing the code? Whatever helps to get things moving

[artslob]

I will try to find some time to check it

[strangemonad]

+1 to the suggestion @Tobotimus made ere and in #1307 (comment). This workflow perfectly supports apps that you might install, say via pipx

[bneijt]

I have created poetry-lock-package to do exactly this. Having a cluster that provisions on demand we could provisiin the right versions without manual scripting and hosting of a requirements file (already had a private repo).

I have created a feature request with the export plugin, but have not had any response there yet.

python-poetry/poetry-plugin-export#1

https://pypi.org/project/poetry-lock-package/

[aaronsteers]

Short of adding the --locked support to poetry build, I think a hard fail by way of forcing the build-packaged versions to match the current lock file (perhaps via --locking=strict or --versions=strict) could also help mitigate this issue.

So - either allowing the build artifacts to be dictated by the lock file (--locked, as proposed above) or failing hard if poetry lock or poetry update would change the presently-locked active versions. This alternative would allow us as developers to force-failure if we've not yet run our tests on the latest versions that would be bundled - perhaps with an error like...:

Could not build because the --versioning=strict option was specified and the "foobar" library would be updated from 1.0.2 to 1.0.3. Please update the version constraints in pyproject.toml or run poetry update or poetry lock and try again.

The reason I slightly prefer this "strict mode" personally is that I don't believe as a consumer of the library or package that I should need to inspect both pyproject.toml and poetry.lock. Excepting for the passage of time between builds, as a user of a library or app, I'd like to believe that pyproject.toml is the sole authority of what was packaged. This does not negate my fondness for the poetry.lock file at dev-time, but just to call out the competing values of developers versus consumers.

[stinodego]

This feature is something I currently need, because of a workflow where we build wheels and publish those to a DataBricks runtime, where we then install them. I would like to have all the dependencies locked down.

I wouldn't mind doing some work on this; implementing the functionality as described in this issue. Would that be welcomed?

[bneijt]

This feature is something I currently need, because of a workflow where we build wheels and publish those to a DataBricks runtime, where we then install them. I would like to have all the dependencies locked down.

I wouldn't mind doing some work on this; implementing the functionality as described in this issue. Would that be welcomed?

I've created https://pypi.org/project/poetry-lock-package/ to do this and I've been looking for support from the poetry team to incorporate this into poetry. I have not had any response yet.

My guess is the best place for this is in the export plugin, which has been moved outside of the poetry code to here: https://github.com/python-poetry/poetry-export-plugin
I've opened up an issue, but never had any response there either: python-poetry/poetry-plugin-export#1

If you consider opening a pull request, I think it's best to check discord first and then create a PR for the export plugin.

[stinodego]

I will definitely check out your work, @bneijt ! That was already on my list, actually. Still, I think it's a great feature to be incorporated into poetry, so I would definitely be happy to contribute.

EDIT: While the solution of @bneijt creates a second wheel, I'd propose keeping a single wheel, with adjusted metadata. So having this in the export plugin does not make sense to me.

[spoorn]

I created a poetry plugin that adds support for building wheel files via commandline using locked dependencies in poetry.lock, also adds support for data_files: https://github.com/spoorn/poeblix, https://pypi.org/project/poeblix/

[mic4ael]

Any update on this issue?

[gsemet]

I think we should try to work on a plugin or another project that will repackage the wheel file, if poetry team does not understand why it is an important feature

[neersighted]

This is something we are interested in implementing (it has been discussed extensively as 'locked wheels' among the team and with interested parties on Discord), and will require the 'lock file aware core' that @abn has been prototyping. However, there are simply lots of immediate problems to be solved, and limited bandwidth among the individuals capable of and motivated to seeing such an invasive refactor through.

[bneijt]

Could you elaborate on the extensive rework? I've been using this functionality as a separate one page script for more than a year now and I still think this could either be an extra part of the metadata of the project (as extra dependencies section with pinned/locked versions) or a feature in a plugin like the export plugin python-poetry/poetry-plugin-export#1

I'll try find the discord discussion to find out what the refactor is about, but if you could write a summary that would also be very helpful to document these concerns.

[sinoroc]

@bneijt I found these two (not sure how up-to-date they are):

• Make core aware of poetry lock files poetry-core#83
• Experimental: Add support for building pinned artifacts poetry-core#71

[neersighted]

A MVP of reading the lockfile and a maintainable design are not quite the same thing (especially as we have to figure out a way to deal with locked and unlocked builds and how projects opt in and out to this new behavior).

It's not insurmountable, but it's not something trivial to merge either.

[tharradine]

In addition to @spoorn's poeblix, it looks like cloud-custodian/poetry-plugin-freeze is another plugin that attempts to solve this issue.

[hanslovsky]

I believe this is the appropriate issue to report this use case that would require that I can pip install a python package that uses poetry as build system. In my case, I need to specify specific sources for some dependencies, e.g. cuml-cu12 is not a functional package on PyPI. Instead, I need to add nvidia's index, which I do via

# set up pip indices
[[tool.poetry.source]]
name = "nvidia-pypi"
url = "https://pypi.nvidia.com"
priority = "explicit"

cuml-cu12 = {version = "~23.10", source = "nvidia-pypi", optional = true}

When I install via

poetry lock
poetry install

I get the correct cuml-cu12 package from nvidia. When I use

pip install git+ssh://git@my-git-host/organization/repo.git

it installs https://pypi.org/project/cuml-cu12/ instead (the poetry.lock file is checked into the repo along with pyproject.toml).

Unfortunately, I cannot provide a minimal example because it is internal code but I will try to extract an MWE and share here.

[bneijt]

In reply to the above, this ticket is about embedding the information that poetry has in the poetry.lock file into the wheel/package to allow pip to take advantage of it without requiring the full poetry build system.

IMHO poetry is a development tool meant to produce reproducible builds for developers.

What I have been working with for a long time now is creating a separate package that contains all the lock file dependencies as python package dependencies, but it would be way cleaner if it was an option poetry could add to the package metadata as an optional dependency list instead.

@hanslovsky There are multiple options for your current issue:

• Create a lock package and push that to a package repository (for example with poetry-lock-package)
• Add a requirements file to your git repository with the dependencies (poetry export -f requirements.txt --output requirements.txt) and point pip to that file.
• Point poetry install --no-dev ... at the repository instead of pip
• Wait for this ticket to be closed, then install the package from poetry with the pip install your_package[pinned]

Hope any of those will help.

[DBS-ST-VIT]

A function like proposed here may had saved us from our production outage today.

May i clarify our use case: We are using poetry to develop a python3 based CLI tool, which is quite important for an application we are running. Of course, this tool is also having some external dependencies from pypi. An maintainer put something like >=41.0 in the requirements.txt of an library, that his project is dependening on. We all know, that this is a very bad practice, but we also know, that we cannot get rid of something, if we are using external dependencies.

So normally, this is not a problem, since the Lockfile of poetry is pinning those implicit dependencies and gives us (paired with our usage of renovate bot and Unit Tests in CI pipelines) full transparency, which dependency update may cause problems.

Due to the fact, that we are building the project using poetry build and installing it later somewhere else on some systems using pipx, the lockfile has basically no effect in production. Further, if we are installing the .whl-based artifact today, it may uses other dependencies than tomorrow, since some implicit dependencies are may be upgraded.

Saying that, we need to understand, that poetry build is currently not usable for reproducible/reliable builds, that shall be used in production environments (at least without the workarounds, that were already suggested by some other community members - many thanks for that!).

[gsemet]

I think pipx run with the —spec feature can really play a good place in this subject. Potentially will will have 2 versions of the same app running at the same time, so you need to have 2 virtualenv. And distributing through pipy is really a must have.

because distributing the wheel on pipy and a locked requirement on another way is really not manageable. I would like to have a locked « flavor » of my app distributed through pipy alongside my non-locked lib

[sinoroc]

There are plugins that might be able to help (I have not tested them myself):

• https://pypi.org/project/poetry-plugin-freeze/
• https://pypi.org/project/poeblix/
• https://pypi.org/project/poetry-stickywheel-plugin/

[camsteffen]

Here is a general flow for build and install that I landed on.

Build:

poetry export -o requirements.txt
poetry build --format=wheel

Install:

pip install -r requirements.txt
pip install --no-deps dist/*.whl
pip check

[gsemet]

You probably want to merge the pip install together, but the problem is still you cannot distribute the requirements.txt on pypi

[camsteffen]

You can merge them. I split them to optimize for docker build caching.

[sinoroc]

Are the development dependencies in the requirements.txt file? If yes, then maybe you might consider using something like python -m pip install --constraint requirements.txt --no-deps dist/*.whl instead.

Also note that, as far as I know, it is possible to do something like this: pip install --requirement http://example.com/requirements.txt (not as great as having things on PyPI, but maybe a compromise to be considered).

[camsteffen]

poetry export does not include dev dependencies by default.