pyproject.toml make dependencies list static for production

[david-nano]

In our pyproject.toml we have some packages with >=, for example:

[tool.poetry.dependencies]
python = "^3.8"
dynaconf = "^3.1.11"
internal-package=">=1.0.0"

To prepare the package for production we want to make it stable with what test during the CI, hence during the build we want to change the version of each package to be static, but in this case >= we want to use the latest one, in this example, internal-package latest version is 1.2.3, I want to have a new pyproject.toml which looks like:

[tool.poetry.dependencies]
python = "^3.8"
dynaconf = "^3.1.11"
internal-package = "1.2.3"

What do you think the right way is to do it?
I've looked on poetry export and then poetry add but got a lot of errors during the process. (For example one of them: cannot import name 'Pool' from 'poetry.repositories')

[abn]

Isn't this what the lockfile allows for? I am not sure what the value of doing this is.

You could do this.

poetry add internal-package==1.2.3

Regarding the import errors, you might have a broken install. Can you provide the output of running the command with -vvv?

[abn]

So, in effect, you want the pinned dependencies in your wheels.

I have to warn you that this is an anti-pattern in majority cases. And should only be done imho if the wheel is an application. Pushing this to an index would be bad form.

There was, a long time ago, discussions around this. That led to python-poetry/poetry-core#71. I do want to finish that at some point.

In the meantime, if the target environments are using pip, you can consider using https://pip.pypa.io/en/stable/user_guide/#constraints-files.

All that said, I would definitely urge you to consider what you really want to achieve. The better solution might be to give an upper bound in your pyproject for the dependency if in fact you don't want use with untested versions.

[david-nano]

I know it's bad practice, but for fast development, we need to keep >= for developers in the pyproject.toml but whenever we do a build in CI, make sure to publish it with the static version we test with.
Let's assume during the build the current version of numpy was 1.2.3 but when delivered it was 1.2.4, I want to make sure the build will keep the tested version. But for local development, keep >= so developers can keep using the latest versions and know when the new version causes issues.

[radoering]

I haven't looked into python-poetry/poetry-core#71, but wouldn't it be possible to keep main dependencies as is and create an extra called "pinned" that overwrites each dependency with an exact version? That way we avoid the anti-pattern and still provide a way to use the exact tested dependencies. (Of course, that would require us to be able to calculate the resulting marker for each dependency similar to poetry-plugin-export.)

[dimbleby]

at best this sort of thing would only lock your direct dependencies, indirect dependencies remain free to float according to whatever loose specifications your direct dependencies gave.

if what you care about is reproducible builds then pinning strict requirements in your wheels is the wrong solution. Use poetry.lock all along; or if you want to step outside of the poetry ecosystem then provide a constraints.txt and ask the consumers of your package to use that.

[abn]

@david-nano is the project only consumed internally? And if this is a common pattern within your org, a cleaner solution could be to control what's available in your production environments.

This can be done by creating a PyPI index proxy with package version filters. Both Sonatype Nexus and JFrog artifactory support this. Alternatively if you want right control over delivered artifacts, you could even just not proxy to PyPI. Build and publish trusted versions into a private index. Developers can retain use of full index while controlled environments can pull from a "trusted" and "controlled" source.

All of these have trade offs. Going the private index route is a bit heavy on the initial investment, but it is a tried and tested pattern.

And as pointed out above, just having top level dependencies locked might actually lead to more issues and not meet your intent.