Switch from pip to pipenv or Poetry.

[jenstroeger]

Currently the package uses pip and Python’s setuptools as described in Packaging and distributing projects. However, problems with recursive dependency checks, deep version collisions, or cumbersome pinning make working with pip somewhat unreliable.

Consider switching to pipenv or Poetry instead…

[jenstroeger]

Other packaging and env-management tools recommended during conversations with @hynek and @behnazh, are flit and asdf and pyenv. Also, note this comment for more references.

Another recommended read in this context is the post setup.py vs requirements.txt and Python Application Dependency Management.

[jenstroeger]

Consider also the discussion on PEP 665: A file format to list Python dependencies for reproducibility of an application (and ensuing discussion).

[behnazh]

The PEP 665 makes a lot of sense. It also mentions how npm has solved the reproducibility problem by using package.json.

But for the time being Poetry seems to be doing well enough in dependency management and reproducibility I think.

[jenstroeger]

Interesting conversation regarding flit on python.org.

[behnazh]

Interesting conversation regarding flit on python.org.

Interesting indeed. I didn't find any solid reasoning in the thread for Flit though. In particular, I tend to disagree with this argument:

my default is that any packaging-related project that wants to be part of PyPA should be welcomed. We shouldn’t need to justify including projects, rather we should require justification from people who want to say “no” to a request to join.

Welcoming any project without requiring justifications brings us to where we are now, which might be convenient for some developers but quite chaotic compared to npm.

[jenstroeger]

So, more conversations and I’m starting to agree with @hynek’s view: all we need is the pyproject.toml file to declare the dependencies within the [build-system] section, and if necessary pin/hash the rest.

There are more details in this Python Application Dependency Management blog post, the setuptools docs, and as an example we can take a look at structlog’s pyproject.toml — which is a package, not an application, though.

And in Python’s Python Packaging User Guide we have:

• To learn how to manage dependencies in a version controlled project, see the tutorial on managing application dependencies
• To learn how to package and distribute your projects, see the tutorial on packaging and distributing

[jenstroeger]

Here’s another choice by the Stripe folks, and a discussion to consider: Remove Poetry and reinstate setup.py #583

I’d like to know @ob-stripe’s and @koobs’ thoughts on that subject (and this issue) now, two years after that decision, and their experiences since.

[hynek]

I think you can test that yourself. Try installing pendulum (same author as Poetry, packaged w/ Poetry) into a Python 3.10 virtualenv.

[jenstroeger]

Hmm… https://github.com/regebro/pyroma 🤔

[jenstroeger]

Another interesting thread: Should I be pinning my dependencies?

TL;DR — if we’re building a package, don’t pin because the package needs to live amongst other packages and their dependencies; if we’re building an app, do pin and package accordingly (a somewhat neglected area of the Python ecosystem).

[behnazh]

Looking at the Remediation section in Scorecard security analysis results on this repo for pinning dependencies, its recommendation is consistent with the idea of pinning dependencies for apps and not pinning them for packages. It also references another interesting article, which gives practical examples of the consequences of pinning dependencies in packages.

I like to look at this problem from all these perspectives:

1. supply chain security
2. reproducibility
3. conflicts in dependency resolution

As also recommended by Scorecard, checking in lock files to the repo can be used as a signature of the state of dependencies, which is essential for the first two concerns. It can, however, still not be perfect for the third issue. However, using dependency update tools, such as Dependabot can mitigate this problem when dependencies are guaranteed to be the latest, which also pass the CI tests. Or am I missing something?

I personally use poetry these days and this is how I use it: I keep the un-pinned dependencies in pyproject.toml and also check in the poetry.lock to have a snapshot of my dependency graph for supply chain security and reproducibility reasons.

[jenstroeger]

Interesting comment:

I think that in general, packaging Python applications is not a well-supported scenario. The packaging ecosystem is mostly focused on packaging libraries. Yes, entry points allow your library to expose a command interface, but the packaging capabilities are still designed around libraries.

[jenstroeger]

Today I stumbled over the hashin package which allows to add integrity hashes for packages, much like Node’s package-lock.json file.

[jenstroeger]

This conversation is growing lengthy, so I wanted to pick it up again with a few thoughts based on what we’ve learned over the past months about packaging Python projects.

PR #74 adds Makefile support and commit 34b8953 adds an interesting feature to that Makefile support: make requirements creates a snapshot of the Python virtual environment (aka. pip freeze) including integrity hashes. Given that new Makefile support, here’s my proposal.

Packages

If a user builds a package then specify only direct package dependencies in setup.py using standard version range specifications, and install that package using e.g. pip.

Applications

If a user builds an application then specify only direct package dependencies in setup.py using standard version range specifications, just like for packages. In addition to that, use the new make requirements to pin and hash all recursive dependencies for the application, and install the application using that requirements file.

Note that this does not address pip’s weak point to reliably detect and resolve dependency conflicts, but that’s not the point of this issue anyway.

Also note that npm uses package-lock.json as well as npm-shrinkwrap.json which overrides the package lock; more specifically it says:

The recommended use-case for npm-shrinkwrap.json is applications deployed through the publishing process on the registry: for example, daemons and command-line tools intended as global installs or devDependencies. It's strongly discouraged for library authors to publish this file, since that would prevent end users from having control over transitive dependency updates.

Case in point 🤓

[jenstroeger]

Closing this because we decided that the current approach of using pip and setup.py and requirements.txt is sufficient. There are issues around deep package conflict detection and resolution but that’s not anything we can/should address with this template — users may choose to use Poetry or Pipenv.