Make targets build-debs[-notest] fails on new circular dependency between PyPI packages flit_core and tomli

[cfm]

Description

Since October 10, staging-test-with-rebase fails in make build-debs-notest. This is likely to bork make build-debs for the current production release cycle too.

Expected Behavior

https://app.circleci.com/pipelines/github/freedomofpress/securedrop/3075/workflows/6c173f25-a21a-40f6-a476-7b6f9c996b2b/jobs/57150

Actual Behavior

https://app.circleci.com/pipelines/github/freedomofpress/securedrop/3076/workflows/4a156c36-7049-4645-a6a8-c5583f658e99/jobs/57152

Comments

This looks like pypa/flit#451, for which the resolution may be to update pip ≥ 21.

[cfm]

To be exact: flit/flit_core 3.4.0 switched its TOML parser from toml, which has no dependencies, to tomli, which requires flit_core.

Dependency graph (excluding version constraints):

graph TD

setuptools-rust --> setuptools
setuptools-rust --> setuptools_scm["setuptools_scm[toml]"]
setuptools_scm --> setuptools
setuptools_scm --> tomli
tomli --> flit_core
flit_core --> tomli

stg-6137-translation-requirements is now testing whether pip ≥ 21 can handle this circular dependency. We'll then have to decide whether that upgrade is acceptable (pending a diff review) at this point in the release process.

Meanwhile, stg-6137-translation-requirements-2 is testing whether pinning flit_core<3.4 in our top-level translation-requirements.in will satisfy the version constraints of the setuptools-rust dependency graph.

[conorsch]

Summarizing synchronous discussion from today. We basically have two options here:

1. Stop building all python reqs from source, and fetch wheels from pypi. Doing so avoids the dependency conflicts in the build-deps described above, but involves a significant shift in our dependency management policies, so we consider it a last resort.
2. Update pip within the build environment from Focal's system version, 20.0.2-5ubuntu1.6, to PyPI version 21.3. This is also a slight departure, but it unbreaks the build, so we're happy with it for now.

Option 2, then, is our path forward. It's got some shortcomings in that we cannot pin the hash in all places, particularly in the debian/rules file used by dh-virtualenv. Punting on a larger refactor of the build logic until we're out of QA period. As part of the pip update, I will not be submitting a diff review, since pip is a foundational tool in the Python ecosystem, and re-reviewing that widely used dependency feels superfluous.

I'll aim to have a PR up by EOD, then toss to @cfm for review and backporting!