Synapse's docker images (and CI?) do not pin the specific build requirements of poetry

[DMRobertson]

Synapse's docker file installs a specific version of poetry in build stage 0:

synapse/docker/Dockerfile

Lines 58 to 61 in 08297f2

	# We install poetry in its own build stage to avoid its dependencies conflicting with
	# synapse's dependencies.
	RUN --mount=type=cache,target=/root/.cache/pip \
	pip install --user "poetry==1.3.2"

and uses it to export a requirements.txt file for the other stages to use (without having to install poetry).

synapse/docker/Dockerfile

Lines 78 to 85 in 08297f2

	# Export the dependencies, but only if we're actually going to use the Poetry lockfile.
	# Otherwise, just create an empty requirements file so that the Dockerfile can
	# proceed.
	RUN if [ -z "$TEST_ONLY_IGNORE_POETRY_LOCKFILE" ]; then \
	/root/.local/bin/poetry export --extras all -o /synapse/requirements.txt ${TEST_ONLY_SKIP_DEP_HASH_VERIFICATION:+--without-hashes}; \
	else \
	touch /synapse/requirements.txt; \
	fi

The former step is not fully reproducible¹, because pip will use the latest version of poetry's build-time dependencies to install poetry. (As of poetry 1.3.2 those dependencies are the unbounded range poetry-core >= 1.1.0, see here). I advised the reporter to patch the Dockerfile in order to get Synapse 1.61 built again.

We saw this manifest in the report below: a version of poetry-core was released under which previous versions of poetry would fail to build, see #13849 (comment).

It seems like the Python packaging ecosystem doesn't have an obvious way to pin build-time dependencies. (See for instance https://discuss.python.org/t/pinning-build-dependencies/8363 or pypa/pip#9542.) The best approach that I can see is described in pypa/pip#9542 (comment), namely

• install the build-time dependencies into the environment you're interested in
• pip install --no-build-isolation ...

---

Original issue:

Unable to build v1.61.0 docker image with clean caches

Just tried to build v1.61.0 on Linux and Windows and both fail.

I know this worked a few weeks ago and I basically think that OP was right with the version pining.

Motivation: I want to bisect and thus build v1.61.0 .. v1.62.0 for identifying a patch which destroyed the generate call for us (I'll open a new ticket for that one).

Here is the error:

git checkout tags/v1.61.0
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile -t fred .
[+] Building 76.9s (14/25)
 => [internal] load build definition from Dockerfile                                                                                                                                                                                    0.1s
 => => transferring dockerfile: 5.28kB                                                                                                                                                                                                  0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                       0.1s
 => => transferring context: 170B                                                                                                                                                                                                       0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                                                                                                                              1.3s
 => [auth] docker/dockerfile:pull token for registry-1.docker.io                                                                                                                                                                        0.0s
 => docker-image://docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                                                                                1.0s
 => => resolve docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                                                                                    0.0s
 => => sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 2.00kB / 2.00kB                                                                                                                                          0.0s
 => => sha256:ad87fb03593d1b71f9a1cfc1406c4aafcb253b1dabebf569768d6e6166836f34 528B / 528B                                                                                                                                              0.0s
 => => sha256:1e8a16826fd1c80a63fa6817a9c7284c94e40cded14a9b0d0d3722356efa47bd 2.37kB / 2.37kB                                                                                                                                          0.0s
 => => sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546 9.94MB / 9.94MB                                                                                                                                          0.7s
 => => extracting sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546                                                                                                                                               0.2s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/python:3.9-slim                                                                                                                                                                      0.0s
 => [internal] load build context                                                                                                                                                                                                       0.3s
 => => transferring context: 7.41MB                                                                                                                                                                                                     0.2s
 => [stage-2 1/5] FROM docker.io/library/python:3.9-slim                                                                                                                                                                                0.2s
 => [builder 2/7] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked  apt-get update && apt-get install -y     build-essential     libffi-dev     libjpeg-dev     51.4s
 => [stage-2 2/5] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked   apt-get update && apt-get install -y     curl     gosu     libjpeg62-turbo     libpq5      70.1s
 => [requirements 2/6] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked  apt-get update && apt-get install -y git     && rm -rf /var/lib/apt/lists/*            60.6s
 => ERROR [requirements 3/6] RUN --mount=type=cache,target=/root/.cache/pip   pip install --user "poetry-core==1.1.0a7" "git+https://github.com/python-poetry/poetry.git@fb13b3a676f476177f7937ffa480ee5cff9a90a5"                     13.0s
------
 > [requirements 3/6] RUN --mount=type=cache,target=/root/.cache/pip   pip install --user "poetry-core==1.1.0a7" "git+https://github.com/python-poetry/poetry.git@fb13b3a676f476177f7937ffa480ee5cff9a90a5":
#13 3.856 Collecting git+https://github.com/python-poetry/poetry.git@fb13b3a676f476177f7937ffa480ee5cff9a90a5
#13 3.858   Cloning https://github.com/python-poetry/poetry.git (to revision fb13b3a676f476177f7937ffa480ee5cff9a90a5) to /tmp/pip-req-build-tufn8q4p
#13 3.866   Running command git clone --filter=blob:none --quiet https://github.com/python-poetry/poetry.git /tmp/pip-req-build-tufn8q4p
#13 7.242   Running command git rev-parse -q --verify 'sha^fb13b3a676f476177f7937ffa480ee5cff9a90a5'
#13 7.250   Running command git fetch -q https://github.com/python-poetry/poetry.git fb13b3a676f476177f7937ffa480ee5cff9a90a5
#13 7.839   Running command git checkout -q fb13b3a676f476177f7937ffa480ee5cff9a90a5
#13 9.253   Resolved https://github.com/python-poetry/poetry.git to commit fb13b3a676f476177f7937ffa480ee5cff9a90a5
#13 9.720   Installing build dependencies: started
#13 12.36   Installing build dependencies: finished with status 'done'
#13 12.37   Getting requirements to build wheel: started
#13 12.46   Getting requirements to build wheel: finished with status 'done'
#13 12.46   Preparing metadata (pyproject.toml): started
#13 12.77   Preparing metadata (pyproject.toml): finished with status 'error'
#13 12.77   error: subprocess-exited-with-error
#13 12.77
#13 12.77   × Preparing metadata (pyproject.toml) did not run successfully.
#13 12.77   │ exit code: 1
#13 12.77   ╰─> [26 lines of output]
#13 12.77       Traceback (most recent call last):
#13 12.77         File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
#13 12.77           main()
#13 12.77         File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
#13 12.77           json_out['return_val'] = hook(**hook_input['kwargs'])
#13 12.77         File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
#13 12.77           return hook(metadata_directory, config_settings)
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/masonry/api.py", line 40, in prepare_metadata_for_build_wheel
#13 12.77           poetry = Factory().create_poetry(Path(".").resolve(), with_groups=False)
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/factory.py", line 62, in create_poetry
#13 12.77           package = self.configure_package(
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/factory.py", line 156, in configure_package
#13 12.77           cls._add_package_group_dependencies(
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/factory.py", line 102, in _add_package_group_dependencies
#13 12.77           cls.create_dependency(
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/factory.py", line 366, in create_dependency
#13 12.77           dependency = Dependency(name, constraint, groups=groups)
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/packages/dependency.py", line 66, in __init__
#13 12.77           self.constraint = constraint  # type: ignore[assignment]
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/packages/dependency.py", line 110, in constraint
#13 12.77           self._constraint = parse_constraint(constraint)
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/semver/helpers.py", line 31, in parse_constraint
#13 12.77           constraint_objects.append(parse_single_constraint(constraint))
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/semver/helpers.py", line 147, in parse_single_constraint
#13 12.77           raise ParseConstraintError(f"Could not parse version constraint: {constraint}")
#13 12.77       poetry.core.semver.exceptions.ParseConstraintError: Could not parse version constraint: (>=20.4.3
#13 12.77       [end of output]
#13 12.77
#13 12.77   note: This error originates from a subprocess, and is likely not a problem with pip.
#13 12.78 error: metadata-generation-failed
#13 12.78
#13 12.78 × Encountered error while generating package metadata.
#13 12.78 ╰─> See above for output.
#13 12.78
#13 12.78 note: This is an issue with the package mentioned above, not pip.
#13 12.78 hint: See above for details.
#13 12.79 WARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.
#13 12.79 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
------
executor failed running [/bin/sh -c pip install --user "poetry-core==1.1.0a7" "git+https://github.com/python-poetry/poetry.git@fb13b3a676f476177f7937ffa480ee5cff9a90a5"]: exit code: 1

Originally posted by @qknight in #11537 (comment)

Footnotes

1. The steps which run apt-get update and run the abitrary sh.rustup.sh script aren't reproducible either, if anyone's counting. ↩

[DMRobertson]

I can reproduce this with DOCKER_BUILDKIT=1 docker build --no-cache -f docker/Dockerfile -t fred . on b8bf612

[DMRobertson]

Regressed in #12385 according to a bisection

[DMRobertson]

#13 12.77 poetry.core.semver.exceptions.ParseConstraintError: Could not parse version constraint: (>=20.4.3

I think the version requirement comes from https://github.com/python-poetry/poetry/blob/fb13b3a676f476177f7937ffa480ee5cff9a90a5/pyproject.toml#L54 in the specific version of poetry we use in that version of the lockfile. But what I don't understand is why this is suddenly a problem now, given that we built the images just fine at the time.

[DMRobertson]

But what I don't understand is why this is suddenly a problem now,

Seems to be python-poetry/poetry-core#461.

Related:

• fix virtualenv constraint python-poetry/poetry#6376
• fix virtualenv constraint (PEP 440 compliant) python-poetry/poetry#6402
• keyring and virtualenv are not using calver python-poetry/poetry#6505

Frustrating. Goes to show that we should have properly somehow frozen the environment for poetry itself (C.f. #12853). I hate Python packaging.

I suspect the easiest way to fix this will be to try to rebuild from v1.61 with a patched dockerfile. Let me get back to you on that.

[DMRobertson]

Running

pip install "poetry-core==1.1.0a7" "git+https://github.com/python-poetry/poetry.git@fb13b3a676f476177f7937ffa480ee5cff9a90a5" -v

in a clean virtualenv includes this line:

  Running command /home/dmr/workspace/dummy-python/.direnv/python-3.10.6/bin/python /tmp/pip-standalone-pip-ieecjtw7/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-dxlrb3q_/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'poetry-core>=1.0.0'

which I think comes from here. This pulls in poetry-core 1.2.0 to try and build the pinned version of poetry itself... but this is now broken as python-poetry/poetry#6376 noted.

We were using an unreleased poetry version so maybe this serves us right. But it's still incredibly frustrating.

[DMRobertson]

Given that we have no control over what version pip chooses when it installs the build-backend, I think the only option is to build using poetry 1.2.

diff --git a/docker/Dockerfile b/docker/Dockerfile
index 7af0e51f9..29010b436 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -55,7 +55,7 @@ RUN \
 # NB: In poetry 1.2 `poetry export` will be moved into a plugin; we'll need to also
 # pip install poetry-plugin-export (https://github.com/python-poetry/poetry-plugin-export).
 RUN --mount=type=cache,target=/root/.cache/pip \
-  pip install --user "poetry-core==1.1.0a7" "git+https://github.com/python-poetry/poetry.git@fb13b3a676f476177f7937ffa480ee5cff9a90a5"
+  pip install --user "poetry-core==1.2.0" "poetry==1.2.1"
 
 WORKDIR /synapse

This doesn't create a bitwise identical container, but I think it's the best we can do.

``` $ container-diff diff -t apt -t pip -t sizelayer -t metadata -t history matrixdotorg/synapse:v1.61.0 daemon://fred:latest ERRO[0011] failed to locate python package for corresponding package metadata matrix-synapse ERRO[0011] failed to locate python package for corresponding package metadata matrix-synapse

-----Apt-----

Packages found only in matrixdotorg/synapse:v1.61.0: None

Packages found only in fred:latest: None

Version differences:
PACKAGE IMAGE1 (matrixdotorg/synapse:v1.61.0) IMAGE2 (fred:latest)
-base-files 11.1 deb11u3, 340K 11.1 deb11u2, 340K
-curl 7.74.0-1.3 deb11u1, 426K 7.74.0-1.3 deb11u3, 429K
-dpkg 1.20.10, 6.8M 1.20.9, 6.7M
-gpgv 2.2.27-2 deb11u1, 882K 2.2.27-2, 882K
-gzip 1.10-4 deb11u1, 242K 1.10-4, 242K
-libc-bin 2.31-13 deb11u3, 3.6M 2.31-13 deb11u2, 3.6M
-libc6 2.31-13 deb11u3, 12.5M 2.31-13 deb11u2, 12.5M
-libcurl4 7.74.0-1.3 deb11u1, 732K 7.74.0-1.3 deb11u3, 739K
-libexpat1 2.2.10-2 deb11u3, 411K 2.2.10-2 deb11u2, 411K
-liblzma5 5.2.5-2.1~deb11u1, 277K 5.2.5-2, 277K
-libpq5 13.7-0 deb11u1, 788K 13.8-0 deb11u1, 786K
-libssl-dev 1.1.1n-0 deb11u2, 7.8M 1.1.1n-0 deb11u3, 7.8M
-libssl1.1 1.1.1n-0 deb11u2, 4M 1.1.1n-0 deb11u3, 4M
-libsystemd0 247.3-7, 865K 247.3-6, 864K
-libudev1 247.3-7, 282K 247.3-6, 281K
-libxslt1.1 1.1.34-4, 498K 1.1.34-4 deb11u1, 502K
-openssl 1.1.1n-0 deb11u2, 1.4M 1.1.1n-0 deb11u3, 1.4M
-publicsuffix 20211207.1025-0 deb11u1, 330K 20220811.1734-0 deb11u1, 336K
-sysvinit-utils 2.96-7 deb11u1, 79K 2.96-7, 79K
-tzdata 2021a-1 deb11u3, 3.3M 2021a-1 deb11u2, 3.3M
-zlib1g 1:1.2.11.dfsg-2 deb11u1, 166K 1:1.2.11.dfsg-2, 166K

-----History-----

Docker history lines found only in matrixdotorg/synapse:v1.61.0:
-/bin/sh -c #(nop) ADD file:134f25aec8adf83cb940ba073a3409ca85dbb5ae592b704f95193e7d2539a3bc in /
-/bin/sh -c #(nop) ENV PYTHON_VERSION=3.9.13
-/bin/sh -c set -eux; savedAptMark="$(apt-mark showmanual)"; apt-get update; apt-get install -y --no-install-recommends dpkg-dev gcc gnupg dirmngr libbluetooth-dev libbz2-dev libc6-dev libexpat1-dev libffi-dev libgdbm-dev liblzma-dev libncursesw5-dev libreadline-dev libsqlite3-dev libssl-dev make tk-dev uuid-dev wget xz-utils zlib1g-dev ; wget -O python.tar.xz "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz"; wget -O python.tar.xz.asc "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc"; GNUPGHOME="$(mktemp -d)"; export GNUPGHOME; gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys "$GPG_KEY"; gpg --batch --verify python.tar.xz.asc python.tar.xz; command -v gpgconf > /dev/null && gpgconf --kill all || :; rm -rf "$GNUPGHOME" python.tar.xz.asc; mkdir -p /usr/src/python; tar --extract --directory /usr/src/python --strip-components=1 --file python.tar.xz; rm python.tar.xz; cd /usr/src/python; gnuArch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)"; ./configure --build="$gnuArch" --enable-loadable-sqlite-extensions --enable-optimizations --enable-option-checking=fatal --enable-shared --with-system-expat --without-ensurepip ; nproc="$(nproc)"; make -j "$nproc" LDFLAGS="-Wl,--strip-all" ; make install; cd /; rm -rf /usr/src/python; find /usr/local -depth ( ( -type d -a ( -name test -o -name tests -o -name idle_test ) ) -o ( -type f -a ( -name '.pyc' -o -name '.pyo' -o -name 'libpython*.a' ) ) ) -exec rm -rf '{}' + ; ldconfig; apt-mark auto '.' > /dev/null; apt-mark manual $savedAptMark; find /usr/local -type f -executable -not ( -name 'tkinter' ) -exec ldd '{}' ';' | awk '/=>/ { print $(NF-1) }' | sort -u | xargs -r dpkg-query --search | cut -d: -f1 | sort -u | xargs -r apt-mark manual ; apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; rm -rf /var/lib/apt/lists/; python3 --version
-/bin/sh -c set -eux; for src in idle3 pydoc3 python3 python3-config; do dst="$(echo "$src" | tr -d 3)"; [ -s "/usr/local/bin/$src" ]; [ ! -e "/usr/local/bin/$dst" ]; ln -svT "$src" "/usr/local/bin/$dst"; done
-/bin/sh -c #(nop) ENV PYTHON_PIP_VERSION=22.0.4
-/bin/sh -c #(nop) ENV PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/6ce3639da143c5d79b44f94b04080abf2531fd6e/public/get-pip.py
-/bin/sh -c #(nop) ENV PYTHON_GET_PIP_SHA256=ba3ab8267d91fd41c58dbce08f76db99f747f716d85ce1865813842bb035524d

Docker history lines found only in fred:latest:
-/bin/sh -c #(nop) ADD file:d48a85028743f16ed927507322e3e3beee187fbfadd0b781d4b89de197c64b5b in /
-/bin/sh -c #(nop) ENV PYTHON_VERSION=3.9.10
-/bin/sh -c set -eux; savedAptMark="$(apt-mark showmanual)"; apt-get update; apt-get install -y --no-install-recommends dpkg-dev gcc gnupg dirmngr libbluetooth-dev libbz2-dev libc6-dev libexpat1-dev libffi-dev libgdbm-dev liblzma-dev libncursesw5-dev libreadline-dev libsqlite3-dev libssl-dev make tk-dev uuid-dev wget xz-utils zlib1g-dev ; wget -O python.tar.xz "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz"; wget -O python.tar.xz.asc "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc"; GNUPGHOME="$(mktemp -d)"; export GNUPGHOME; gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys "$GPG_KEY"; gpg --batch --verify python.tar.xz.asc python.tar.xz; command -v gpgconf > /dev/null && gpgconf --kill all || :; rm -rf "$GNUPGHOME" python.tar.xz.asc; mkdir -p /usr/src/python; tar --extract --directory /usr/src/python --strip-components=1 --file python.tar.xz; rm python.tar.xz; cd /usr/src/python; gnuArch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)"; ./configure --build="$gnuArch" --enable-loadable-sqlite-extensions --enable-optimizations --enable-option-checking=fatal --enable-shared --with-system-expat --with-system-ffi --without-ensurepip ; nproc="$(nproc)"; make -j "$nproc" LDFLAGS="-Wl,--strip-all" ; make install; cd /; rm -rf /usr/src/python; find /usr/local -depth ( ( -type d -a ( -name test -o -name tests -o -name idle_test ) ) -o ( -type f -a ( -name '.pyc' -o -name '.pyo' -o -name '.a' ) ) ) -exec rm -rf '{}' + ; ldconfig; apt-mark auto '.' > /dev/null; apt-mark manual $savedAptMark; find /usr/local -type f -executable -not ( -name 'tkinter' ) -exec ldd '{}' ';' | awk '/=>/ { print $(NF-1) }' | sort -u | xargs -r dpkg-query --search | cut -d: -f1 | sort -u | xargs -r apt-mark manual ; apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; rm -rf /var/lib/apt/lists/*; python3 --version
-/bin/sh -c set -eux; for src in idle3 pydoc3 python3 python3-config; do dst="$(echo "$src" | tr -d 3)"; [ -s "/usr/local/bin/$src" ]; [ ! -e "/usr/local/bin/$dst" ]; ln -svT "/usr/local/bin/$src" "/usr/local/bin/$dst"; done
-/bin/sh -c #(nop) ENV PYTHON_PIP_VERSION=21.2.4
-/bin/sh -c #(nop) ENV PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/38e54e5de07c66e875c11a1ebbdb938854625dd8/public/get-pip.py
-/bin/sh -c #(nop) ENV PYTHON_GET_PIP_SHA256=e235c437e5c7d7524fbce3880ca39b917a73dc565e0c813465b7a7a329bb279a

-----Metadata-----

Image metadata differences between matrixdotorg/synapse:v1.61.0 and fred:latest:

matrixdotorg/synapse:v1.61.0
-Env: PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,LANG=C.UTF-8,GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568,PYTHON_VERSION=3.9.10,PYTHON_PIP_VERSION=21.2.4,PYTHON_SETUPTOOLS_VERSION=58.1.0,PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/38e54e5de07c66e875c11a1ebbdb938854625dd8/public/get-pip.py,PYTHON_GET_PIP_SHA256=e235c437e5c7d7524fbce3880ca39b917a73dc565e0c813465b7a7a329bb279a
-Labels: org.opencontainers.image.documentation:https://github.com/matrix-org/synapse/blob/master/docker/README.md org.opencontainers.image.licenses:Apache-2.0 org.opencontainers.image.source:https://github.com/matrix-org/synapse.git org.opencontainers.image.url:https://matrix.org/docs/projects/server/synapse

fred:latest
-Env: PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,LANG=C.UTF-8,GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568,PYTHON_VERSION=3.9.13,PYTHON_PIP_VERSION=22.0.4,PYTHON_SETUPTOOLS_VERSION=58.1.0,PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/6ce3639da143c5d79b44f94b04080abf2531fd6e/public/get-pip.py,PYTHON_GET_PIP_SHA256=ba3ab8267d91fd41c58dbce08f76db99f747f716d85ce1865813842bb035524d
-Labels: gitsha1:b8bf61230c0d51231429b2d15973a8fd1cd76906 org.opencontainers.image.documentation:https://github.com/matrix-org/synapse/blob/master/docker/README.md org.opencontainers.image.licenses:Apache-2.0 org.opencontainers.image.source:https://github.com/matrix-org/synapse.git org.opencontainers.image.url:https://matrix.org/docs/projects/server/synapse

-----Pip-----

Packages found only in matrixdotorg/synapse:v1.61.0: None

Packages found only in fred:latest: None

Version differences:
PACKAGE IMAGE1 (matrixdotorg/synapse:v1.61.0) IMAGE2 (fred:latest)
-pip 22.0.4, 6.9M 21.2.4, 5.1M

-----SizeLayer-----

Layer size differences between matrixdotorg/synapse:v1.61.0 and fred:latest:
LAYER SIZE1 SIZE2
0 326M 323.7M
1 326M 323.7M
2 326M 323.7M
3 326M 323.7M
4 326M 323.7M
5 326M 323.7M
6 326M 323.7M
7 326M 323.7M
8 326M 323.7M

</details>

[reivilibre]

If we wanted to pin this for the future, we could manually do e.g. pip install poetry==1.2.1 and then pip freeze > poetry_requirements.txt.
In the Dockerfile, we could pip install -r poetry_requirements.txt instead of installing a poetry version with floating versions of (transitive) dependencies.
That would give us some pinned deps for Poetry to use in our Dockerfile; sort of like bootstrapping. That won't help with existing Dockerfiles but it would solve this going forward.

[DMRobertson]

If we wanted to pin this for the future, we could manually do e.g. pip install poetry==1.2.1 and then pip freeze > poetry_requirements.txt. In the Dockerfile, we could pip install -r poetry_requirements.txt

Agreed. I was worried about managing poetry's dependencies itself during #11537 but clearly dropped the ball on this one. :(

However: I'm not sure if pip will use the current virtualenv's contents when fetching a build-backend to install poetry. (That is why we pull in poetry-core 1.2.x, and then we hit python-poetry/poetry#6376, which "made poetry uninstallable via pip" according to python-poetry/poetry#6402).

[dklimpel]

Is this issue still up to date?

[DMRobertson]

It is still the case that the dockerfile does not pin poetry's dependencies, yes:

synapse/docker/Dockerfile

Lines 60 to 61 in d4eba44

	RUN --mount=type=cache,target=/root/.cache/pip \
	pip install --user "poetry==1.3.2"

[richvdh]

It is still the case that the dockerfile does not pin poetry's dependencies, yes:

@DMRobertson could you update the issue summary, then? Currently it reads as if there is a specific problem with v1.61, which seems like a difficult thing to care about.

[DMRobertson]

I have updated the description; I hope it is comprehensible.

However on reflection I think we should probably ignore (close?) this:

• I proposed a patch to the dockerfile as a workaround for the reporter
• We don't have the time or resources to make Python's packaging ecosystem bulletproof
• We ourselves are still going to be running pipx install poetry==a.b.c without using any kind of build-system pinning.

Let's discuss among the team.

[DMRobertson]

Closing as WONTFIX.