complement.sh: don't use remote base synapse image#18210
Closed
AndrewFerr wants to merge 9 commits intoelement-hq:developfrom
Closed
complement.sh: don't use remote base synapse image#18210AndrewFerr wants to merge 9 commits intoelement-hq:developfrom
AndrewFerr wants to merge 9 commits intoelement-hq:developfrom
Conversation
Ensure that the synapse-workers & complement-synapse images are built from the local base synapse image that was just built, instead of from the latest synapse image pulled from a public registry.
as there is no risk of pulling these from a public registry
AndrewFerr
commented
Mar 4, 2025
scripts-dev/complement.sh
Outdated
| SYNAPSE_TAG=matrixdotorg/synapse | ||
| $CONTAINER_RUNTIME build \ | ||
| -t "$SYNAPSE_TAG" \ | ||
| -t "$LOCAL_REGISTRY/$SYNAPSE_TAG" \ |
Member
Author
There was a problem hiding this comment.
Alternatively, this could have added a custom tag to the image name (like :testing, instead of the implicit default of :latest), but using a dummy registry prevents accidentally pushing the built image to anywhere public.
because the tag is really just the part of a name after the colon
A tag of "latest" is included by default
3 tasks
`localhost:5000` may be used for other development images, so use a registry name that is less likely to conflict with an existing one. Also remove the port as having one makes it seem significant, but it really isn't.
AndrewFerr
commented
Mar 20, 2025
Comment on lines
+185
to
+187
| # Tag local builds with a dummy registry so that later builds | ||
| # may reference them instead of pulling from a remote registry | ||
| LOCAL_REGISTRY=synapse-registry.localhost |
Member
Author
There was a problem hiding this comment.
Note that the reason for doing this is because FROM <image> in a Dockerfile seems to always prefer remote images over local ones, at least in my dev environment.
Contributor
There was a problem hiding this comment.
What build tool are you using?
I think the default for Docker is to prefer local and then look for a remote image although I don't see any docs describing this behavior.
There is even a docker build flag to always pull from remote (which implies as opposed to using the local if available):
--pull: Always attempt to pull all referenced images-- https://docs.docker.com/reference/cli/docker/buildx/build/#options
Member
Author
There was a problem hiding this comment.
I'm using plain upstream Docker with buildx:
~$ docker system info
Client: Docker Engine - Community
Version: 28.1.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.23.0
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.35.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
<snip>
Server Version: 28.1.1
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: local
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
runc version: v1.2.5-0-g59923ef
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.14.6-300.fc42.x86_64
Operating System: Fedora Linux 42 (Workstation Edition)
OSType: linux
Architecture: x86_64
CPUs: 20
Total Memory: 31.01GiB
Name: <REDACTED>
ID: <REDACTED>
Docker Root Dir: /var/lib/docker
Debug Mode: false
Username: <REDACTED>
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: false
Contributor
There was a problem hiding this comment.
Here are my details for reference:
docker system info
$ docker system info
Client:
Version: 28.0.4
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: 0.22.0
Path: /usr/lib/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: 2.34.0
Path: /usr/lib/docker/cli-plugins/docker-compose
Server:
Containers: 4
Running: 4
Paused: 0
Stopped: 0
Images: 51
Server Version: 28.0.4
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: true
Native Overlay Diff: false
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 1a43cb6a1035441f9aca8f5666a9b3ef9e70ab20.m
runc version:
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.6.84-1-MANJARO
Operating System: Manjaro Linux
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 62.02GiB
Name: <REDACTED>
ID: <REDACTED>
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: false
Member
Author
|
This PR is because I hit issues with the Complement script's attempt to build the As it turns out, this was because my Docker build driver was something other than Reinstalling Docker from upstream repos (as opposed to Fedora's) lets the workers image build as intended, meaning this PR is only needed for compatibility with other build drivers. |
| @@ -0,0 +1 @@ | |||
| In scripts-dev/complement.sh, ensure that the synapse-workers & complement-synapse images are built from the local base synapse image that was just built, instead of from the latest synapse image pulled from a public registry. | |||
Contributor
There was a problem hiding this comment.
I'm confused how the complement.sh script was running for you without building the images locally.
I see that it's possible to skip_docker_build in complement.sh if you use --fast or --editable but otherwise should be always building the image.
Member
Author
There was a problem hiding this comment.
I'm confused how the
complement.shscript was running for you without building the images locally.
It works only when I make no local changes to the base Dockerfile, in which case the build will pull the Synapse image from docker.io & build the workers image on top of that. (There is no problem with building the complement image on top of the workers image, since the latter is not on docker.io, so the build falls back to using the local just-built workers image.)
Contributor
There was a problem hiding this comment.
@AndrewFerr Seems to work fine for me.
- Edit
docker/Dockerfilewith some cache breaking command likeRUN echo "asdf1" -
$ COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestJumpToDateEndpoint ... => => writing image sha256:2ebd0b1d108b710dc9a28dacb5cea15cf2917d7edc675cf1593ec7a8a03b77b0 => => naming to docker.io/matrixdotorg/synapse ... => => writing image sha256:1e4677f59072dc8ddc706692a1e3e6138b8f5589c3eda87d352c76f796d2d505 => => naming to docker.io/matrixdotorg/synapse-workers ... => => writing image sha256:c2b38b637c9a71cf781052c75a828dcc31c70f28f0997930ed74713911cf9bb1 => => naming to docker.io/library/complement-synapse => => naming to ghcr.io/element-hq/synapse/complement-synapse ... -
$ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE complement-synapse latest c2b38b637c9a About a minute ago 472MB ghcr.io/element-hq/synapse/complement-synapse latest c2b38b637c9a About a minute ago 472MB matrixdotorg/synapse-workers latest 1e4677f59072 2 minutes ago 350MB matrixdotorg/synapse latest 2ebd0b1d108b 2 minutes ago 326MB
I can't find a good way to know for sure whether these images are using each other in their FROM declarations but given they are rebuilding, I'm assuming they are using each other.
Can you explain further how you are reproducing?
Member
Author
There was a problem hiding this comment.
Apologies for having missed this. Now that I'm hitting this again, it's a good time to finally follow up.
An easy way to reproduce is to check out a release tag before running complement.sh:
- first it builds the base image & tags it as
docker.io/matrixdotorg/synapse:latest - then for the workers image, it tries pulling
docker.io/matrixdotorg/synapse:latestremotely - the pulled
synapse:latest!= the locally-builtsynapse:latest, as the former will be for a version more recent than whatever release tag was checked out
Merging this PR branch before running the script fixes the issue for me.
Contributor
There was a problem hiding this comment.
Can you give some concrete steps please?
I'm not able to reproduce with this:
git checkout v1.113.0(a release from a year ago)COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestSendMessageWithTxn- Complement builds and runs just fine
Member
Author
There was a problem hiding this comment.
This just bit me again, so I will give another crack at writing valid repro steps:
git checkout develop- Edit
synapse/__init__.pyto throw an exception on startup COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh
Expected result: The Complement image build should succeed, but all Complement tests should fail because of the added exception.
The (bad) result I get: Complement tests pass despite the added exception, because the Complement image was built on the upstream matrixdotorg/synapse:latest image instead of the locally-built one that has the exception.
Contributor
There was a problem hiding this comment.
Using your exact steps, the tests fail as expected for me.
COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh
(I ran a subset of tests so this would run faster)
Notice Exception: asdf fake error in the output because of the synapse/__init__.py modification to throw an exception on startup.
COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestKnocking
[+] Building 29.8s (34/34) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 7.54kB 0.0s
=> resolve image config for docker-image://docker.io/docker/dockerfile:1 0.5s
=> CACHED docker-image://docker.io/docker/dockerfile:1@sha256:b6afd42430b15f2d2a4c5a02b919e98a525b785b1aaff16747d2f623364e39b6 0.0s
=> [internal] load metadata for docker.io/library/python:3.13-slim-trixie 0.9s
=> [internal] load metadata for docker.io/library/debian:trixie 0.7s
=> [internal] load metadata for ghcr.io/astral-sh/uv:python3.13-trixie 0.4s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 241B 0.0s
=> [stage-3 1/7] FROM docker.io/library/python:3.13-slim-trixie@sha256:452ed869040b0d92371f1d444da1997de6cd468b74c23149c0062c1c8b9ffbb3 3.1s
=> => resolve docker.io/library/python:3.13-slim-trixie@sha256:452ed869040b0d92371f1d444da1997de6cd468b74c23149c0062c1c8b9ffbb3 0.1s
=> => sha256:452ed869040b0d92371f1d444da1997de6cd468b74c23149c0062c1c8b9ffbb3 10.37kB / 10.37kB 0.0s
=> => sha256:8eefb09a1cc010be02c0010084e605fd941f73402329a35a521b205e5650d270 1.75kB / 1.75kB 0.0s
=> => sha256:c706381c8c539c38051fd75ded4572d74b7f264dfc6a719d0a63e6a2374c6ad0 5.53kB / 5.53kB 0.0s
=> => sha256:d7ecded7702a5dbf6d0f79a71edc34b534d08f3051980e2c948fba72db3197fc 29.78MB / 29.78MB 1.1s
=> => sha256:ad0b8045daedbe4696e98a0ae904667b5f831981ca90fcca4ccd35fad1f0f8ad 1.29MB / 1.29MB 1.1s
=> => sha256:964deb1231a98bfaa82551ea9ab166390e9c43b027cd86bb22f8228513be8d18 11.73MB / 11.73MB 1.1s
=> => extracting sha256:d7ecded7702a5dbf6d0f79a71edc34b534d08f3051980e2c948fba72db3197fc 0.6s
=> => sha256:da5a8471b1b2231540185a3bd2d17929640b173b8b386380e22179ac08eb9f06 249B / 249B 1.4s
=> => extracting sha256:ad0b8045daedbe4696e98a0ae904667b5f831981ca90fcca4ccd35fad1f0f8ad 0.1s
=> => extracting sha256:964deb1231a98bfaa82551ea9ab166390e9c43b027cd86bb22f8228513be8d18 0.3s
=> => extracting sha256:da5a8471b1b2231540185a3bd2d17929640b173b8b386380e22179ac08eb9f06 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 10.46MB 0.1s
=> [requirements 1/4] FROM ghcr.io/astral-sh/uv:python3.13-trixie@sha256:bf199483122c9d880fcde04cf8cb7408473c5403ff0a902e11e70e122112418a 0.0s
=> [runtime-deps 1/6] FROM docker.io/library/debian:trixie@sha256:01a723bf5bfb21b9dda0c9a33e0538106e4d02cce8f557e118dd61259553d598 4.4s
=> => resolve docker.io/library/debian:trixie@sha256:01a723bf5bfb21b9dda0c9a33e0538106e4d02cce8f557e118dd61259553d598 0.1s
=> => sha256:3a615c1937a785880ecac9e086fe3f46687d9ba4884d201e03a940e39ef96b1e 451B / 451B 0.0s
=> => sha256:01a723bf5bfb21b9dda0c9a33e0538106e4d02cce8f557e118dd61259553d598 8.93kB / 8.93kB 0.0s
=> => sha256:75a32e5b9e4aecc3454bab811b66fb98c8a44679915cf93f7d47370f6a7fb643 1.02kB / 1.02kB 0.0s
=> => sha256:13cc39f8244ac66bf1dd9149e1da421ab1bbc80d612dc14fe368753e7be17b33 49.29MB / 49.29MB 3.1s
=> => extracting sha256:13cc39f8244ac66bf1dd9149e1da421ab1bbc80d612dc14fe368753e7be17b33 1.0s
=> CACHED [requirements 2/4] WORKDIR /synapse 0.0s
=> [requirements 3/4] COPY pyproject.toml poetry.lock /synapse/ 0.3s
=> [requirements 4/4] RUN --mount=type=cache,target=/root/.cache/uv if [ -z "$TEST_ONLY_IGNORE_POETRY_LOCKFILE" ]; then uvx --with poetry-plug 3.6s
=> CACHED [builder 2/9] RUN mkdir /rust /cargo 0.0s
=> CACHED [builder 3/9] RUN curl -sSf https://sh.rustup.rs | sh -s -- -y --no-modify-path --default-toolchain stable --profile minimal 0.0s
=> [builder 4/9] COPY --from=requirements /synapse/requirements.txt /synapse/ 0.1s
=> [builder 5/9] RUN --mount=type=cache,target=/root/.cache/uv uv pip install --prefix="/install" --no-deps -r /synapse/requirements.txt 6.1s
=> [runtime-deps 2/6] RUN rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/ke 0.4s
=> [runtime-deps 3/6] RUN dpkg --add-architecture arm64 0.4s
=> [runtime-deps 4/6] RUN dpkg --add-architecture amd64 0.5s
=> [runtime-deps 5/6] RUN --mount=type=cache,target=/var/cache/apt,sharing=locked --mount=type=cache,target=/var/lib/apt,sharing=locked apt-ge 5.0s
=> [builder 6/9] COPY synapse /synapse/synapse/ 0.2s
=> [builder 7/9] COPY rust /synapse/rust/ 0.1s
=> [builder 8/9] COPY pyproject.toml README.rst build_rust.py Cargo.toml Cargo.lock /synapse/ 0.3s
=> [runtime-deps 6/6] RUN for arch in arm64 amd64; do mkdir -p /install-${arch}/var/lib/dpkg/status.d/ && for deb in /tmp/debs-${arch}/*.d 2.3s
=> [builder 9/9] RUN --mount=type=cache,target=/root/.cache/uv --mount=type=cache,target=/synapse/target,sharing=locked --mount=type=cache,ta 11.9s
=> [stage-3 2/7] COPY --from=runtime-deps /install-amd64/etc /etc 0.1s
=> [stage-3 3/7] COPY --from=runtime-deps /install-amd64/usr /usr 0.3s
=> [stage-3 4/7] COPY --from=runtime-deps /install-amd64/var /var 0.2s
=> [stage-3 5/7] COPY --from=builder --exclude=.lock /install /usr/local 1.0s
=> [stage-3 6/7] COPY ./docker/start.py /start.py 0.1s
=> [stage-3 7/7] COPY ./docker/conf /conf 0.1s
=> exporting to image 1.0s
=> => exporting layers 0.9s
=> => writing image sha256:5786e475e38e6bc096be4c4e28dd99fcb41d14f197e2cac88ce8ac73b4e02e70 0.0s
=> => naming to docker.io/matrixdotorg/synapse 0.0s
[+] Building 5.4s (28/28) FINISHED docker:default
=> [internal] load build definition from Dockerfile-workers 0.1s
=> => transferring dockerfile: 3.76kB 0.0s
=> resolve image config for docker-image://docker.io/docker/dockerfile:1-labs 0.1s
=> CACHED docker-image://docker.io/docker/dockerfile:1-labs@sha256:dce1c693ef318bca08c964ba3122ae6248e45a1b96d65c4563c8dc6fe80349a2 0.0s
=> [internal] load metadata for ghcr.io/astral-sh/uv:python3.13-trixie 0.1s
=> [internal] load metadata for docker.io/matrixdotorg/synapse:latest 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 241B 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 53.88kB 0.0s
=> [stage-1 1/13] FROM docker.io/matrixdotorg/synapse:latest 0.4s
=> [deps_base 1/7] FROM ghcr.io/astral-sh/uv:python3.13-trixie@sha256:bf199483122c9d880fcde04cf8cb7408473c5403ff0a902e11e70e122112418a 0.0s
=> CACHED [deps_base 2/7] RUN rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf. 0.0s
=> CACHED [deps_base 3/7] RUN curl -fsSL https://packages.redis.io/gpg | gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg && 0.0s
=> CACHED [deps_base 4/7] RUN --mount=type=cache,target=/var/cache/apt,sharing=locked --mount=type=cache,target=/var/lib/apt,sharing=l 0.0s
=> CACHED [deps_base 5/7] RUN rm /etc/nginx/sites-enabled/default && ln -sf /dev/stdout /var/log/nginx/access.log && ln -sf /dev/s 0.0s
=> CACHED [deps_base 6/7] RUN --mount=type=cache,target=/root/.cache/uv uv pip install --link-mode=copy --prefix="/uv/usr/local" supervisor~=4 0.0s
=> CACHED [deps_base 7/7] RUN mkdir -p /uv/etc/supervisor/conf.d 0.0s
=> [stage-1 2/13] COPY --from=deps_base --parents /usr/lib/*-linux-gnu/libicu* / 1.0s
=> [stage-1 3/13] COPY --from=deps_base /usr/bin/redis-server /usr/local/bin 0.2s
=> [stage-1 4/13] COPY --from=deps_base /uv / 0.2s
=> [stage-1 5/13] COPY --from=deps_base /usr/sbin/nginx /usr/sbin 0.2s
=> [stage-1 6/13] COPY --from=deps_base /usr/share/nginx /usr/share/nginx 0.2s
=> [stage-1 7/13] COPY --from=deps_base /usr/lib/nginx /usr/lib/nginx 0.2s
=> [stage-1 8/13] COPY --from=deps_base /etc/nginx /etc/nginx 0.2s
=> [stage-1 9/13] COPY --from=deps_base /var/log/nginx /var/log/nginx 0.1s
=> [stage-1 10/13] COPY --from=deps_base --chown=www-data:root /var/lib/nginx /var/lib/nginx 0.2s
=> [stage-1 11/13] COPY ./docker/conf-workers/* /conf/ 0.1s
=> [stage-1 12/13] COPY ./docker/prefix-log /usr/local/bin/ 0.1s
=> [stage-1 13/13] COPY ./docker/configure_workers_and_start.py /configure_workers_and_start.py 0.1s
=> exporting to image 1.3s
=> => exporting layers 1.2s
=> => writing image sha256:88682dfbc20a1abce15ac4f5f6438d7157e39214ddc994adbd15aeb3031249dc 0.0s
=> => naming to docker.io/matrixdotorg/synapse-workers 0.0s
[+] Building 24.5s (22/22) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 2.77kB 0.0s
=> resolve image config for docker-image://docker.io/docker/dockerfile:1 0.1s
=> CACHED docker-image://docker.io/docker/dockerfile:1@sha256:b6afd42430b15f2d2a4c5a02b919e98a525b785b1aaff16747d2f623364e39b6 0.0s
=> [internal] load metadata for docker.io/matrixdotorg/synapse-workers:latest 0.0s
=> [internal] load metadata for docker.io/library/postgres:13-trixie 1.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [stage-1 1/13] FROM docker.io/matrixdotorg/synapse-workers:latest 1.2s
=> [postgres_base 1/1] FROM docker.io/library/postgres:13-trixie@sha256:1732a9c549b6ed8cf7185c7f39fd0487c3aa768bf6648f89e1d9b0b9b0e6f43d 7.4s
=> => resolve docker.io/library/postgres:13-trixie@sha256:1732a9c549b6ed8cf7185c7f39fd0487c3aa768bf6648f89e1d9b0b9b0e6f43d 0.1s
=> => sha256:5bacde1ae2d4533b7222789835dbdbe5428bfdbf47f1c0db05c045ab93060da0 1.17kB / 1.17kB 0.2s
=> => sha256:1732a9c549b6ed8cf7185c7f39fd0487c3aa768bf6648f89e1d9b0b9b0e6f43d 10.24kB / 10.24kB 0.0s
=> => sha256:63886a08b044f0cfcffcbd0a2d5fd1853b81e7f2ce46b00d4d145b53ab8f8424 3.63kB / 3.63kB 0.0s
=> => sha256:34e7affe3804836dc049e9508a6e03da8607fd95fbec7be0d8153a8644c5d013 10.17kB / 10.17kB 0.0s
=> => sha256:c1f41a82e202385d53d0678a5f04ea4559588e22b959d9c49da2723fd00a1a82 6.44MB / 6.44MB 0.7s
=> => sha256:062b3b1203c0d3a415d1e8dd255dd016989463e973b29b9408ac75c96d82a095 1.26MB / 1.26MB 0.4s
=> => extracting sha256:5bacde1ae2d4533b7222789835dbdbe5428bfdbf47f1c0db05c045ab93060da0 0.1s
=> => sha256:0349cc8c591fafa8f2b8d254205e9a600b1ca1dba9c9e2971d0be81c24f99741 8.20MB / 8.20MB 0.7s
=> => sha256:85ded6cd843ad09d83a99fea8a871c5d43c407972a35b814c10447f2451a1695 1.31MB / 1.31MB 0.8s
=> => extracting sha256:c1f41a82e202385d53d0678a5f04ea4559588e22b959d9c49da2723fd00a1a82 0.1s
=> => sha256:2a5662fcca089469e657c6c11ae438988e58b6b59ceae919b1dbff9a735c91ee 116B / 116B 1.0s
=> => sha256:9a21eff9e4f6b087e9d1515ff9c814d55f9486ad5a14b44648f20fe1da8190c9 3.14kB / 3.14kB 1.0s
=> => sha256:315c5a49c3d058660037f7f252c0870f8505e35951bb64a6b41fc17d56268b41 108.81MB / 108.81MB 3.9s
=> => extracting sha256:062b3b1203c0d3a415d1e8dd255dd016989463e973b29b9408ac75c96d82a095 0.1s
=> => sha256:38d80feef54e6fdd2edefd5b8d89b4c6469e305da59d231cc58354788439dcb5 9.45kB / 9.45kB 1.4s
=> => sha256:04f6eb12164f125dd878b3759879afabf884b042c0bb660d2ef4a8ebea1641ad 129B / 129B 1.3s
=> => extracting sha256:0349cc8c591fafa8f2b8d254205e9a600b1ca1dba9c9e2971d0be81c24f99741 0.2s
=> => sha256:51f3f8ab4bb5d5e7d14e32f596afbe6f1b112bec43c6251ea4e8d6a63dd3be2c 166B / 166B 1.7s
=> => sha256:626110df8c5f65ed837a42a72e318791f0795304fd85439c6e5f5b6c7e902c16 6.08kB / 6.08kB 1.8s
=> => extracting sha256:85ded6cd843ad09d83a99fea8a871c5d43c407972a35b814c10447f2451a1695 0.1s
=> => sha256:a8bfeed3b03a815b361901c0cdf3812e47f7a08e0a38c54bd2468bf1e84c949f 185B / 185B 2.1s
=> => extracting sha256:2a5662fcca089469e657c6c11ae438988e58b6b59ceae919b1dbff9a735c91ee 0.0s
=> => extracting sha256:9a21eff9e4f6b087e9d1515ff9c814d55f9486ad5a14b44648f20fe1da8190c9 0.0s
=> => extracting sha256:315c5a49c3d058660037f7f252c0870f8505e35951bb64a6b41fc17d56268b41 1.9s
=> => extracting sha256:38d80feef54e6fdd2edefd5b8d89b4c6469e305da59d231cc58354788439dcb5 0.0s
=> => extracting sha256:04f6eb12164f125dd878b3759879afabf884b042c0bb660d2ef4a8ebea1641ad 0.0s
=> => extracting sha256:51f3f8ab4bb5d5e7d14e32f596afbe6f1b112bec43c6251ea4e8d6a63dd3be2c 0.0s
=> => extracting sha256:626110df8c5f65ed837a42a72e318791f0795304fd85439c6e5f5b6c7e902c16 0.0s
=> => extracting sha256:a8bfeed3b03a815b361901c0cdf3812e47f7a08e0a38c54bd2468bf1e84c949f 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 177B 0.0s
=> [stage-1 2/13] RUN adduser --system --uid 999 postgres --home /var/lib/postgresql 1.0s
=> [stage-1 3/13] COPY --from=postgres_base /usr/lib/postgresql /usr/lib/postgresql 0.3s
=> [stage-1 4/13] COPY --from=postgres_base /usr/share/postgresql /usr/share/postgresql 0.2s
=> [stage-1 5/13] COPY --from=postgres_base --chown=postgres /var/run/postgresql /var/run/postgresql 0.2s
=> [stage-1 6/13] RUN gosu postgres initdb --locale=C --encoding=UTF-8 --auth-host password 8.0s
=> [stage-1 7/13] RUN echo "ALTER USER postgres PASSWORD 'somesecret'" | gosu postgres postgres --single 0.9s
=> [stage-1 8/13] RUN echo "CREATE DATABASE synapse" | gosu postgres postgres --single 3.2s
=> [stage-1 9/13] RUN mv /conf/shared.yaml.j2 /conf/shared-orig.yaml.j2 0.5s
=> [stage-1 10/13] COPY conf/workers-shared-extra.yaml.j2 /conf/shared.yaml.j2 0.1s
=> [stage-1 11/13] WORKDIR /data 0.1s
=> [stage-1 12/13] COPY conf/postgres.supervisord.conf /etc/supervisor/conf.d/postgres.conf 0.1s
=> [stage-1 13/13] COPY conf/start_for_complement.sh / 0.1s
=> exporting to image 1.4s
=> => exporting layers 1.3s
=> => writing image sha256:e2d6351bbcbf5e297a2c78bd82b89a9963e49f2a657eb86fd1d8b503acb6fab4 0.0s
=> => naming to docker.io/library/complement-synapse 0.0s
=> => naming to ghcr.io/element-hq/synapse/complement-synapse 0.0s
Images built; running complement with -timeout=60m -run TestKnocking ./tests/csapi ./tests ./tests/msc3874 ./tests/msc3890 ./tests/msc3391 ./tests/msc3757 ./tests/msc3930 ./tests/msc3902 ./tests/msc3967 ./tests/msc4140 ./tests/msc4155 ./tests/msc4306
2025/11/05 15:13:18 config: &{BaseImageURI:complement-synapse DebugLoggingEnabled:false AlwaysPrintServerLogs:false EnvVarsPropagatePrefix:PASS_ SpawnHSTimeout:30s KeepBlueprints:[] HostMounts:[] BaseImageURIs:map[] PackageNamespace:csapi CACertificate:0xc0008c4608 CAPrivateKey:0xc0008f88c0 BestEffort:false HostnameRunningComplement:host.docker.internal EnableDirtyRuns:true HSPortBindingIP:127.0.0.1 PostTestScript:}
testing: warning: no tests to run
PASS
ok github.com/matrix-org/complement/tests/csapi 1.261s [no tests to run]
2025/11/05 15:13:20 config: &{BaseImageURI:complement-synapse DebugLoggingEnabled:false AlwaysPrintServerLogs:false EnvVarsPropagatePrefix:PASS_ SpawnHSTimeout:30s KeepBlueprints:[] HostMounts:[] BaseImageURIs:map[] PackageNamespace:fed CACertificate:0xc00098c608 CAPrivateKey:0xc0009c0150 BestEffort:false HostnameRunningComplement:host.docker.internal EnableDirtyRuns:true HSPortBindingIP:127.0.0.1 PostTestScript:}
=== RUN TestKnockingInMSC3787Room
2025/11/05 15:13:20 Sharing [SERVER_NAME=hs1 SYNAPSE_LOG_TESTING=1 SYNAPSE_COMPLEMENT_USE_WORKERS= SYNAPSE_COMPLEMENT_DATABASE=sqlite] host environment variables with container
2025/11/05 15:13:51 ============================================
2025/11/05 15:13:51 dirty : Server logs:
Complement Synapse launcher
Args:
Env: SYNAPSE_COMPLEMENT_DATABASE=sqlite SYNAPSE_COMPLEMENT_USE_WORKERS= SYNAPSE_COMPLEMENT_USE_ASYNCIO_REACTOR=
Certificate request self-signature ok
subject=CN=hs1
Traceback (most recent call last):
File "<frozen runpy>", line 189, in _run_module_as_main
File "<frozen runpy>", line 112, in _get_module_details
File "/usr/local/lib/python3.13/site-packages/synapse/__init__.py", line 37, in <module>
raise Exception("asdf fake error")
Exception: asdf fake error
Generating a random secret for SYNAPSE_REGISTRATION_SHARED_SECRET
Generating a random secret for SYNAPSE_MACAROON_SECRET_KEY
Generating synapse config file /data/homeserver.yaml
Generating log config file /data/log.config
Setting ownership on /data to 991:991
Traceback (most recent call last):
File "/start.py", line 290, in <module>
main(sys.argv, os.environ)
~~~~^^^^^^^^^^^^^^^^^^^^^^
File "/start.py", line 228, in main
return generate_config_from_template(
config_dir, config_path, environ, ownership
)
File "/start.py", line 147, in generate_config_from_template
subprocess.run(args, check=True)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/subprocess.py", line 577, in run
raise CalledProcessError(retcode, process.args,
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['gosu', '991:991', '/usr/local/bin/python', '-m', 'synapse.app.homeserver', '--config-path', '/data/homeserver.yaml', '--keys-directory', '/data', '--generate-keys']' returned non-zero exit status 1.
Generating base homeserver config
Traceback (most recent call last):
File "/configure_workers_and_start.py", line 1136, in <module>
main(sys.argv[1:], os.environ)
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/configure_workers_and_start.py", line 1073, in main
generate_base_homeserver_config()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/configure_workers_and_start.py", line 612, in generate_base_homeserver_config
subprocess.run([sys.executable, "/start.py", "migrate_config"], check=True)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/subprocess.py", line 577, in run
raise CalledProcessError(retcode, process.args,
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/local/bin/python', '/start.py', 'migrate_config']' returned non-zero exit status 1.
2025/11/05 15:13:51 ============== dirty : END LOGS ==============
2025/11/05 15:13:51 ============== While dirty deploying complement_fed_dirty_hs1 : START ALL COMPLEMENT DOCKER PORT BINDINGS ==============
2025/11/05 15:13:51 Container: 98c5c775116d7acafc15793dd454061c4ebc2c8cc0b0ade10ecc3a94df8681a6: [/complement_fed_dirty_hs1]
2025/11/05 15:13:51 (host) -> (container)
2025/11/05 15:13:51 =============== While dirty deploying complement_fed_dirty_hs1 : END ALL COMPLEMENT DOCKER PORT BINDINGS ===============
test_package.go:191: CreateDirtyDeployment failed: CreateDirtyServer: Failed to deploy image complement-synapse : dirty: failed to check server is up. timed out checking for homeserver to be up: inspect container 98c5c775116d7acafc15793dd454061c4ebc2c8cc0b0ade10ecc3a94df8681a6 => health: unhealthy
--- FAIL: TestKnockingInMSC3787Room (30.67s)
=== RUN TestKnocking
2025/11/05 15:13:51 Sharing [SERVER_NAME=hs1 SYNAPSE_LOG_TESTING=1 SYNAPSE_COMPLEMENT_USE_WORKERS= SYNAPSE_COMPLEMENT_DATABASE=sqlite] host environment variables with container
2025/11/05 15:13:51 ============== While dirty deploying complement_fed_dirty_hs1 : START ALL COMPLEMENT DOCKER PORT BINDINGS ==============
2025/11/05 15:13:51 Container: 98c5c775116d7acafc15793dd454061c4ebc2c8cc0b0ade10ecc3a94df8681a6: [/complement_fed_dirty_hs1]
2025/11/05 15:13:51 (host) -> (container)
2025/11/05 15:13:51 =============== While dirty deploying complement_fed_dirty_hs1 : END ALL COMPLEMENT DOCKER PORT BINDINGS ===============
test_package.go:191: CreateDirtyDeployment failed: CreateDirtyServer: Failed to deploy image complement-synapse : ContainerCreate: Error response from daemon: Conflict. The container name "/complement_fed_dirty_hs1" is already in use by container "98c5c775116d7acafc15793dd454061c4ebc2c8cc0b0ade10ecc3a94df8681a6". You have to remove (or rename) that container to be able to reuse that name.
--- FAIL: TestKnocking (0.00s)
FAIL
FAIL github.com/matrix-org/complement/tests 33.802s
2025/11/05 15:13:20 config: &{BaseImageURI:complement-synapse DebugLoggingEnabled:false AlwaysPrintServerLogs:false EnvVarsPropagatePrefix:PASS_ SpawnHSTimeout:30s KeepBlueprints:[] HostMounts:[] BaseImageURIs:map[] PackageNamespace:msc3874 CACertificate:0xc000a00008 CAPrivateKey:0xc000938070 BestEffort:false HostnameRunningComplement:host.docker.internal EnableDirtyRuns:true HSPortBindingIP:127.0.0.1 PostTestScript:}
testing: warning: no tests to run
PASS
ok github.com/matrix-org/complement/tests/msc3874 3.000s [no tests to run]
2025/11/05 15:13:18 config: &{BaseImageURI:complement-synapse DebugLoggingEnabled:false AlwaysPrintServerLogs:false EnvVarsPropagatePrefix:PASS_ SpawnHSTimeout:30s KeepBlueprints:[] HostMounts:[] BaseImageURIs:map[] PackageNamespace:msc3890 CACertificate:0xc000a8e008 CAPrivateKey:0xc000950070 BestEffort:false HostnameRunningComplement:host.docker.internal EnableDirtyRuns:true HSPortBindingIP:127.0.0.1 PostTestScript:}
testing: warning: no tests to run
PASS
ok github.com/matrix-org/complement/tests/msc3890 0.912s [no tests to run]
2025/11/05 15:13:17 config: &{BaseImageURI:complement-synapse DebugLoggingEnabled:false AlwaysPrintServerLogs:false EnvVarsPropagatePrefix:PASS_ SpawnHSTimeout:30s KeepBlueprints:[] HostMounts:[] BaseImageURIs:map[] PackageNamespace:msc3391 CACertificate:0xc000b50008 CAPrivateKey:0xc000b1a070 BestEffort:false HostnameRunningComplement:host.docker.internal EnableDirtyRuns:true HSPortBindingIP:127.0.0.1 PostTestScript:}
testing: warning: no tests to run
PASS
ok github.com/matrix-org/complement/tests/msc3391 0.413s [no tests to run]
2025/11/05 15:13:19 config: &{BaseImageURI:complement-synapse DebugLoggingEnabled:false AlwaysPrintServerLogs:false EnvVarsPropagatePrefix:PASS_ SpawnHSTimeout:30s KeepBlueprints:[] HostMounts:[] BaseImageURIs:map[] PackageNamespace:msc3757 CACertificate:0xc000c54008 CAPrivateKey:0xc000c1c070 BestEffort:false HostnameRunningComplement:host.docker.internal EnableDirtyRuns:true HSPortBindingIP:127.0.0.1 PostTestScript:}
testing: warning: no tests to run
PASS
ok github.com/matrix-org/complement/tests/msc3757 1.725s [no tests to run]
2025/11/05 15:13:18 config: &{BaseImageURI:complement-synapse DebugLoggingEnabled:false AlwaysPrintServerLogs:false EnvVarsPropagatePrefix:PASS_ SpawnHSTimeout:30s KeepBlueprints:[] HostMounts:[] BaseImageURIs:map[] PackageNamespace:msc3930 CACertificate:0xc000cba008 CAPrivateKey:0xc000c84070 BestEffort:false HostnameRunningComplement:host.docker.internal EnableDirtyRuns:true HSPortBindingIP:127.0.0.1 PostTestScript:}
testing: warning: no tests to run
PASS
ok github.com/matrix-org/complement/tests/msc3930 1.423s [no tests to run]
2025/11/05 15:13:18 config: &{BaseImageURI:complement-synapse DebugLoggingEnabled:false AlwaysPrintServerLogs:false EnvVarsPropagatePrefix:PASS_ SpawnHSTimeout:30s KeepBlueprints:[] HostMounts:[] BaseImageURIs:map[] PackageNamespace:msc3902 CACertificate:0xc000c54008 CAPrivateKey:0xc000c16070 BestEffort:false HostnameRunningComplement:host.docker.internal EnableDirtyRuns:true HSPortBindingIP:127.0.0.1 PostTestScript:}
testing: warning: no tests to run
PASS
ok github.com/matrix-org/complement/tests/msc3902 1.286s [no tests to run]
2025/11/05 15:13:18 config: &{BaseImageURI:complement-synapse DebugLoggingEnabled:false AlwaysPrintServerLogs:false EnvVarsPropagatePrefix:PASS_ SpawnHSTimeout:30s KeepBlueprints:[] HostMounts:[] BaseImageURIs:map[] PackageNamespace:msc3967 CACertificate:0xc0009b0608 CAPrivateKey:0xc0000e00e0 BestEffort:false HostnameRunningComplement:host.docker.internal EnableDirtyRuns:true HSPortBindingIP:127.0.0.1 PostTestScript:}
testing: warning: no tests to run
PASS
ok github.com/matrix-org/complement/tests/msc3967 1.478s [no tests to run]
2025/11/05 15:13:18 config: &{BaseImageURI:complement-synapse DebugLoggingEnabled:false AlwaysPrintServerLogs:false EnvVarsPropagatePrefix:PASS_ SpawnHSTimeout:30s KeepBlueprints:[] HostMounts:[] BaseImageURIs:map[] PackageNamespace:msc4140 CACertificate:0xc000c5c008 CAPrivateKey:0xc000c28070 BestEffort:false HostnameRunningComplement:host.docker.internal EnableDirtyRuns:true HSPortBindingIP:127.0.0.1 PostTestScript:}
testing: warning: no tests to run
PASS
ok github.com/matrix-org/complement/tests/msc4140 1.458s [no tests to run]
2025/11/05 15:13:18 config: &{BaseImageURI:complement-synapse DebugLoggingEnabled:false AlwaysPrintServerLogs:false EnvVarsPropagatePrefix:PASS_ SpawnHSTimeout:30s KeepBlueprints:[] HostMounts:[] BaseImageURIs:map[] PackageNamespace:msc4155 CACertificate:0xc000a9e008 CAPrivateKey:0xc00091a070 BestEffort:false HostnameRunningComplement:host.docker.internal EnableDirtyRuns:true HSPortBindingIP:127.0.0.1 PostTestScript:}
testing: warning: no tests to run
PASS
ok github.com/matrix-org/complement/tests/msc4155 0.950s [no tests to run]
2025/11/05 15:13:20 config: &{BaseImageURI:complement-synapse DebugLoggingEnabled:false AlwaysPrintServerLogs:false EnvVarsPropagatePrefix:PASS_ SpawnHSTimeout:30s KeepBlueprints:[] HostMounts:[] BaseImageURIs:map[] PackageNamespace:msc4306 CACertificate:0xc0009a4008 CAPrivateKey:0xc0008e8070 BestEffort:false HostnameRunningComplement:host.docker.internal EnableDirtyRuns:true HSPortBindingIP:127.0.0.1 PostTestScript:}
testing: warning: no tests to run
PASS
ok github.com/matrix-org/complement/tests/msc4306 3.284s [no tests to run]
FAIL
We have different behavior somehow. Anything you want me to share to compare?
Member
Author
There was a problem hiding this comment.
docker system info is a good start. Here's mine:
Client: Docker Engine - Community
Version: 28.5.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.29.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.40.3
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 80
Server Version: 28.5.1
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: local
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b98a3aace656320842a23f4a392a33f46af97866
runc version: v1.3.0-0-g4ca628d1
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.17.6-300.fc43.x86_64
Operating System: Fedora Linux 43 (Workstation Edition)
OSType: linux
Architecture: x86_64
CPUs: 20
Total Memory: 31.01GiB
Name: <REDACTED>
ID: <REDACTED>
Docker Root Dir: /var/lib/docker
Debug Mode: false
Username: <REDACTED>
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: false
Contributor
There was a problem hiding this comment.
We already shared that here -> #18210 (comment)
Member
Author
There was a problem hiding this comment.
Ah, so we did...
Looking at it again, the only difference between our configs is in the Storage Driver section. I do have the containerd image store enabled for multi-platform builds. And sure enough, disabling it lets the build work as expected.
I've scoured through docker/moby/buildkit issues, but I can't find exactly why using the containerd image store causes builds to prefer remote images. The closest issue I can find is that changing the Buildx driver to something other than docker prevents local images from being usable at all, but 1) my driver is docker whether I use the containerd image store or not, and 2) it's not that my builds can't use local images at all; it's that they just prefer remote ones.
At this point I'll chalk this up to Docker bugs/edge-cases, and will just use the default image store until the next time I need multi-arch builds. Hopefully this PR will at least help out anyone else who runs into these issues 🙂
Thanks for your continued patience on this.
Member
Author
|
Closing this because the root cause of my build issues was my Docker configuration: #18210 (comment) This PR could still be used to make builds compatible with the kind of config I had, but it's not the responsibility of this repo to cover non-standard configs. |
This was referenced Feb 17, 2026
Contributor
We ended up introducing this idea into the codebase via #19475 A bit forced by CI but good to be compatible with whatever Docker image store/storage driver is used. Really unfortunate that they behave subtly differently. @AndrewFerr Thanks for chatting this through to find the root cause as it helped me figure out what was going wrong in CI in the end ❤️ |
MadLittleMods
added a commit
that referenced
this pull request
Feb 18, 2026
…mages being chosen over local) (#19475) Fix remote images being chosen over the local ones we just built with Complement in CI (any Docker environment using the `containerd` image store). This problem means that Complement jobs in CI don't actually test against the code from the PR (since 2026-02-10). This PR approaches the problem the same way that @AndrewFerr proposed in #18210. This is better than the alternative listed below as we can just make our code compatible with whatever image store is being used. ### Problem Spawning from #19460 (comment) where we found that our Complement jobs in CI don't actually test against the code from the PR at the moment. This is caused by a change in Docker Engine 29.0.0: > `containerd` image store is now the default for **fresh installs**. This doesn't apply to daemons configured with `userns-remap` (see [moby#47377](moby/moby#47377)). > > *-- 29.0.0 (2025-11-10), https://docs.docker.com/engine/release-notes/29/#2900* And our `ubuntu-latest` GitHub runner (`Current runner version: '2.331.0'`) [points](https://github.com/actions/runner-images/blob/ubuntu24/20260209.23/images/ubuntu/Ubuntu2404-Readme.md) to using Docker client/server `29.1.5` 🎯 This Docker version bump happened on actions/runner-images@416418d (2026-02-10) (`28.0.4` -> `29.1.5`). Specific PR: actions/runner-images#13633 --- I found this because I reviewed and remembered #18210 was a thing that @AndrewFerr ran into. And then running `dockers system prune` also revealed the problematic `containerd` in CI. Checking the Docker changelogs, I found the new default culprit and then could trace down where the GitHub runners made the dependency update. --------- Co-authored-by: Andrew Ferrazzutti <andrewf@element.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is because I hit issues with the Complement script's attempt to build the
synapse-workersimage, as Docker refused to build it from my localmatrixdotorg/synapseimage & insisted on using the latest remote synapse image from Docker Hub instead.As it turns out, this was because my Docker build driver was something other than
docker, which is what's needed to letFROM <image>prefer local images: docker/buildx#159 (comment)Reinstalling Docker from upstream repos (as opposed to Fedora's) lets the workers image build as intended, meaning this PR is only needed for compatibility with other build drivers.
Pull Request Checklist
EventStoretoEventWorkerStore.".code blocks.(run the linters)