python:3.9 - Failed run apt update from the last version of the image

[aviadpriel]

as titled

today morning we found that our builds failed
we are using python:3.9

when we run apt update the build failed

it looks that it fails because of the upgrade to bookworm

we get this errors

#5 [build 2/4] RUN apt-get update &&     apt-get install -y --no-install-recommends     poppler-utils
#5 sha256:fdb97fe2ebc3fcf79626c6628756f871cc0d6b0fc02ba21a2949fe33d31a2373
#5 0.240 Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
#5 0.247 Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
#5 0.248 Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
#5 0.277 Err:1 http://deb.debian.org/debian bookworm InRelease
#5 0.277   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY F8D2585B8783D481
#5 0.303 Err:2 http://deb.debian.org/debian bookworm-updates InRelease
#5 0.303   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY 6ED0E7B82643E131
#5 0.329 Err:3 http://deb.debian.org/debian-security bookworm-security InRelease
#5 0.329   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 54404762BBB6E853 NO_PUBKEY BDE6D2B9216EC7A8
#5 0.330 Reading package lists...
#5 0.337 W: GPG error: http://deb.debian.org/debian bookworm InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY F8D2585B8783D481
#5 0.337 E: The repository 'http://deb.debian.org/debian bookworm InRelease' is not signed.
#5 0.337 W: GPG error: http://deb.debian.org/debian bookworm-updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY 6ED0E7B82643E131
#5 0.337 E: The repository 'http://deb.debian.org/debian bookworm-updates InRelease' is not signed.
#5 0.337 W: GPG error: http://deb.debian.org/debian-security bookworm-security InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 54404762BBB6E853 NO_PUBKEY BDE6D2B9216EC7A8
#5 0.337 E: The repository 'http://deb.debian.org/debian-security bookworm-security InRelease' is not signed.
#5 0.337 E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
#5 0.337 E: Sub-process returned an error cod

[Cookiehook]

We're seeing the same issues on python:3.10. Worked around by using python:3.10-bullseye for now.

[aviadpriel]

yes, same for python:3.9-bullseye

but we have couple of security fixes on Debian 12 that we want to get

[a1723]

The main question is: Will it be fixed in python:xx images or now we must always used images with "bullseye" postfix ?

[yosifkit]

I am unable to reproduce. I'd suggest updating docker and libseccomp on the host. Newer base OS's use newer system calls and an older libseccomp can block them since they are unknown to it. You can verify that it is libseccomp by running the bookworm image with --security-opt seccomp=unconfined.

This is similar to the update to Ubuntu focal: docker-library/mongo#606 (comment)

$ docker run -it python:3.9-bookworm bash
Unable to find image 'python:3.9-bookworm' locally
3.9-bookworm: Pulling from library/python
bba7bb10d5ba: Already exists
ec2b820b8e87: Already exists
284f2345db05: Already exists
fea23129f080: Already exists
7c62c924b8a6: Already exists
b2210932934e: Pull complete
ee9c01829d92: Pull complete
d6285f41f1b6: Pull complete
Digest: sha256:98f018a1afd67f2e17a4abd5bfe09b998734ba7c1ee54780e7ed216f8b8095c3
Status: Downloaded newer image for python:3.9-bookworm
root@0b679f272394:/# apt-get update
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [24.2 kB]
Fetched 9176 kB in 1s (7771 kB/s)
Reading package lists... Done
root@0b679f272394:/#

[jimbudarz]

I see the same for python:3.8-slim and others after the update from buster -> bookworm.
I think the image needs an update of the GPG public keys for the bookworm apt repos.

EDIT: Until this is fixed, you can use the -buster versions of images, such as python:3.8-slim-buster

[jmmclean]

hitting us as well, using python:3.7

[keever]

This broke our ci docker build jobs that used python:3.9-slim We used docker v24.0.2 to do the build.

[muasif80]

It seems like the pointers for the latest image tags are moved to bookworm. The python:3.9-slim-bullseye might work instead of python:3.9-slim or maybe python:3.9-slim-buster will work too. The official 3.9 image github update shows that the version is updated from buster to bookworm.

[jacobeturpin]

Can confirm that our builds were also impacted using python:3.8-slim. Modifying to python:3.8-slim-bullseye resolved the issue, so we're temporarily using that until the issue is resolved.

[mitsuo0114]

Same here in out CI tool. I could solved with downgrading.

I'm assuming following change in Debian may have some affect to CI. (I haven't checked deeply)

https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#python3-pep-668

---

5.2.2. Python Interpreters marked externally-managed
The Debian provided python3 interpreter packages (python3.11 and pypy3) are now marked as being externally-managed, following PEP-668. The version of python3-pip provided in Debian follows this, and ?will refuse to manually install packages on Debian's python interpreters, unless the --break-system-packages option is specified.

If you need to install a Python application (or version) that isn't packaged in Debian, we recommend that you install it with pipx (in the pipx Debian package). pipx will set up an environment isolated from other applications and system Python modules, and install the application and its dependencies into that.

---

[aebm]

Hi, that is a "problem" with the change of the debian base image used. It seems related to seccomp and it is mentioned here debuerreotype/docker-debian-artifacts#197 (comment)

Example (PS: Don't run you containers with more privileges that necessary =) )

docker run --name=dummy --rm -ti python:3.11-slim bash -c 'apt-get update && apt-get install -y less && echo OK'
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [28.3 kB]
Fetched 9180 kB in 2s (4697 kB/s)                         
Reading package lists... Done
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code

vs

docker run --name=dummy --rm -ti --privileged python:3.11-slim bash -c 'apt-get update && apt-get install -y less && echo OK'
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [28.3 kB]
Fetched 9180 kB in 2s (4844 kB/s)                          
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  less
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 131 kB of archives.
After this operation, 321 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bookworm/main amd64 less amd64 590-2 [131 kB]
Fetched 131 kB in 0s (3313 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package less.
(Reading database ... 8386 files and directories currently installed.)
Preparing to unpack .../archives/less_590-2_amd64.deb ...
Unpacking less (590-2) ...
Setting up less (590-2) ...
OK

[aebm]

I can confirm that it "works" when unconfined. Don't try this at home

docker run --name=dummy --rm -ti --security-opt seccomp=unconfined python:3.11-slim bash -c 'apt-get update && apt-get install -y less && echo OK'
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [28.3 kB]
Fetched 9180 kB in 2s (4821 kB/s)                          
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  less
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 131 kB of archives.
After this operation, 321 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bookworm/main amd64 less amd64 590-2 [131 kB]
Fetched 131 kB in 0s (2989 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package less.
(Reading database ... 8386 files and directories currently installed.)
Preparing to unpack .../archives/less_590-2_amd64.deb ...
Unpacking less (590-2) ...
Setting up less (590-2) ...
OK

[tianon]

The proper fix is updating libseccomp2 (and Docker/runc) on your host. 👍

[yosifkit]

(basically the same comment as redis/docker-library-redis#365 (comment))

Root cause: it is Docker with libseccomp so a newer syscall used in Debian Bookworm packages/libs is being blocked. (Thanks for confirming @aebm)

libseccomp lets you configure allowed syscalls for a process. Docker sets a default seccomp profile for all containers such that only certain syscalls are allowed and everything else is blocked (so, newer syscalls that are not yet known to libseccomp or docker are blocked).

• verify that it is libseccomp by running the Bookworm-based image with --security-opt seccomp=unconfined
  • use only as a test; it is less secure to keep running them "unconfined"
  • learn more: https://docs.docker.com/engine/security/seccomp/
• one fix:
  • update libseccomp and docker on the host running the containers
• one workaround:
  • switch to the *bullseye images (in the python images, these will continue to be maintained/updated until the respective Python end of life or the next Debian release, Debian Trixie)

[chazapp]

We've encountered this issue during docker builds on CircleCI runners using an old machine image (ubuntu-2004:202101-01). Updating the machine config to ubuntu-2004:current solved the issue for us.

[aporwal18]

Facing this same issue, any proper fix to this?

[yosifkit]

Facing this same issue, any proper fix to this?

#837 (comment):

• update libseccomp and docker on the host running the containers

[mke21]

I've got this problem running in gitlab-ci with the worker running on a private Kubernetes cluster. I'm not sure how to fix it apart from using the older bullseye image which obviously is a stopgap solution. This is the setup for the build:

variables:
  DOCKER_HOST: tcp://localhost:2375
  DOCKER_TLS_CERTDIR: ""

services:
  - name: docker:stable-dind

image: docker:stable

[sergei-lobanov]

Any plans for a normal fix?

[tianon]

Any plans for a normal fix?

#837 (comment)

The proper fix is updating libseccomp2 (and Docker/runc) on your host. 👍