SSL problems on python:3.11.8-bookworm image?

[babinos87]

I decided to upgrade an image of mine to Python 3.11, and I thought to use the bookworm version (I've used bullseye so far).

This is the image I try: https://hub.docker.com/layers/library/python/3.11.8-bookworm/images/sha256-72afb375030b13c8c9cb72ba1d8c410f25307c2dbbd7d59f9c6ccea5cb152ff9?context=explore

I can build the image for amd64 architecture (no errors), but on arm64 I get SSL errors for bookworm, like (truncated snippet):

#18 [linux/arm64 4/9] RUN . /usr/src/app/.virtualenvs/webEnv311/bin/activate     && pip3 install --no-cache-dir -r /tmp/requirements.txt --trusted-host https://my-pypi.domain.io     && rm -f /tmp/requirements.txt

#18 [linux/arm64 4/9] RUN <log removed I just install requiremements>

#18 16.08 WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
#18 16.91 WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /packages/
#18 17.42 WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /packages/
#18 18.42 WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /packages/
#18 20.43 WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /packages/
#18 24.44 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /packages/
...
#18 32.01 Could not fetch URL https://pypi.org/simple/arrow/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/arrow/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
#18 32.09 ERROR: Could not find a version that satisfies the requirement arrow<2.0.0,>=1.3.0 (from versions: none)
#18 32.10 ERROR: No matching distribution found for arrow<2.0.0,>=1.3.0
#18 32.17 WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.

I was able to reproduce this locally. And I can confirm that when building 3.11.8-bullseye, everything works fine on both amd64 and arm64 architectures (builds succeed without those errors).

Is there some problem with the Python 3.11.8 on the bookworm image when building for arm64 (did not check other versions apart from amd64 which works).

[tianon]

A couple of ideas -- it looks like you're connecting to a custom python modules server, and I think bookworm might've updated some of the configuration to block older TLS versions? Is it possible your server doesn't support some of the newer TLS standards?

It could also be something like the libseccomp2 issues we've seen that crop up in weird looking ways (and could show up like what you're seeing) -- the easiest/quickest way to try is --security-opt seccomp=unconfined on a docker run test (but with the understanding that doing so is insecure). If that works / the correct solution there is to make sure your Docker, runc, and most importantly libseccomp2 on the host are up-to-date (possibly even more up-to-date than your distro has available 🙈).

I can't reproduce a failure of TLS in that image, even on arm64: 😅

$ docker run -it --rm --platform linux/arm64/v8 python:3.11-bookworm bash
Unable to find image 'python:3.11-bookworm' locally
3.11-bookworm: Pulling from library/python
c2964e85ea54: Pull complete 
d3436c315a5d: Pull complete 
603ae72c83b1: Pull complete 
bcabfc6c415b: Pull complete 
f22e038e21dd: Pull complete 
ef86414c697c: Pull complete 
b3c965daa95e: Pull complete 
9e26762c5e73: Pull complete 
Digest: sha256:8e697181d24bd77cc4251fdd37e4cdd6d725c5de2ed63b9bc8db77357400c5e2
Status: Downloaded newer image for python:3.11-bookworm
root@333fb6b04f60:/# pip install requests
Collecting requests
  Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting charset-normalizer<4,>=2 (from requests)
  Downloading charset_normalizer-3.3.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (33 kB)
Collecting idna<4,>=2.5 (from requests)
  Downloading idna-3.6-py3-none-any.whl.metadata (9.9 kB)
Collecting urllib3<3,>=1.21.1 (from requests)
  Downloading urllib3-2.2.1-py3-none-any.whl.metadata (6.4 kB)
Collecting certifi>=2017.4.17 (from requests)
  Downloading certifi-2024.2.2-py3-none-any.whl.metadata (2.2 kB)
Downloading requests-2.31.0-py3-none-any.whl (62 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 518.3 kB/s eta 0:00:00
Downloading certifi-2024.2.2-py3-none-any.whl (163 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 163.8/163.8 kB 5.6 MB/s eta 0:00:00
Downloading charset_normalizer-3.3.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (136 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.6/136.6 kB 6.6 MB/s eta 0:00:00
Downloading idna-3.6-py3-none-any.whl (61 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.6/61.6 kB 2.9 MB/s eta 0:00:00
Downloading urllib3-2.2.1-py3-none-any.whl (121 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.1/121.1 kB 5.1 MB/s eta 0:00:00
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests
Successfully installed certifi-2024.2.2 charset-normalizer-3.3.2 idna-3.6 requests-2.31.0 urllib3-2.2.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@333fb6b04f60:/# python3
Python 3.11.8 (main, Feb 13 2024, 09:03:56) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get('https://google.com')
>>> r.status_code
200

[babinos87]

Thanks for replying @tianon !

I am using docker buildx build and --security-opt seccomp=unconfined setting is not available to it.

I created an insecure builder though:

docker buildx create --name insecure-builder --driver docker-container --platform amd64,arm64 --bootstrap --buildkitd-flags '--allow-insecure-entitlement security.insecure'

and then tried to build again like:

docker buildx build \
    --load \
    --builder insecure-builder \
    --allow security.insecure \
    --platform linux/arm64 \
    -t my_image:dev-latest -f myDockerfile.Dockerfile .

and I still got the same error.

Keep looking at it. This one seems relevant? actions/setup-python#721

UPDATE: I tried the test that @tianon demonstrated but I used a requirements file that uses a private PyPI that we trust. All seem to work in this case (with bookworm on an arm64 processor).

So my next idea is that this is some issue with qemu builder / docker buildx build command, when using bookworm?

[babinos87]

After more digging, I see that the error appears when I try to install ANY python package in a virtual environment (using pip install virtualenv) inside the container, when I try to build with docker buildx. For example, this fails:

RUN . ${VENV_HOME}/${VENV_NAME}/bin/activate \
    && pip3 install --no-cache-dir -r requirements.txt

but this works:

# No virtual environment
RUN pip3 install --no-cache-dir -r requirements.txt

It's very hard to reproduce and debug, because it seems to appear only when I try to build a multi arch image with docker buildx; when I run my container (3.11.8-bookworm) and run all steps manually, I can install things via pip in the virtual environment.

[babinos87]

Strangely, the issue does not appear when I removed from my Dockerfile:

RUN apt-get install --allow-downgrades -y --no-install-recommends python3-dev

[tianon]

When you install python3-dev, you're getting a package that's likely very incompatible with the package from this image (and twice as much Python). 😅

We'd block distro packages of Python if we could (like we do in PHP images), but Python is often used for system-level utilities so that'd be too likely to block other important packages. 😞

[babinos87]

Yeah, what you wrote makes sense.

I think we had this install since an old installation of the app, in older images where we needed some stuff for other packages, safely for us to remove.

Not sure what could be done if someone needs to install those extra python headers, if they are not included in the distributed image, but I guess it's another topic.

Closing.