Add target-test-base images

[tomassebestik]

This PR creates pre-build Docker images that can be used as base image (FROM: ghcr.io/espressif/github-esp-dockerfiles/target-test-base-jammy:py3.8.17 for the target-test-env production image.

The goal is to have this image ready with all Python versions for future updates in GHCR.

This repository is public, so image can be pulled from anywhere by anyone. Building this image on GitHub is faster and easier than doing the same on our Gitlab Docker build runner, and downloading this image does not require any token or special access.

The main features of these images

• based on the latest Ubuntu 22.04 LTS (Jammy), with all non-default Python versions (3.8, 3.9, 3.10, 3.11)
• multi-architectural, can be used on PC/VM (amd64) and RaspberryPI 32bit (arm)
• the size of the images is relatively small (~300 MB)
• can be used also in GitHub actions and for various local developer tests
• the GitHub action builds all versions in parallel and stores them in GHCR
• builds can be triggered manually on specified versions with workflow dispatch: (will work after this PR merged to master)
• pip prefers our registry (https://dl.espressif.com/pypi/) with Python wheels over PyPi.org (https://pypi.org/simple)
• pip prefers binary (wheel, even lower version) over compiling from source code
• smarter cache for image build

Related

• (internal Gitlab MR): https://gitlab.espressif.cn:6688/idf/esp-dockerfiles/-/merge_requests/438
• (internal JIRA ticket): https://jira.espressif.com:8443/browse/RDT-510

[tomassebestik]

@hfudev @antmak @dobairoland @radimkarnis PTAL (sorry for branching the discussion between the Gitlab repo and this GH, but I think it makes sense ...)

---

I tested on both amd64 and arm (both RPI4 and RPI3b+), seems to work great.

Note that block:

# This prefers the Espressif pypi wheel registry (https://dl.espressif.com/pypi/) over the public PyPI (fallback)
ENV PIP_INDEX_URL=https://dl.espressif.com/pypi/
ENV PIP_EXTRA_INDEX_URL=https://pypi.org/simple

# Prioritize using binary (wheel) packages over source packages for faster installations and to avoid unnecessary compilations.

ENV PIP_PREFER_BINARY=true

... it basically doing this: For example, when installing cryptography using pip install cryptography>=2.1.4 on RPI machine:

• the latest current version of the package is cryptography==41.0.3
• pip first checks our registry (https://dl.espressif.com/pypi/, it will find a wheel for cryptography==41.0.2; this is older than the latest version, but it is a binary (wheel)
• then checks Pypi.org, finds cryptography==41.0.3; this is the latest, but there is no wheel for arm
• preference is set to wheels (using ENV PIP_PREFER_BINARY=true), so it installs cryptography==41.0.2 from https://dl.espressif.com/pypi and doesn't fail (because it still meets our requirement >=2.1.4)
• set it up this way using ENV in the Dockerfile, all subsequent calls to pip install ... should perform this way

You can test this (on any architecture) by:

docker run --rm -it ghcr.io/espressif/github-esp-dockerfiles/target-test-base-jammy:py3.8.17

---

• Build of target-test-env image based on this base image: https://gitlab.espressif.cn:6688/idf/esp-dockerfiles/-/jobs/38530653

[dobairoland]

I remember that "prefer binary" wasn't a good enough solution for all scenarios of using cryptography. That is why we use the "only binary" option.

[dobairoland]

Nice work @tomassebestik! LGTM.

[tomassebestik]

I remember that "prefer binary" wasn't a good enough solution for all scenarios of using cryptography. That is why we use the "only binary" option.

@dobairoland I remember that too... but surprisingly, it's working here now (and also in GL build job):

$ docker run --rm -it ghcr.io/espressif/github-esp-dockerfiles/target-test-base-jammy:py3.8.17                                                                                                                                                  

root@e9d68a1050f4:/# pip install cryptography
Looking in indexes: https://dl.espressif.com/pypi/, https://pypi.org/simple
Collecting cryptography
  Downloading https://dl.espressif.com/pypi/cryptography/cryptography-40.0.2-cp38-cp38-linux_armv7l.whl (1.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 8.6 MB/s eta 0:00:00
Collecting cffi>=1.12 (from cryptography)
  Downloading https://dl.espressif.com/pypi/cffi/cffi-1.15.1-cp38-cp38-linux_armv7l.whl (397 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 397.8/397.8 kB 7.9 MB/s eta 0:00:00
Collecting pycparser (from cffi>=1.12->cryptography)
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 kB 1.8 MB/s eta 0:00:00
Installing collected packages: pycparser, cffi, cryptography
Successfully installed cffi-1.15.1 cryptography-40.0.2 pycparser-2.21

root@e9d68a1050f4:/# uname -a
Linux e9d68a1050f4 5.10.17-v7+ #1421 SMP Thu May 27 13:59:01 BST 2021 armv7l armv7l armv7l GNU/Linux

(pulling older version of cryptography from our registry, not trying to build latest)

Maybe using https://dl.espressif.com/pypi as the main pip index and https://pypi.org/simple as a fallback in combination with prefer binary does the trick ... ?

Anyway ... when package is set with --only-binary in requirements, --only-binary should overwrite these rules, so keeping it like that it probably doesn't hurt anything.