bump debian to bookworm

[raphaelauv]

it's still using bullseye for the build of CI image

#22 [main  1/33] FROM docker.io/library/python:3.8-slim-bullseye@sha256:6eb2e25b7f582a59c8e78e8cd82b3afaeb3463f961abd33d777841dbfec26ae6

@potiuk do you know why ? thanks

[potiuk]

Yes. Because for non-committers, "build-image" is run from "main" not from your PR and in main you have ---build-arg PYTHON_BASE_IMAGE=python:3.8-slim-bullseye. The way to solve it is to push your branch as main branch in your fork (and let it run there as "push" workflow (you can also manually build image locally and you will see (I am quite sure) that it won't build. Smth like:

git push origin your-branch:main -f 

[potiuk]

The way to solve it later is to make a PR from "airflow" repo (then image build happens in-ci-workflow and the workflow has enough permissions to be able to push such image to airflow registry. But you need "committer" privileges for it.

[raphaelauv]

thanks for the explanation

I could build locally the CI image without mysql ->

breeze ci-image build  --force-build --python-image python:3.11-slim-bookworm --python 3.11

no mysql release for debian 12 yet -> https://dev.mysql.com/downloads/mysql/

[potiuk]

Yeah. MySQL is traditionally some 6 months behind everyone else, I really wish we drop mysql , I even raised it a few times on devlist .... "Elephant in the room" https://lists.apache.org/thread/dp78j0ssyhx62008lbtblrc856nbmlfb

[raphaelauv]

#31941

[marzoli]

Up please!

[raphaelauv]

[raphaelauv]

#35286

[raphaelauv]

@potiuk

"467B942D3A79BD29" is the current "key" for mysql

how do I compute the value from the debian12 release ?

http://repo.mysql.com/apt/debian/dists/bookworm/Release.gpg

thank you for your help

[Taragolis]

"467B942D3A79BD29" is the current "key" for mysql

Is it not working? For me it is look like the same for bookworm and bullseye

There is no info in MySQL page yet about bookworm (maybe not updated), but attached signature in page is the same as we use now

gpg: key 467B942D3A79BD29: public key "MySQL Release Engineering <mysql-build@oss.oracle.com>" imported

Some reference (not currently updated)

• https://dev.mysql.com/doc/refman/8.0/en/checking-gpg-signature.html
• https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/

[raphaelauv]

sorry @Taragolis just realize the error is about mssql

[Taragolis]

sorry @Taragolis just realize the error is about mssql

y and s

Seems like Microsoft change a bit installation method, and do not suggest anymore to use deprecated apt-key add

In addition I could see some changes in pre-defined repositories list

Debian 11, https://packages.microsoft.com/config/debian/11/prod.list

deb [arch=amd64,armhf,arm64] https://packages.microsoft.com/debian/11/prod bullseye main

Debian 12, https://packages.microsoft.com/config/debian/12/prod.list

deb [arch=amd64,arm64,armhf signed-by=/usr/share/keyrings/microsoft-prod.gpg] https://packages.microsoft.com/debian/12/prod bookworm main

[raphaelauv]

I can build locally the image
breeze ci-image build --force-build --python-image python:3.11-slim-bookworm --python 3.11

---

But you need "committer" privileges for it.

@potiuk or @Taragolis what should we do about that ? thanks

[potiuk]

But you need "committer" privileges for it.
@potiuk or @Taragolis what should we do about that ? thanks

I think you should be able push your change as "main" to your fork, and when you enable the "Test" workflow in your fork, it should build and run tests there. Try it out and if not, we might use the opportunity to fix any issues with it @raphaelauv

Simply:

git push `<my_repo>` mybranch:main -f

Ping me when you try it and enable workflows and I will help to solve any issues

[potiuk]

BTW. Yes. i think it's about enough time after bookworm was released (more than year) to get all the "other" software to catch up already (without us loosing time on it).

As mentioneed before, we were not in a hurry because Bullseye will be supported from July 2024 to 2026 as part of Debian LTS, but we should definitely look closely to switching. And 2.8.0 might be the first time to support it as default and 2.9 we could drop bullseye.

However I would very much like to do it like we did with buster-> bullseye switch - see release notes for 2.3 and 2.4 in our docker image changelog: https://airflow.apache.org/docs/docker-stack/changelog.html#airflow-2-4

We switched CI to bullseye by default in 2.3 (buster was deprecated) but we still continued to build buster (just build) in CI until 2.4 - and users could choose buster or bullsee by a flag in breeze or choosing buster or bullseye as base image (breeze flag was doing exactly this - switched the base image). Only in 2.4 we dropped buster.

This means that once we figure out the right changes to get bookworm working, we should bring them to the code eventually as conditional code (they will slightly differ). This is for example why currently "Build CI image" job fails - because the code in "main" will pass buster image as the base image and it misses gcc11-dev dependency. Also it means that we have to add extra job to at least build bullseye in CI - or maybe at least a subset of tests to run on bullseye still.

But we can do it as next step. For now let's make the main build succeed for @raphaelauv and then we figure out next steps.

[raphaelauv]

this is the run in the main branch of my fork -> https://github.com/raphaelauv/airflow/actions/runs/6727169212

[potiuk]

Looks pretty good. celery integration test fails and production image need some fixes but other than that it seems almost working

[raphaelauv]

celery integration fail was flaky ( rerun worked )

with my last commit I could build locally the prod image

 breeze prod-image build

github action on my fork is in progress

[potiuk]

Looks great then. I might want to take it from here and add option to choose bookworm or bullseye as breeze option (but it might be a good exercise for you - happy to help with it :)

[raphaelauv]

I might want to take it from here 

I agree , thanks

[potiuk]

Smth still not fine with K8S tests though it might be a CI issue - maybe you could run K8S tests locally and see if this is replicable locally (there is a really nice chapter explaining how to do in our TESTING.rst and it's a wizard like approach to setup testing with it). In the meantime I will add the bookworm switches to breeze.

[potiuk]

I pushed a fixup with --debian-version switch. Now you can choose which version you want to build. It also has release notes, documentation in "Build imaages" about it, and CI job that will run only in "main" / "canary" builds to test if the production "bullseye" images are building correctly for all Python versions.

I also opened an "Apache repo" PR to test if the K8S problem is specific to your environment only

[potiuk]

(you are co-author BTW).

[potiuk]

Closing this one as we have #35376 now (and looks like K8S tests are all green there so we are good.

Thanks Again for being persistent and leading it!