CI Images are now pre-build and stored in registry

[potiuk]

With this change we utilise the latest pull_request_target
event type from Github Actions and we are building the
CI image only once (per version) for the entire run.

This safes from 2 to 10 minutes per job (!) depending on
how much of the Docker image needs to be rebuilt.

It works in the way that the image is built only in the
build-or-wait step. In case of direct push run or
scheduled runs, the build-or-wait step builds and pushes
to the GitHub registry the CI image. In case of the
pull_request runs, the build-and-wait step waits until
separate build-ci-image.yml workflow builds and pushes
the image and it will only move forward once the image
is ready.

This has numerous advantages:

1. Each job that requires CI image is much faster because
   instead of pulling + rebuilding the image it only pulls
   the image that was build once. This saves around 2 minutes
   per job in regular builds but in case of python patch level
   updates, or adding new requirements it can save up to 10
   minutes per job (!)

2. While the images are buing rebuilt we only block one job waiting
   for all the images. The tests will start running in parallell
   only when all images are ready, so we are not blocking
   other runs from running.

3. Whole run uses THE SAME image. Previously we could have some
   variations because the images were built at different times
   and potentially releases of dependencies in-between several
   jobs could make different jobs in the same run use slightly
   different image. This is not happening any more.

4. Also when we push image to github or dockerhub we push the
   very same image that was built and tested. Previously it could
   happen that the image pushed was slightly different than the
   one that was used for testing (for the same reason)

5. Similar case is with the production images. We are now building
   and pushing consistently the same images accross the board.

6. Documentation building is split into two parallel jobs docs
   building and spell checking - decreases elapsed time for
   the docs build.

7. Last but not least - we keep the history of al the images

   • those images contain SHA of the commit. This means
     that we can simply download and run the image locally to reproduce
     any problem that anyone had in their PR (!). This is super useful
     to be able to help others to test their problems.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

Hello everyone. This is a major overhaul of the way how we are utilizing GitHub Actions - something that was enabled by recent features released by GitHub Actions (namely "workflow_run" feature).

I've been working on it last week and heavily tested it on my fork https://github.com/potiuk/airflow/actions. I hope there will be rather little number of teething problems, but I am already very familiar with how GA work and I will be able to fix any problems quickly. I am also going to watch it once we merge it to make sure it works as expected.

See commit description for what is achieved by this change. I have just one thing to say - this is my "dream" architecture of the CI builds that I had in mind at the very beginning of my work on Airflow, one that could only be achieved by the most recent changes by GitHub. I really hope this is one of the last fundamental changes in the scripting for CI because I literally run out of ideas what can be improved (just kidding - there are always small things ;).

It has many nice properties but the most important ones:

• 5-12 minutes saved for each Job (Builds of images are done only once not for each job). Not per whole run - but per Job (!). This will help both - increase number of parallell PRs that can be run and decrease the feedback time for each build. There were sometimes much slower builds when python base image was upgraded or Dockerfile changed - this problem will be gone.

• the jobs/runs are fully consistent - all jobs in the same build use exactly the same image prepared only once.

• full trackability and reproducibility of each run - we keep the images in GitHub registry and you can recreate the exact failed run by running ./breeze --github-image-id <RUN_ID> or ./breeze --github-image-id <COMMIT_ID> for merged runs.

• I cleaned up outputs of the job so that they only show relevant information

• I cleaned up initialization code for bash scripts - removed some duplicates and organized it better and I fully documented it - describing the purpose of all options (that was the lat script refactoring I planned)

It's quite a huge change, and I can try to split it into smaller ones (but conceptually it is one big overhaul of the way our CI works)

When you can start from the workflows at the end of the documentation: https://github.com/PolideaInternal/airflow/blob/prebuild-ci-images-in-github-actions/CI.rst - I prepared some sequence diagrams of the CI architecture (using mermaid - which is an absolutely cool tool for converting markdownish descriptions of diagrams into really nice diagrams). It explains all the "whys" and also "hows".

NOTE! For Review/Merge I needed to disable waiting for images, so the speedups are not visible yet - I have to merge it to master in order to enable the "Build Image" workflows. I also use "master" version of my own Github Cancel Action which I developed for that purpose - I will release it's v2 version and switch to it once we get a few days of the builds working in Airflow.

I developed https://github.com/potiuk/cancel-workflow-runs new Github Action for "Cancel Workflow Run" that is a swiss-army-knife of Run cancelling and I plan to share it with Apache Beam and other Apache projects that might need it as well.

I really look forward to review comments and merging it eventually. This will help all of contributors and committers to move faster. This is literally completion of 2 years of the "dream" architecture for our CI :).

[potiuk]

CC: @ad-m

[mik-laj]

This change is huge. What can we reduce its size? I can see a few changes that we can extract and do as a separate change:

• merging to --github-organisation and --github-repo flags into --github-repository
• building documentation and spell checking separately,.
• renaming ci_*..sh scripts
• pylint fixes in docs/build_docs.py
• renaming variable in scripts/ci/in_container/run_prepare_backport_readme.sh, scripts/ci/in_container/run_test_package_import_all_classes.sh
• less verbose output from helm,kubectl - scripts/ci/libraries/_verbosity.sh
• pyllint fixes in scripts/tools/list-integrations.py.

This may seem like a lot of extra work, but I think it's the only way we can be sure about this change and not get a lot of teething problems. Now it is very painful for me to review this change because I have to remember every change and think about the consequences of many changes at once.

If you like, I will be happy to help you break down this change to make it easier to review.

[potiuk]

This change is huge. What can we reduce its size? I can see a few changes that we can extract and do as a separate cha

Yep. I am happy to do it. I will try to separate out some of the changes and keep on rebasing this one until the "gist" of the change remains as last commit