Skip to content

Comments

Enable pushing early image cache to ghcr.io registry#26119

Merged
potiuk merged 1 commit intoapache:mainfrom
potiuk:add-early-cache-push
Sep 1, 2022
Merged

Enable pushing early image cache to ghcr.io registry#26119
potiuk merged 1 commit intoapache:mainfrom
potiuk:add-early-cache-push

Conversation

@potiuk
Copy link
Member

@potiuk potiuk commented Sep 1, 2022

In order to react quicker to changes in setup.py, setup.cfg and
Dockerfiles, we can push an early image cache to the ghcr.io cache.

This build is done way before constraints are regenerated so it is
using the "current" constraints at the moment the merge happens.
This should be safe - if the image fails to build (we ignore that)
the cache will not be pushed, but if it will, then even if we merge
a new commit quickly and the rest of the build is cancelled, the
cache will be refreshed with new Dockerfile/setup.py/setup.cfg and
the subsequent builds will run much faster.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk potiuk requested review from ashb and kaxil as code owners September 1, 2022 21:43
@potiuk potiuk force-pushed the add-early-cache-push branch 2 times, most recently from 221c8ab to 731c39c Compare September 1, 2022 21:51
@potiuk potiuk changed the title Enable pushing early image cache to ghdcr.io repo Enable pushing early image cache to ghcr.io repo Sep 1, 2022
@potiuk potiuk force-pushed the add-early-cache-push branch from 731c39c to 3e72ab9 Compare September 1, 2022 21:52
@potiuk potiuk changed the title Enable pushing early image cache to ghcr.io repo Enable pushing early image cache to ghcr.io registry Sep 1, 2022
@potiuk potiuk requested a review from mik-laj September 1, 2022 21:52
@potiuk
Copy link
Member Author

potiuk commented Sep 1, 2022

This is a reasult of discussion with @o-nikolas and @mik-laj on Slack: https://apache-airflow.slack.com/archives/C015SLQF059/p1661905832810429

This change should make our cache of images much "fresher" even in case our main starts failing. It does not attempt to upgrade constraints - it will try to build the image with current constraints and push it as cache. This is done early in the build workflow and it will not wait for all tests to finish (like the final cache pushing which also upgrades constraints).

Result of this change will be that even if tests fail, if the images can be build and succeed, they will be pushed as cache quickly - and this will make subsequent builds to be much faster.

@potiuk potiuk force-pushed the add-early-cache-push branch 3 times, most recently from 6c671af to c011ad2 Compare September 1, 2022 22:47
In order to react quicker to changes in setup.py, setup.cfg and
Dockerfiles, we can push an early image cache to the ghcr.io cache.

This build is done way before constraints are regenerated so it is
using the "current" constraints at the moment the merge happens.
This should be safe - if the image fails to build (we ignore that)
the cache will not be pushed, but if it will, then even if we merge
a new commit quickly and the rest of the build is cancelled, the
cache will be refreshed with new Dockerfile/setup.py/setup.cfg and
the subsequent builds will run much faster.
@potiuk potiuk force-pushed the add-early-cache-push branch from c011ad2 to fd0e518 Compare September 1, 2022 22:49
@potiuk potiuk merged commit 479ab54 into apache:main Sep 1, 2022
@potiuk potiuk deleted the add-early-cache-push branch September 1, 2022 22:56
@o-nikolas
Copy link
Contributor

Thanks for the quick turnaround!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants