Add support for ARM platform #22127
Merged
Add support for ARM platform #22127
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
potiuk
requested review from
ashb,
mik-laj,
jedcunningham,
kaxil and
turbaszek
as code owners
boring-cyborg
bot
added
area:dev-tools
area:helm-chart
|
I am refreshing the images now and will test it after the refreshing is done - with both Linux and Mac OS M1, what we should end-ip with is that our "ghcr.io" images get multi-platform and when running Breeze on M1, you will pull arm64, and on intel, you will pull amd64. The images should be refreshed in max 2 hours (it takes a lot of time to build ARM image on linux - I will optimize it later). Once it is, I will ping here and I would love if someone with intel-based MacOS could test it as well. Make sure to checkout my branch :) |
|
Building all images in parallel:
|
This was referenced Mar 9, 2022
potiuk
force-pushed
the
add-arm-support
branch
2 times, most recently
from
0bf3074
to
7d51e73
Compare
dstandish
approved these changes
Mar 9, 2022
|
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
github-actions
bot
added
the
full tests needed
mik-laj
reviewed
Mar 9, 2022
potiuk
force-pushed
the
add-arm-support
branch
2 times, most recently
from
d2fb3c2
to
6d2d3ff
Compare
This support is mostly for the developers, not for CI full chain yet. It has several limitations: * no MySQL client support * no MsSQL client support * no CI tests yet What is implemented: * automated detection of ARM/AMD architecture when building and running breeze * automated cache refresh on CI for ARM/AMD Currently only development (ghcr.io) images are supported for ARM. Fixes: apache#18849 Fixes: apache#17494 Relates to: apache#15635 The images published in DockerHub for now are AMD64 only. We will run development with M1 images for some time and later we will likely make our DockerHub images multi-platform as well. Also Hadolint does not have ARM images yet so we had to disable it and we should re-enable it back after the support is added. See hadolint/hadolint#411
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Mar 10, 2022
Previous apache#22127 had `--push` added in a wrong place. It should only be added when cache is being built and it was added always, with resulted in authentication error as login has not been performed before the --push.
potiuk
added a commit
that referenced
this pull request
Mar 10, 2022
Previous #22127 had `--push` added in a wrong place. It should only be added when cache is being built and it was added always, with resulted in authentication error as login has not been performed before the --push.
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Mar 11, 2022
The apache#22127 change introduced a change how execution of docker commnds was done (due to LD_PRELOAD change) and they started to use entrypoint_ci, however this caused undesired effect of running database initialization and printing extra lines which was not needed and cluttered the output. This PR introduced SKIP_ENVIRONMENT_INITIALIZATION that (if set to true) skips the entire initialization of the entrypoint_ci
potiuk
added a commit
that referenced
this pull request
Mar 11, 2022
The #22127 change introduced a change how execution of docker commnds was done (due to LD_PRELOAD change) and they started to use entrypoint_ci, however this caused undesired effect of running database initialization and printing extra lines which was not needed and cluttered the output. This PR introduced SKIP_ENVIRONMENT_INITIALIZATION that (if set to true) skips the entire initialization of the entrypoint_ci
41 tasks
jedcunningham
added
the
changelog:skip
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This support is mostly for the developers, not for CI full chain yet.
It has several limitations:
What is implemented:
running breeze
Currently only development (ghcr.io) images are supported for ARM.
Fixes: #18849
Relates to: #15635
The images published in DockerHub for now are AMD64 only. We will
run development with M1 images for some time and later we will
likely make our DockerHub images multi-platform as well.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.