Sync v2-9-stable with v2-9-test to release 2.9.0 #38647
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(cherry picked from commit 731b616)
When running compatibility check of providers with airflow, we install airflow first (with constraints) and providers as the next step (without constraints). This might - in some cases lead to accidental upgrade of airflow if `pip` decides to do so. By adding airflow package to the list of providers to install, we will keep it from being accidentally upgraded. (cherry picked from commit c855f1f)
Some of the providers specified in list of packages might have a >= specification, for that case we should avoid treating all of the specification as part of the provider id and strip the part after >=. (cherry picked from commit d6d7264)
(cherry picked from commit d4c2ea4)
Some of the variable names (importnat only for fork builds) have not been moved to the new workflows. This PR updates variabels used in various workflows to be set consistently. (cherry picked from commit 4ca80cc)
Default Python version had misleading description as input and it has not been used in prod-image-build (falling back to the version installed in runner). This PR synchronizes it across all workflows. (cherry picked from commit fdd3f93)
(cherry picked from commit 7494ab2)
If we find that the dag is unpaused already, we should just inform the user instead of raising error. Same with pausing (cherry picked from commit a64bb40)
When we build PROD images in CI, we need constraints, to know which version of the depencies to use. Those constraints are build by earlier steps in the build and uploaded as artifacts so that they can be re-used by PROD image build. However depending on the type of PROD build (in main or in v2-*) different constraints are used: for main build we use source constraints because we are installing provider packages build from main sources, in v2* builds we are using PyPI constraints, because we are installing provider packages from PyPI. This made the dependency tree a bit complicated because PROD builds would have to wait for constraints generation rather than use quickly prepared source constraints and in #38533 we introduced an extra step in PROD build for v2* branches to additionally generate constraint in-the job that builds PROD images. This however has some consequences, such as potentially incosistencies between constraints used during other parts of the build, because PROD images can be potentially built much later than the generated constraints for the rest of the build. Also it introduced the need for the PROD builds to pull CI images to generate the constraints which adds an overhead in the process and makes PROD builds in v2 branches few minutes longer than they could be. This PR approaches it differently. Since now with `uv` constraints generation is way faster (few minutes, we could easily make PROD builds use always the same constraint artifacts generated in single job and pay a little delay penalty of starting the PROD builds but get it back at PROD build time (because constraints will be generated only once). Also that removes (back) the need of PROD builds to pull the CI images and removes the need to generate source constraints at CI image build time - which will decrease the build time a little. This PR makes necessary changes to make it happens: * removes source constraints generation in CI images * extracting generate-constraints job to be separate workflow and merging "docs" instead (we are hitting the limit of 20 workflows) * add dependency from all PROD build to wait for generate-constraints * remove CI image pulling and related steps from PROD image build * switches PROD to download and use the full set of constraints generated by `generate-constraints` job (cherry picked from commit 32e04a4)
The workflow for docs building had been moved to a different sub-workflow and it needs secrets in canary build to upload the docs to AWS. This PR adds the secret inheriting to the docs workflow to fix the problem. (cherry picked from commit 85f3b46)
During the last week refactor input on mypy checks was moved to another workflow and mypy checks were not running for a day. This change brings it back. (cherry picked from commit a2f5307)
The config names all use "object storage", but the class is still using "object store". Since we're not renaming the configs, let's rename the class for consistency. (cherry picked from commit 0371ea8)
Co-authored-by: Tom Rutter <tom.rutter@zurich.com> (cherry picked from commit 7613795)
(cherry picked from commit 1d8413e)
The `total_entries` count should reflect how many log entries match the filters provided, not simply how many rows are in the table total. (cherry picked from commit 90e7b3f)
These are the attrs we allow sorting on, not filtering on. (cherry picked from commit e700f41)
(cherry picked from commit 35d1899)
|
#38358 is ready, I moved the data migration to db utils, and I tested it in both sides, it works as expected. |
ephraimbuddy
requested review from
bolkedebruin,
potiuk,
ashb and
jedcunningham
as code owners
bbovenzi
approved these changes
Apr 2, 2024
jedcunningham
approved these changes
Apr 2, 2024
(cherry picked from commit 6f0e5bc)
(cherry picked from commit 8455eb2)
(cherry picked from commit 39b3eb2)
Cleaning up docker is a single bash script to execute - it is defined as a composite action but since it's just one bash commenad to execute, it is better to just keep it as a bash script instead - which wil make it easier to add other scripts. (cherry picked from commit bc7b68b)
(cherry picked from commit 9c61147)
(cherry picked from commit e20a710)
* Improve stability of remove_task_decorator function * fix statics * test * remove test --------- Co-authored-by: Sam Wheating <samwheating@gmail.com> (cherry picked from commit f1301da)
Co-authored-by: Wei Lee <weilee.rx@gmail.com> (cherry picked from commit 32c88a2)
We have now a separate job that prepares and uploads artifacts when PROD image is being prepared for K8S testing. This saves a bit of time to prepare PROD image. However this step had a bug - it checked out the `main` version of code when preparing airflow and provider packages, so it was really testing the "previous" merge not the "current" PR. This PR switches the job to prepare artifacts to use the same checkout strategy as the other build-image workflow jobs. (cherry picked from commit eb541ae)
The image tag makes no difference. While we are using the image to run Helm tests (for now) the fact that we are using specific image shoudl not be part of the job name (it's printed in the steps when the image is pulled). (cherry picked from commit 7526440)
) Depending on selective checks, but also on the job executed, we choose whether to run job on public runners or self-hosted runners. So far the set of labels to select the runners were passed in a bit inconsistent way. Outputs of selective checks can only be strings and the `run-as` accepts array of strings (labels) - so we were using fromJSON to convert between the two. And we used runs-on inputs on a number of our workflows to pass the selection. However this meant that runs-on could be either string or array and that sometimes we passed public/self-hosted labels as strings directly and some of those were hard-coded. This PR changes it consistently across the board to introduce consistent approach: * build info have no selective checks results yet, so for them runs-on is hardcoded * similarly for "windows" and release jobs that are manually run without running selective checks * selective checks will produce three outuputs - JSON stringiified array of labels: * default (one that is selected depending on who runs the build) * public (for cases where we want to force the builds to use public runners * self-hosted (for cases where we want to force the builds to use self-hosted runners * all the outputs are named `<type>-runs-on-as-string` to make it clear they are all strings * all inputs of workflows expectings strings are named the same (with as-string suffix and <type> prefix) * whenever a job is run, we pass "runs-on" parameter to be `fromJSON` with appropriate type we want to use passed as input This will make it easier to reason on which job is using which type of runner and it will make it easier in the future to make it more flexible when we add ASF self-hosted runners and possibly our own K8S runners, or when we would want to change labels for public runners or self-hosted runners. (cherry picked from commit 9da08a5)
The "UPGRADE_TO_NEWER_DEPENDENCIES" build arg was a bit misleading as it was not a true/false value but rather "random hash"/"false". This PR makes it a bit more explicit: * the arg is named UPGRADE_INVALIDATION_STRING now * it's default is "" not false and all conditions are set to check for != or == "". The inputs/outputs in CI job remain as they were - "true/false" but this PR makes it clearer. (cherry picked from commit 1934c8b)
(cherry picked from commit 6ef381b)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Time for
2.9.0rc1!