Fix edge case with constraints conflicting with setup dependencies#33847
Merged
Fix edge case with constraints conflicting with setup dependencies#33847
Conversation
There was an edge case with race condition between merging the setup.py change to main and refreshing constraint causing a failure of PROD image building when the new setup.py had conflicting dependencies with the past constraints. This happened on Aug 28 with azure-mgmt-containerinstance which got upgrade to a new 8.0.0 version which was conflicting with previous setup.py: * azure-mgmt-containerinstance>=1.5.0,<2.0 to * azure-mgmt-containerinstance>=7.0.0,<9.0.0 The problem was that PROD builds used the constraints from main to build the image, because constraints were not produced in the CI builds for the same job (constraints wer only produced in the PR that had "upgrade to newer depedencies". The regular PR CI builds have a build-in protection against such edge case - they run with `--upgrade-on-failure` flag that causes the PRs to upgrade their constraints eagerly if they fail on constraints failure. This PR fixes the edge case by utilising the result of the flag --upgrade-on failure. After this PR, constraints will always be uploaded by CI jobs, not only when "upgrade to newer dependencies" PR is run. this means that when CI build falls back as result of `--upgrade-on-failure` the newly generated constraints are uploaded as artifacts and used by PROD image build. As a side-effect, every PR will produce the constraints used by that build as artifact. This is actually pretty good thing for diagnosis of dependency problems.
potiuk
requested review from
Taragolis,
ephraimbuddy,
hussein-awala and
uranusjr
eladkal
approved these changes
Aug 28, 2023
hussein-awala
approved these changes
Aug 28, 2023
Taragolis
approved these changes
Aug 28, 2023
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There was an edge case with race condition between merging the setup.py change to main and refreshing constraint causing a failure of PROD image building when the new setup.py had conflicting dependencies with the past constraints. This happened on Aug 28 with azure-mgmt-containerinstance which got upgrade to a new 8.0.0 version which was conflicting with previous setup.py:
to
The problem was that PROD builds used the constraints from main to build the image, because constraints were not produced in the CI builds for the same job (constraints wer only produced in the PR that had "upgrade to newer depedencies".
The regular PR CI builds have a build-in protection against such edge case - they run with
--upgrade-on-failureflag that causes the PRs to upgrade their constraints eagerly if they fail on constraints failure.This PR fixes the edge case by utilising the result of the flag --upgrade-on failure.
After this PR, constraints will always be uploaded by CI jobs, not only when "upgrade to newer dependencies" PR is run. this means that when CI build falls back as result of
--upgrade-on-failurethe newly generated constraints are uploaded as artifacts and used by PROD image build.As a side-effect, every PR will produce the constraints used by that build as artifact. This is actually pretty good thing for diagnosis of dependency problems.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.