Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Less aggressive eager upgrade of requirements #8267

Merged

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented Apr 13, 2020

With this change requirements are only eagerly upgraded when
generating requirements when setup.py changes. They are also
eagerly upgraded when you run ./breeze generate-requirements
locally. Still the cron job will use the eager update mechanism
when building the docker image which means that CRON jobs will
still detect cases where upgrede of requirements causes failure
either at the installation time or during tests.


Make sure to mark the boxes below before creating PR: [x]


In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

@potiuk potiuk force-pushed the less-aggressive-requirements-upgrade branch from 2e46817 to f52d997 Compare April 13, 2020 09:32
@boring-cyborg boring-cyborg bot added provider:Apache provider:google Google (including GCP) related issues labels Apr 13, 2020
@potiuk potiuk mentioned this pull request Apr 13, 2020
5 tasks
With this change requirements are only eagerly upgraded when
generating requirements when setup.py changes. They are also
eagerly upgraded when you run ./breeze generate-requirements
locally. Still the cron job will use the eager update mechanism
when building the docker image which means that CRON jobs will
still detect cases where upgrede of requirements causes failure
either at the installation time or during tests.
@potiuk potiuk force-pushed the less-aggressive-requirements-upgrade branch from f52d997 to e9ea2af Compare April 13, 2020 10:24
@codecov-io
Copy link

codecov-io commented Apr 13, 2020

Codecov Report

Merging #8267 into master will decrease coverage by 0.42%.
The diff coverage is 87.50%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #8267      +/-   ##
==========================================
- Coverage   88.34%   87.91%   -0.43%     
==========================================
  Files         937      937              
  Lines       45231    45235       +4     
==========================================
- Hits        39959    39769     -190     
- Misses       5272     5466     +194     
Impacted Files Coverage Δ
airflow/providers/google/cloud/hooks/cloud_sql.py 67.58% <0.00%> (ø)
airflow/security/kerberos.py 76.08% <ø> (ø)
airflow/providers/google/cloud/operators/gcs.py 85.79% <66.66%> (+0.07%) ⬆️
airflow/models/dag.py 91.15% <100.00%> (+0.01%) ⬆️
airflow/providers/apache/pinot/hooks/pinot.py 91.60% <100.00%> (+0.06%) ⬆️
airflow/providers/google/cloud/hooks/dataflow.py 90.80% <100.00%> (ø)
airflow/utils/process_utils.py 70.67% <100.00%> (+0.22%) ⬆️
...flow/providers/apache/cassandra/hooks/cassandra.py 21.25% <0.00%> (-72.50%) ⬇️
...w/providers/apache/hive/operators/mysql_to_hive.py 35.84% <0.00%> (-64.16%) ⬇️
airflow/providers/redis/operators/redis_publish.py 50.00% <0.00%> (-50.00%) ⬇️
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c1f9478...e9ea2af. Read the comment docs.

CURRENT_SETUP_PY_HASH=$(md5sum "${AIRFLOW_SOURCES}/setup.py")
STORED_SETUP_PY_HASH=$(cat "${STORED_SETUP_PY_HASH_FILE}" 2>/dev/null || true)

if [[ ${STORED_SETUP_PY_HASH} != "${CURRENT_SETUP_PY_HASH}" ]]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if [[ ${STORED_SETUP_PY_HASH} != "${CURRENT_SETUP_PY_HASH}" ]]; then
if [[ "${STORED_SETUP_PY_HASH}" != "${CURRENT_SETUP_PY_HASH}" ]]; then

@potiuk potiuk merged commit 45c8983 into apache:master Apr 13, 2020
potiuk added a commit that referenced this pull request Apr 15, 2020
With this change requirements are only eagerly upgraded when
generating requirements when setup.py changes. They are also
eagerly upgraded when you run ./breeze generate-requirements
locally. Still the cron job will use the eager update mechanism
when building the docker image which means that CRON jobs will
still detect cases where upgrede of requirements causes failure
either at the installation time or during tests.

(cherry picked from commit 45c8983)
@potiuk potiuk added this to the Airflow 1.10.11 milestone Apr 20, 2020
cfei18 pushed a commit to cfei18/incubator-airflow that referenced this pull request Mar 5, 2021
With this change requirements are only eagerly upgraded when
generating requirements when setup.py changes. They are also
eagerly upgraded when you run ./breeze generate-requirements
locally. Still the cron job will use the eager update mechanism
when building the docker image which means that CRON jobs will
still detect cases where upgrede of requirements causes failure
either at the installation time or during tests.

(cherry picked from commit 45c8983)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:dev-tools provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants