Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Improve speed of regular upgrades when dependencies change. (apache#3…
…7360) When regular dependencies change we enable UPGRADE_TO_NEWER_DEPENDENCIES flag. After some recent improvements in installing dependencies from the branch tip where we can install airflow using `[devel-ci]` extra from the branch tip, this opens up another option for speeding up the installations, when some dependencies change. So far - UPGRADE_TO_NEWER_DEPENDENCIES necessitated to reinstall all dependencies with airflow attempting to upgrade them eagerly. This takes ~ 28 minutes on CI now from the scratch. However, this is not needed. We can still start from the set of dependencies pre-installed by installation from branch tip, and only trigger "UPGRADE_TO_NEWER_DEPENDENCIES" after that - so that the "URL" installation cache does not get invalidated. This should limit the time needed to build such an image to ~ 8 minutes in CI when cache is built. In the future we might further optimize it - as we now have all the mechanisms necessary to do that in selective checks. For example we could trigger UPGRADE_TO_NEWER_DEPENDENCIES only when some dependencies are changes, and not when new dependencies are added, further limiting the time needed to build such image to some ~ 3/4 minutes. But this should be a separate PR and should be carefuly thought about, as it might also be that this will lead to conflicts with the latest constraints..
- Loading branch information