Optimize CI builds for unimportant pyproject.toml changes #37305

potiuk · 2024-02-10T13:40:43Z

When dependencies change in pyproject.toml, we should run build
with upgrade-to-newer-dependencies, however we should not run
it when dependencies in pyproject.toml do not change.

That saves about 30 minutes of elapsed time of the build and heavily
limits the number of tests executed. It takes about 30 minutes now
to build the image that has "upgrade-to-newer-dependencies", we only
usually run default Python image in such case and we do not run many
tests that are not needed (for example K8S tests).

This PR optimizes out the case where non-dependency changes only are
done in pyproject.toml. This happens for example when you only change
docstrings and remove ruff rules in [[tools.ruff]] section.

We compare the dependencies and optional dependencies coming from
the change and only when there is a change in those, we set the
upgrade-to-newer-dependencies flag. We also print what changed.

Similarly full-tests-needed is only set when build-system changes
or when dependencies change, because then we want to make sure new
dependencies are working on all Python versions.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

potiuk · 2024-02-10T13:41:26Z

This one should help with significantly speeding D401-only PRS.

When dependencies change in pyproject.toml, we should run build with `upgrade-to-newer-dependencies`, however we should not run it when dependencies in pyproject.toml do not change. That saves about 30 minutes of elapsed time of the build and heavily limits the number of tests executed. It takes about 30 minutes now to build the image that has "upgrade-to-newer-dependencies", we only usually run default Python image in such case and we do not run many tests that are not needed (for example K8S tests). This PR optimizes out the case where non-dependency changes only are done in pyproject.toml. This happens for example when you only change docstrings and remove ruff rules in [[tools.ruff]] section. We compare the dependencies and optional dependencies coming from the change and only when there is a change in those, we set the `upgrade-to-newer-dependencies` flag. We also print what changed. Similarly full-tests-needed is only set when build-system changes or when dependencies change, because then we want to make sure new dependencies are working on all Python versions.

The apache#37305 optimized CI jobs to limit tests run when pyproject.toml is changed without impacting dependencies, optional_dependencies and build-system. However it impacts results of other unit tests as they were using `HEAD` as commit in selective checks, Instead of using HEAD, we are using neutral commit (one only changing INTHEWILD) for all such tests, so that we do not introduce side effects coming from the incoming PR.

The #37305 optimized CI jobs to limit tests run when pyproject.toml is changed without impacting dependencies, optional_dependencies and build-system. However it impacts results of other unit tests as they were using `HEAD` as commit in selective checks, Instead of using HEAD, we are using neutral commit (one only changing INTHEWILD) for all such tests, so that we do not introduce side effects coming from the incoming PR.

When dependencies change in pyproject.toml, we should run build with `upgrade-to-newer-dependencies`, however we should not run it when dependencies in pyproject.toml do not change. That saves about 30 minutes of elapsed time of the build and heavily limits the number of tests executed. It takes about 30 minutes now to build the image that has "upgrade-to-newer-dependencies", we only usually run default Python image in such case and we do not run many tests that are not needed (for example K8S tests). This PR optimizes out the case where non-dependency changes only are done in pyproject.toml. This happens for example when you only change docstrings and remove ruff rules in [[tools.ruff]] section. We compare the dependencies and optional dependencies coming from the change and only when there is a change in those, we set the `upgrade-to-newer-dependencies` flag. We also print what changed. Similarly full-tests-needed is only set when build-system changes or when dependencies change, because then we want to make sure new dependencies are working on all Python versions. (cherry picked from commit 6ce5225)

The #37305 optimized CI jobs to limit tests run when pyproject.toml is changed without impacting dependencies, optional_dependencies and build-system. However it impacts results of other unit tests as they were using `HEAD` as commit in selective checks, Instead of using HEAD, we are using neutral commit (one only changing INTHEWILD) for all such tests, so that we do not introduce side effects coming from the incoming PR. (cherry picked from commit 08e655a)