-
Notifications
You must be signed in to change notification settings - Fork 13.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use uv
as packaging tool used in CI builds
#37683
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Excited 😄
Looks good, I hope the CI will pass
Not yet (expected) - but it's close :). |
And the numbers are REALLY REALLY good - see https://lists.apache.org/thread/t7pd78ktcs2s4b8mv3trwb7bdyjthcpz |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks Great.
108d490
to
1994e33
Compare
1994e33
to
40ad435
Compare
40ad435
to
256981b
Compare
The pintodb python package had a very old and very low lower limit, that introduced possible weird resolutions for dependencies when pinotdb had been downgraded to **really** low version - this has been observed in apache#37683 when we test replacing pip with uv in our CI environment. Bumping it to version >= 5.1.0 released last year in september - that was the first one that was not 0.N.*.
The pintodb python package had a very old and very low lower limit, that introduced possible weird resolutions for dependencies when pinotdb had been downgraded to **really** low version - this has been observed in #37683 when we test replacing pip with uv in our CI environment. Bumping it to version >= 5.1.0 released last year in september - that was the first one that was not 0.N.*.
256981b
to
e6a18e3
Compare
The openapi validators in older versions do not work well for some moto tests which use the validators. Bumping specifically the min versions of those validators should help with better dependency resolving (wheb using uv and lowest transitive versions strategy it will downgrade the validators and will cause the tests to fail as seen in apache#37683. Those are only test and development dependencies and we already use the min version specified in our CI, so it should have no impact on production airflow (but should help wiht CI/dependency resolution)
The openapi validators in older versions do not work well for some moto tests which use the validators. Bumping specifically the min versions of those validators should help with better dependency resolving (wheb using uv and lowest transitive versions strategy it will downgrade the validators and will cause the tests to fail as seen in #37683. Those are only test and development dependencies and we already use the min version specified in our CI, so it should have no impact on production airflow (but should help wiht CI/dependency resolution)
6f85cbd
to
5571a04
Compare
The `uv` tool released in Feb 2024 by ruff creators provides a way faster drop-in replacement to `pip` and we are using it now in our CI, when it can bring significant speed improvements (and soon possibly more features). This PR replaces `pip install` and `pip uninstall` with equivalent `uv pip install` and `uv pip uninstall` commands, controlled by a single `AIRFLOW_USE_UV` ARG. In CI images it is set to "true" so CI images are prepared using UV, but PROD images (which are also used during CI tests) are built using pip. This way we can get both - stability and compliance for user-facing `pip` installation and speed and new features coming from `uv`.
5571a04
to
0470aee
Compare
I need to close it and re-open it in |
Reopened it as #37692 |
The pintodb python package had a very old and very low lower limit, that introduced possible weird resolutions for dependencies when pinotdb had been downgraded to **really** low version - this has been observed in apache#37683 when we test replacing pip with uv in our CI environment. Bumping it to version >= 5.1.0 released last year in september - that was the first one that was not 0.N.*.
The openapi validators in older versions do not work well for some moto tests which use the validators. Bumping specifically the min versions of those validators should help with better dependency resolving (wheb using uv and lowest transitive versions strategy it will downgrade the validators and will cause the tests to fail as seen in apache#37683. Those are only test and development dependencies and we already use the min version specified in our CI, so it should have no impact on production airflow (but should help wiht CI/dependency resolution)
The pintodb python package had a very old and very low lower limit, that introduced possible weird resolutions for dependencies when pinotdb had been downgraded to **really** low version - this has been observed in #37683 when we test replacing pip with uv in our CI environment. Bumping it to version >= 5.1.0 released last year in september - that was the first one that was not 0.N.*. (cherry picked from commit e6a18e3)
The openapi validators in older versions do not work well for some moto tests which use the validators. Bumping specifically the min versions of those validators should help with better dependency resolving (wheb using uv and lowest transitive versions strategy it will downgrade the validators and will cause the tests to fail as seen in #37683. Those are only test and development dependencies and we already use the min version specified in our CI, so it should have no impact on production airflow (but should help wiht CI/dependency resolution) (cherry picked from commit f3159df)
The openapi validators in older versions do not work well for some moto tests which use the validators. Bumping specifically the min versions of those validators should help with better dependency resolving (wheb using uv and lowest transitive versions strategy it will downgrade the validators and will cause the tests to fail as seen in #37683. Those are only test and development dependencies and we already use the min version specified in our CI, so it should have no impact on production airflow (but should help wiht CI/dependency resolution) (cherry picked from commit f3159df)
The pintodb python package had a very old and very low lower limit, that introduced possible weird resolutions for dependencies when pinotdb had been downgraded to **really** low version - this has been observed in apache#37683 when we test replacing pip with uv in our CI environment. Bumping it to version >= 5.1.0 released last year in september - that was the first one that was not 0.N.*.
The openapi validators in older versions do not work well for some moto tests which use the validators. Bumping specifically the min versions of those validators should help with better dependency resolving (wheb using uv and lowest transitive versions strategy it will downgrade the validators and will cause the tests to fail as seen in apache#37683. Those are only test and development dependencies and we already use the min version specified in our CI, so it should have no impact on production airflow (but should help wiht CI/dependency resolution)
The
uv
tool released in Feb 2024 by ruff creators provides a way faster drop-in replacement topip
and we are using it now in our CI, when it can bring significant speed improvements (and soon possibly more features).This PR replaces
pip install
andpip uninstall
with equivalentuv pip install
anduv pip uninstall
commands, controlled by a singleAIRFLOW_USE_UV
ARG. In CI images it is set to "true" so CI images are prepared using UV, but PROD images (which are also used during CI tests) are built using pip. This way we can get both - stability and compliance for user-facingpip
installation and speed and new features coming fromuv
.^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.