Limit Airflow and breeze to < 3.12 #35123
Conversation
Python 3.12 has a few breaking changes comparing to earlier versions. While 3.7 - 3.11 were largely backwards compatible, Python 3.12 is the first one for a long time that started to break things more aggressively. For now we know that Airflow will not work with Python 3.12 mainly because of distutils removal (https://peps.python.org/pep-0632/) and not because of Airflow's usage of it but pendulum's before version 3. While we are working on getting Pendulum 3 support in apache#34744 and the apache#34746, there are likely other dependencies that have similar issue. Until we fix it and add official 3.12 support, we can limit airflow to not be installable on 3.12.
|
I opened a PR but I am not sure if this is the best idea (devlist message to follow) |
|
Is there an example of how the distutils removal affects breeze? |
Just pendulum installation in some cases and os systems example in this thread from Slack: https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1697895028537189
|
|
Even though we do not pin pendulum there, likely other dependencies of breeze do. |
|
Yeah that looks like not specific to Pendulum but from Poetry. This is the best of the non-optimal solutions. |
|
BTW. As mentioned in the devlist thread https://lists.apache.org/thread/ogzt5y84vqh4rrbdyrdprgrb07ncd8hk - i now think it's better to error out rather than add install-requires, because it will not prevent those kind of errors for Airflow (this might still be a good idea for breeze though as it is installed from source requirements in main). Converting to draft until we have more voices there. |
How do you know it's poetry that's involved BTW @uranusjr ? |
|
It looks like Pendulum uses Poetry’s extension-building functionality to build the C extension. The failure comes from its build script that imports distutils. |
🤯 🤯 🤯 🤯 🤯 🤯 |
ephraimbuddy
added
the
type:misc/internal
Python 3.12 has a few breaking changes comparing to earlier versions. While 3.7 - 3.11 were largely backwards compatible, Python 3.12 is the first one for a long time that started to break things more aggressively. For now we know that Airflow will not work with Python 3.12 mainly because of distutils removal (https://peps.python.org/pep-0632/) and not because of Airflow's usage of it but pendulum's before version 3. While we are working on getting Pendulum 3 support in #34744 and the #34746, there are likely other dependencies that have similar issue. Until we fix it and add official 3.12 support, we can limit airflow to not be installable on 3.12. (cherry picked from commit 020691f)
Python 3.12 has a few breaking changes comparing to earlier versions. While 3.7 - 3.11 were largely backwards compatible, Python 3.12 is the first one for a long time that started to break things more aggressively.
For now we know that Airflow will not work with Python 3.12 mainly because of distutils removal (https://peps.python.org/pep-0632/) and not because of Airflow's usage of it but pendulum's before version 3.
While we are working on getting Pendulum 3 support in #34744 and the #34746, there are likely other dependencies that have similar issue.
Until we fix it and add official 3.12 support, we can limit airflow to not be installable on 3.12.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.