Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add python 3.9 #15515

Merged
merged 1 commit into from Jun 25, 2021
Merged

Add python 3.9 #15515

merged 1 commit into from Jun 25, 2021

Conversation

@potiuk
Copy link
Member

@potiuk potiuk commented Apr 25, 2021


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@potiuk
Copy link
Member Author

@potiuk potiuk commented Apr 25, 2021

cc: @r-richmond - This will not YET work. But after we migrate to PIP 21 fully with #15515 (I hope in a day or two) and after Apache Beam releases their 2.29.0 (Release candidate is out there as of 16th of April https://pypi.org/project/apache-beam/2.29.0rc1/) we might finally be able to pull that one off! In Apache Beam we have pyarrow limitation moved to <4.0.0 and Pyarrow 3.0.0 is already python 3.9 compatible.

@potiuk potiuk force-pushed the add-python-3.9 branch 2 times, most recently from 157e824 to 9f2d4b0 Apr 25, 2021
@potiuk
Copy link
Member Author

@potiuk potiuk commented Apr 25, 2021

BTW. Seems that voting for the RC candidate for Beam has been completed and there is just mechanics of the release, so it is more than likely tha the 2.29.0 will be out today/tomorrow and we will be able to add Python 3.9 finally.

@potiuk
Copy link
Member Author

@potiuk potiuk commented Apr 27, 2021

All Right apache-beam 2.29 released! Let's give it a try :).

@potiuk potiuk force-pushed the add-python-3.9 branch from 9f2d4b0 to cbf0719 Apr 27, 2021
@potiuk
Copy link
Member Author

@potiuk potiuk commented Apr 27, 2021

Not yet :(. We also have sasl dependency which is non-py3.9 compliant.

But there is a release coming for sasl and thrift-sasl soon to address it too. I asked for some timeline information: cloudera/thrift_sasl#36 (comment)

CI.rst Outdated Show resolved Hide resolved
scripts/ci/selective_ci_checks.sh Outdated Show resolved Hide resolved
@r-richmond
Copy link
Contributor

@r-richmond r-richmond commented Jun 6, 2021

@potiuk in case you missed it; it looks like thrift_sasl released 0.4.3 which as far as I can tell was the version we were waiting for.

@potiuk
Copy link
Member Author

@potiuk potiuk commented Jun 7, 2021

@potiuk in case you missed it; it looks like thrift_sasl released 0.4.3 which as far as I can tell was the version we were waiting for.

Cool. I will look at this this week :). 🤞

@potiuk potiuk force-pushed the add-python-3.9 branch from 5664225 to e2ab074 Jun 13, 2021
@potiuk
Copy link
Member Author

@potiuk potiuk commented Jun 13, 2021

Bad news @r-richmond. While thrift-sasl is already good-to-go ...... seems that we had one more hidden dependency on sasl :(

@uranusjr warned me several times that extras in PIP are totally insane, and make it next to impossible to build a reasonable dependency graph - and I must concur with it now..... In our case pyhive[hive] makes us still download and try to compile sasl (which has no chance to succeed as sasl support for 3.9 even if working for alpha release in April has been apparently abandoned).

I added comment to the relevant issue in PyHive:

dropbox/PyHive#380 (comment)

Upvotes, encouraging comments are welcome :)

@r-richmond
Copy link
Contributor

@r-richmond r-richmond commented Jun 14, 2021

Well that is disappointing but certainly outside airflow's control. Thank you for taking another stab at it.

That said given how painful this has been I opened #16426 which was related / based off of #11950 (comment)

@potiuk
Copy link
Member Author

@potiuk potiuk commented Jun 25, 2021

Shall we?

@potiuk
Copy link
Member Author

@potiuk potiuk commented Jun 25, 2021

Just one approve away from having Python 3.9 support in main :D.

@kaxil kaxil closed this Jun 25, 2021
@kaxil kaxil reopened this Jun 25, 2021
kaxil
kaxil approved these changes Jun 25, 2021
@potiuk potiuk merged commit ce44b62 into apache:main Jun 25, 2021
31 of 33 checks passed
@potiuk potiuk deleted the add-python-3.9 branch Jun 25, 2021
@potiuk
Copy link
Member Author

@potiuk potiuk commented Jun 25, 2021

Wooohooo!

@kaxil
Copy link
Member

@kaxil kaxil commented Jun 25, 2021

👯 🕺

yuqian90 added a commit to yuqian90/airflow that referenced this issue Jun 26, 2021
This includes several things:

* added per-provider support for python version. Each provider
  can now declare python versions it does not support
* excluded ldap core extra from Python 3.9.
* skip relevant tests in Python 3.9
@r-richmond
Copy link
Contributor

@r-richmond r-richmond commented Jul 3, 2021

Is python 3.9 not supported on 2.1.1 (i.e. we missed the cutoff) or was that an oversight on the docs?

@millin
Copy link
Contributor

@millin millin commented Jul 3, 2021

Please backport python 3.9 support to 2.1.1

@potiuk
Copy link
Member Author

@potiuk potiuk commented Jul 3, 2021

Yes. It's not "officially" supported yet. It's been added to main and not backported to 2.1.1. The 2.1 line is released from v2-1-stable branch not from main and we try to avoid excessive cherry-picking between the branches (unless it is necessary).

The 3.9 support is three things really:

  • CI tests - those make sense to add/fix for main only
  • Constraint files
  • Docker images

As a workaround you can still install airflow or build a docker image for python 3.9 with Airflow 2.1.1 and it will probably work fine.
This is quite possible (though not guaranteed to work) using the main version of constraints. The full guarantees we get only when we run the full test suite (which only happens in main for Python 3.9).

Here is how you can build Airflow 2.1.1 Python 3.9 image (you can run it from main airlfow sources):

docker build . \
    --build-arg PYTHON_BASE_IMAGE="python:3.9-slim-buster" \
    --build-arg AIRFLOW_VERSION="2.1.1" \
    --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-main" \
    --tag "my-custom-3-9-airflow-image:2.1.1"

Similarly installing airflow locally:

AIRFLOW_VERSION=2.1.1
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-${PYTHON_VERSION}.txt"
pip install --upgrade "apache-airflow[postgres,google]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

Also both Docker and local installation can be run with locally modified version of constraints if you find some conflicts, happy to help if you have problems.

I think as you are eager users that might be perfect opportunity for you to try it and let us know if you see any problems before we release 2.2 which will have full 3.9 support.

@potiuk
Copy link
Member Author

@potiuk potiuk commented Jul 3, 2021

Also this is very much documented in https://github.com/apache/airflow#requirements that 3.9 is for now only available officially for development version (which is 2.2.0dev currently). This is similar to MSSQL support which is currently only available in dev

@r-richmond
Copy link
Contributor

@r-richmond r-richmond commented Jul 7, 2021

Do we know what the next release version will be / Any rough timelines for 2.2?

(context: I'm trying to figure out how long we'd have to wait if we waited for 2.2 for python 3.9 vs if its worth just doing the workarounds you mentioned above).

Relevant milestone info that I can view

  • 2.1.2 milestone has 6 completed items and 0 open items with a due date of July 10th.
  • 2.1.3 milestone has 24 completed items and 17 open items with no due date
  • 2.2.0 milestone has 23 completed items and 32 open items with no due date

p.s. thanks again for tackling this update; I'm really excited about a couple quality of life things in 3.9.

potiuk added a commit to potiuk/airflow that referenced this issue Jul 8, 2021
This includes several things:

* added per-provider support for python version. Each provider
  can now declare python versions it does not support
* excluded ldap core extra from Python 3.9.
* skip relevant tests in Python 3.9

(cherry picked from commit ce44b62)
@ashb
Copy link
Member

@ashb ashb commented Jul 8, 2021

@r-richmond Airflow 2.2 will be roughly 3 months form 2.1.0, so ~end of August.

@potiuk
Copy link
Member Author

@potiuk potiuk commented Jul 8, 2021

In the meantime @r-richmond I released Python 3.9 compatble elasticsearch package and I have done some extra work and cherry-picked the Python3.9 changes to 2-1-stable branch: #16883.

I also generated constraints.

We are just about to release 2.1.2 and If the tests will run fine, we will merge it and then Python 3.9 will be easy to install for 2.1.2. Not sure if we mark it as officially available for 2.1.2, but maybe you will be able to help with testing it ?

We can push an image (I just build it, it builds fine), Also constraints are already there:

924d038

So once we tag it with 2.1.2, you will be able to install it both via docker and in venv using the same constraints pattern as for other versions.

If you confirm it works fine for you we might even make it "officialy available" in 2.1.2.

potiuk added a commit that referenced this issue Jul 8, 2021
This includes several things:

* added per-provider support for python version. Each provider
  can now declare python versions it does not support
* excluded ldap core extra from Python 3.9.
* skip relevant tests in Python 3.9

(cherry picked from commit ce44b62)
potiuk added a commit to potiuk/airflow that referenced this issue Jul 8, 2021
This includes several things:

* added per-provider support for python version. Each provider
  can now declare python versions it does not support
* excluded ldap core extra from Python 3.9.
* skip relevant tests in Python 3.9

(cherry picked from commit ce44b62)
@potiuk
Copy link
Member Author

@potiuk potiuk commented Jul 8, 2021

Seems like we are on a good path to have support for Python 3.9 in 2.1.2 @r-richmond, the change is merged there, constraints are updated and the tests generally pass, so I think even officially we might want to state (once 2.1.2 is released) that we have Python 3.9 support for it already. It would be great if you could test the release candidate once it is out :)

jhtimmins added a commit that referenced this issue Jul 8, 2021
This includes several things:

* added per-provider support for python version. Each provider
  can now declare python versions it does not support
* excluded ldap core extra from Python 3.9.
* skip relevant tests in Python 3.9

(cherry picked from commit ce44b62)
kaxil added a commit to astronomer/airflow that referenced this issue Jul 13, 2021
This includes several things:

* added per-provider support for python version. Each provider
  can now declare python versions it does not support
* excluded ldap core extra from Python 3.9.
* skip relevant tests in Python 3.9

(cherry picked from commit ce44b62)
(cherry picked from commit dcffcc1)
kaxil added a commit to astronomer/airflow that referenced this issue Jul 13, 2021
This includes several things:

* added per-provider support for python version. Each provider
  can now declare python versions it does not support
* excluded ldap core extra from Python 3.9.
* skip relevant tests in Python 3.9

(cherry picked from commit ce44b62)
(cherry picked from commit dcffcc1)
@jedcunningham jedcunningham added this to the Airflow 2.1.2 milestone Sep 10, 2021
@mik-laj mik-laj mentioned this pull request Oct 19, 2021
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

7 participants