Add support of Pendulum 3 #36281

Taragolis · 2023-12-18T16:23:32Z

Related: #35798, ~~the differences is bump to min pendulum 3~~

As suggestion in https://lists.apache.org/thread/5b3xw87ntl0m00d8jwcz13vnvxhybjdp keep support pendulum 2 for a while, even if it has some problem it might help users to migrate to pendulum 3.

Add --downgrade-pendulum by the same way as it implements into the --downgrade-sqlalchemy so we could also run tests against pendulum 2 for a while.

The only one version of pendulum supported - 2.1.2 with older versions such as 2.0.0 I had a problem with initialised Airflow, there is should not be a big problem because we have 2.1.2 for a long period of time in constraints, and I guess for all Airflow 2.x but I do not checked all constraints only latests.

There is some different behaviour with pendulum 3 vs pendulum 2:

String representation:

Pendulum 2

Has T separator between date and time

>>> import pendulum
>>> str(pendulum.now())
'2023-12-18T23:14:57.788852+04:00'

Pendulum 3

Has (1 whitespace) separator between date and time, same as datetime.datetime

>>> import pendulum
>>> str(pendulum.now())
'2023-12-18 23:14:57.788852+04:00'

UTC serialised for timezone in REST API

Pendulum 2

"timezone":  "Timezone('UTC')",

Pendulum 3

"timezone": "UTC",

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

bolkedebruin · 2023-12-18T16:32:24Z

What happens if serialization (serde) encounters a OLD version of a pendulum timezone? I havent verified it myself but I assume the class names have changed?

Taragolis · 2023-12-18T16:39:17Z

Full qualified path is the same. The main difference how UTC timezone works:

In pendulum 2 it pendulum.tz.timezone.FixedTimezone, ~~so it deserialise into this type if it serialised in previous version~~ nevermind we serialise offset=0 to UTC
In pendulum 3 it pendulum.tz.timezone.Timezone

Taragolis · 2023-12-18T16:47:17Z

Anyway It is a good point to add this kind of tests.

tests/api_connexion/endpoints/test_dag_endpoint.py

tests/cli/commands/test_dag_command.py

airflow/example_dags/plugins/workday.py

bolkedebruin · 2023-12-21T11:53:50Z

I like it :-)

Taragolis · 2023-12-23T18:54:03Z

Seems like it close to backcompat with 2, just need to resolve remaining static checks

.github/workflows/ci.yml

Taragolis · 2023-12-24T11:55:20Z

Seems like all major stuff are resolved, so this PR could be reviewed.
The only things are required is add is newsfragment file

airflow/providers/cncf/kubernetes/pod_launcher_deprecated.py

airflow/serialization/serialized_objects.py

potiuk · 2023-12-30T10:50:44Z

Looks green. Does it mean we have it ??

bolkedebruin · 2024-01-06T09:48:47Z

I think we should bite the bullet and start fixing stuff that might be affected. So yes we should have it.

Taragolis · 2024-01-09T12:15:51Z

I will rebase and add release notes today if I feel better

Taragolis · 2024-01-10T14:35:54Z

Just wondering shall we merge it or better to wait for someone else review?

Taragolis · 2024-01-11T00:00:58Z

Fortunately All failures after latest rebase not relevant to changes from this PR and should be fixed by #36728

Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

potiuk

This one looks FANTASTIC

scripts/docker/entrypoint_ci.sh

hussein-awala

Looks great! LGTM

potiuk · 2024-01-12T10:28:38Z

🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉

* Add support of Pendulum 3 * Add backcompat to pendulum 2 * Update airflow/serialization/serialized_objects.py Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * Add newsfragments --------- Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> (cherry picked from commit 2ffa6e4)

* Add support of Pendulum 3 * Add backcompat to pendulum 2 * Update airflow/serialization/serialized_objects.py Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * Add newsfragments --------- Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

boring-cyborg bot added area:core-operators Operators, Sensors and hooks within Core Airflow area:serialization labels Dec 18, 2023

edgarrmondragon mentioned this pull request Dec 18, 2023

Pendulum doesn't support pep 517 builds sdispater/pendulum#781

Closed

Taragolis commented Dec 18, 2023

View reviewed changes

tests/api_connexion/endpoints/test_dag_endpoint.py Outdated Show resolved Hide resolved

tests/cli/commands/test_dag_command.py Outdated Show resolved Hide resolved

uranusjr reviewed Dec 19, 2023

View reviewed changes

airflow/example_dags/plugins/workday.py Outdated Show resolved Hide resolved

Taragolis force-pushed the pendulum-3 branch from 91763bd to 5681d12 Compare December 19, 2023 09:53

Taragolis force-pushed the pendulum-3 branch from edd2d32 to ccb2e4e Compare December 23, 2023 14:51

Taragolis mentioned this pull request Dec 23, 2023

Use pre-release version of pendulum in CI and breeze #35798

Closed

Taragolis force-pushed the pendulum-3 branch from 59e9973 to 3e7f50e Compare December 24, 2023 09:52

potiuk reviewed Dec 24, 2023

View reviewed changes

.github/workflows/ci.yml Show resolved Hide resolved

Taragolis marked this pull request as ready for review December 24, 2023 11:55

Taragolis requested review from ashb, jedcunningham, kaxil, hussein-awala, bolkedebruin and XD-DENG as code owners December 24, 2023 11:55

Taragolis added this to the Airflow 2.8.1 milestone Dec 24, 2023

uranusjr reviewed Dec 24, 2023

View reviewed changes

airflow/providers/cncf/kubernetes/pod_launcher_deprecated.py Show resolved Hide resolved

uranusjr reviewed Dec 24, 2023

View reviewed changes

airflow/serialization/serialized_objects.py Outdated Show resolved Hide resolved

Taragolis force-pushed the pendulum-3 branch from 2be8e29 to e623348 Compare December 27, 2023 10:29

potiuk mentioned this pull request Jan 2, 2024

TypeError: 'module' object is not callable #36524

Closed

2 tasks

Taragolis mentioned this pull request Jan 3, 2024

Airflow Python 3.12 support #36546

Closed

2 tasks

Taragolis mentioned this pull request Jan 4, 2024

Error when opening Airflow - Something bad has happened. #36560

Closed

2 tasks

bolkedebruin approved these changes Jan 6, 2024

View reviewed changes

Taragolis force-pushed the pendulum-3 branch from e623348 to 669d9e1 Compare January 9, 2024 15:43

Taragolis force-pushed the pendulum-3 branch 2 times, most recently from ab31119 to 3caa58d Compare January 10, 2024 21:49

Taragolis and others added 4 commits January 11, 2024 15:14

Add support of Pendulum 3

d8cd7a9

Add backcompat to pendulum 2

63ddfd5

Update airflow/serialization/serialized_objects.py

b0e7220

Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

Add newsfragments

a9a7ac6

Taragolis force-pushed the pendulum-3 branch from 3caa58d to a9a7ac6 Compare January 11, 2024 11:14

potiuk approved these changes Jan 11, 2024

View reviewed changes

potiuk reviewed Jan 11, 2024

View reviewed changes

scripts/docker/entrypoint_ci.sh Show resolved Hide resolved

hussein-awala approved these changes Jan 11, 2024

View reviewed changes

potiuk mentioned this pull request Jan 11, 2024

Pendulum not importing correctly with certain lib #36741

Closed

2 tasks

Taragolis merged commit 2ffa6e4 into main Jan 12, 2024
80 checks passed

Taragolis deleted the pendulum-3 branch January 12, 2024 10:27

ephraimbuddy added the type:misc/internal Changelog: Misc changes that should appear in change log label Jan 15, 2024

ephraimbuddy mentioned this pull request Jan 16, 2024

Status of testing of Apache Airflow 2.8.1rc1 #36808

Closed

68 tasks

potiuk mentioned this pull request Jan 26, 2024

Status of testing Providers that were prepared on January 26, 2024 #36948

Closed

sean-rose mentioned this pull request Mar 1, 2024

feat: Upgrade Airflow from 2.7.3 to 2.8.2 mozilla/telemetry-airflow#1935

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support of Pendulum 3 #36281

Add support of Pendulum 3 #36281

Taragolis commented Dec 18, 2023 •

edited

bolkedebruin commented Dec 18, 2023

Taragolis commented Dec 18, 2023 •

edited

Taragolis commented Dec 18, 2023

bolkedebruin commented Dec 21, 2023

Taragolis commented Dec 23, 2023 •

edited

Taragolis commented Dec 24, 2023

potiuk commented Dec 30, 2023

bolkedebruin commented Jan 6, 2024

Taragolis commented Jan 9, 2024

Taragolis commented Jan 10, 2024

Taragolis commented Jan 11, 2024

potiuk left a comment

hussein-awala left a comment

potiuk commented Jan 12, 2024

Add support of Pendulum 3 #36281

Add support of Pendulum 3 #36281

Conversation

Taragolis commented Dec 18, 2023 • edited

There is some different behaviour with pendulum 3 vs pendulum 2:

String representation:

UTC serialised for timezone in REST API

bolkedebruin commented Dec 18, 2023

Taragolis commented Dec 18, 2023 • edited

Taragolis commented Dec 18, 2023

bolkedebruin commented Dec 21, 2023

Taragolis commented Dec 23, 2023 • edited

Taragolis commented Dec 24, 2023

potiuk commented Dec 30, 2023

bolkedebruin commented Jan 6, 2024

Taragolis commented Jan 9, 2024

Taragolis commented Jan 10, 2024

Taragolis commented Jan 11, 2024

potiuk left a comment

Choose a reason for hiding this comment

hussein-awala left a comment

Choose a reason for hiding this comment

potiuk commented Jan 12, 2024

Taragolis commented Dec 18, 2023 •

edited

Taragolis commented Dec 18, 2023 •

edited

Taragolis commented Dec 23, 2023 •

edited