-
Notifications
You must be signed in to change notification settings - Fork 179
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update dependency apache-airflow to v2.9.0, Python to 3.12 #4159
Conversation
CC @stacimc since we were likely going to deploy the catalog today 😮 I'll jump on this to see why the build is failing |
It looks like the |
Edited/Blocked NotificationRenovate will not automatically rebase this PR, because it does not recognize the last commit author and assumes somebody else may have edited the PR. You can manually request rebase by checking the rebase/retry box above. ⚠ Warning: custom changes will be lost. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Verified test fixes as well!
This PR contains the following updates:
==2.8.4
->==2.9.0
GitHub Vulnerability Alerts
CVE-2024-31869
Airflow versions 2.7.0 through 2.8.4 have a vulnerability that allows an authenticated user to see sensitive provider configuration via the "configuration" UI page when "non-sensitive-only" was set as "webserver.expose_config" configuration (The celery provider is the only community provider currently that has sensitive configurations). You should migrate to Airflow 2.9 or change your "expose_config" configuration to False as a workaround. This is similar, but different to CVE-2023-46288 GHSA-9qqg-mh7c-chfq which concerned API, not UI configuration page.
Release Notes
apache/airflow (apache-airflow)
v2.9.0
Compare Source
Significant Changes
^^^^^^^^^^^^^^^^^^^
Following Listener API methods are considered stable and can be used for production system (were experimental feature in older Airflow versions) (#36376):
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Lifecycle events:
on_starting
before_stopping
DagRun State Change Events:
on_dag_run_running
on_dag_run_success
on_dag_run_failed
TaskInstance State Change Events:
on_task_instance_running
on_task_instance_success
on_task_instance_failed
Support for Microsoft SQL-Server for Airflow Meta Database has been removed (#36514)
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
After
discussion <https://lists.apache.org/thread/r06j306hldg03g2my1pd4nyjxg78b3h4>
__and a
voting process <https://lists.apache.org/thread/pgcgmhf6560k8jbsmz8nlyoxosvltph2>
__,the Airflow's PMC and Committers have reached a resolution to no longer maintain MsSQL as a supported Database Backend.
As of Airflow 2.9.0 support of MsSQL has been removed for Airflow Database Backend.
A migration script which can help migrating the database before upgrading to Airflow 2.9.0 is available in
airflow-mssql-migration repo on Github <https://github.com/apache/airflow-mssql-migration>
_.Note that the migration script is provided without support and warranty.
This does not affect the existing provider packages (operators and hooks), DAGs can still access and process data from MsSQL.
Dataset URIs are now validated on input (#37005)
""""""""""""""""""""""""""""""""""""""""""""""""
Datasets must use a URI that conform to rules laid down in AIP-60, and the value
will be automatically normalized when the DAG file is parsed. See
documentation on Datasets <https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/datasets.html>
_ fora more detailed description on the rules.
You may need to change your Dataset identifiers if they look like a URI, but are
used in a less mainstream way, such as relying on the URI's auth section, or
have a case-sensitive protocol name.
The method
get_permitted_menu_items
inBaseAuthManager
has been renamedfilter_permitted_menu_items
(#37627)""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Add REST API actions to Audit Log events (#37734)
"""""""""""""""""""""""""""""""""""""""""""""""""
The Audit Log
event
name for REST API events will be prepended withapi.
orui.
, depending on if it came from the Airflow UI or externally.Official support for Python 3.12 (#38025)
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
There are a few caveats though:
Pendulum2 does not support Python 3.12. For Python 3.12 you need to use
Pendulum 3 <https://pendulum.eustace.io/blog/announcing-pendulum-3-0-0.html>
_Minimum SQLAlchemy version supported when Pandas is installed for Python 3.12 is
1.4.36
released inApril 2022. Airflow 2.9.0 increases the minimum supported version of SQLAlchemy to
1.4.36
for allPython versions.
Not all Providers support Python 3.12. At the initial release of Airflow 2.9.0 the following providers
are released without support for Python 3.12:
apache.beam
- pending onApache Beam support for 3.12 <https://github.com/apache/beam/issues/29149>
_papermill
- pending on Releasing Python 3.12 compatible papermill client versionincluding this merged issue <https://github.com/nteract/papermill/pull/771>
_Prevent large string objects from being stored in the Rendered Template Fields (#38094)
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
There's now a limit to the length of data that can be stored in the Rendered Template Fields.
The limit is set to 4096 characters. If the data exceeds this limit, it will be truncated. You can change this limit
by setting the
[core]max_template_field_length
configuration option in your airflow config.Change xcom table column value type to longblob for MySQL backend (#38401)
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Xcom table column
value
type has changed fromblob
tolongblob
. This will allow you to store relatively big data in Xcom but process can take a significant amount of time if you have a lot of large data stored in Xcom.To downgrade from revision:
b4078ac230a1
, ensure that you don't have Xcom values larger than 65,535 bytes. Otherwise, you'll need to clean those rows or runairflow db clean xcom
to clean the Xcom table.New Features
""""""""""""
Matomo
as an option for analytics_tool. (#38221)hashable
(#37465)queuedEvent
endpoint to get/delete DatasetDagRunQueue (#37176)DatasetOrTimeSchedule
(#36710)on_skipped_callback
toBaseOperator
(#36374)@task.bash
TaskFlow decorator (#30176, #37875)Improvements
""""""""""""
ExternalPythonOperator
use version fromsys.version_info
(#38377)run_id
column to log table (#37731)tryNumber
to grid task instance tooltip (#37911)ExternalPythonOperator
(#37409)Pathlike
(#36947)nowait
and skip_locked into with_row_locks (#36889)dag/dagRun
in the REST API (#36641)Connexion
from auth manager interface (#36209)Bug Fixes
"""""""""
total_entries
count on the event logs endpoint (#38625)tz
in next run ID info (#38482)chakra
styles to keepdropdowns
in filter bar (#38456)__exit__
is called in decorator context managers (#38383)BaseAuthManager.is_authorized_custom_view
abstract (#37915)/get_logs_with_metadata
endpoint (#37756)encoding
to the SQL engine in SQLAlchemy v2 (#37545)consuming_dags
attr eagerly before dataset listener (#36247)Miscellaneous
"""""""""""""
importlib_metadata
with compat to Python 3.10/3.12stdlib
(#38366)__new__
magic method of BaseOperatorMeta to avoid bad mixing classic and decorated operators (#37937)sys.version_info
for determine Python Major.Minor (#38372)blinker
add where it requires (#38140)> 39.0.0
(#38112)assert
outside of the tests (#37718)flask._request_ctx_stack
(#37522)login
attribute inairflow.__init__.py
(#37565)datetime.datetime.utcnow
byairflow.utils.timezone.utcnow
in core (#35448)is_authorized_cluster_activity
from auth manager (#36175)Doc Only Changes
""""""""""""""""
exception
to templates ref list (#36656)Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Enabled.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR has been generated by Renovate Bot.