Status of testing of Apache Airflow 2.8.2rc3

[ephraimbuddy]

Body

We are kindly requesting that contributors to Apache Airflow RC 2.8.2rc3 help test the RC.

Please let us know by commenting if the issue is addressed in the latest RC.

• Base date for fetching dag grid view must include selected run_id (#34887): @hterik
• Move IMPORT_ERROR from DAG related permissions to view related permissions (#37292): @vincbeck
• fix: sort dag processing stats by last_runtime (#37302): @Narendra-Neerukonda
• Check permissions for ImportError (#37468): @jedcunningham
• Allow pre-population of trigger form values via URL parameters (#37497): @jscheffl @jsgh7276
  Linked issues:
  • DAG triggering page does not reflect conf parameter in the URL query string (#32859)
• Fix few small release issues found during 2.8.2 preparation (#37633): @potiuk
• Sanitize the conn_id to disallow potential script execution (#32867): @andylamp @potiuk
  Linked issues:
  • Connection ID is not sanitized and you can self - javascript yourself (#32770)
• Displaying "actual" try number in TaskInstance view (#34635): @SamWheating
• Fix task id copy button copying wrong id (#34904): @ByPort @Adaverse
  Linked issues:
  • Task ID copy button copying wrong ID (#34846)
• Change AirflowTaskTimeout to inherit BaseException (#35653): @hterik @potiuk
  Linked issues:
  • AirflowTaskTimeout should inherit BaseException instead of Exception? (#35644)
• Fix the bug that affected the DAG end date. (#36144): @avkirilishin
• Fix broken regex for allowed_deserialization_classes (#36147): @barrysteyn @bolkedebruin @tobiaszorzetto
  Linked issues:
  • The allowed_deserialization_classes regex is broken (#34093)
  • Make allowed_deserialization_classes more intuitive (#28829)
• Updated config.yml for environment variable sql_alchemy_connect_args (#36526): @koushik-rout-samsung
• Fix security manager inheritance in fab provider (#36538): @maiconkkl @vincbeck @kn3609571
  Linked issues:
  • Fix Exception FabAirflowSecurityManagerOverride (#36437)
  • fix inheritance checking of security manager in FabAuthManager (#36343)
• Secret masker ignores passwords with special chars (#36692): @Rurushu13 @aritra24
  Linked issues:
  • Airflow does not mask secrets in Database URI's when secret has a special character. (#36688)
• Fixing status icon alignment for various views (#36804): @amoghrajesh
• REST API set description on POST to /variables endpoint (#36820): @ryandutton @Taragolis
  Linked issues:
  • Variables create API does not accept description (#36811)
• Fix webserver always redirecting to home page if user was not logged in (#36833): @csp33
  Linked issues:
  • Webserver redirects to main page when user is not logged in (#36832)
• Fix copy button on a DAG run's config (#36855): @im-perativa @anteverse
  Linked issues:
  • Copy button on a Dag run's config copies only one run config (#36848)
• Fix bug introduced by replacing spaces by + in run_id (#36877): @ephraimbuddy
• Limit flask-session to <0.6 (#36895): @Taragolis
• Bump min version of Alembic to 1.13.1 (#36928): @Tenzer @potiuk
  Linked issues:
  • Airflow 2.8.1 contains migration requiring SQLAlchemy 2+ (#36912)
• Disable dryrun autofetch for ClearInstance and MarkInstance (#36941): @sktemkar16 @im-perativa
  Linked issues:
  • Audit logs entry as clear event even if we click on the tasks. (#36119)
• fix: DagRuns with UPSTREAM_FAILED tasks get stuck in the backfill. (#36954): @charleschang0531 @avkirilishin
  Linked issues:
  • dag stuck in "running" status when running in backfill job (#34448)
• Handle SystemExit raised in the task. (#36986): @avkirilishin @vlieven
  Linked issues:
  • ShortCircuitOperator does not properly retry when python_callable exits unexpectedly (#36809)
• add description to queue_when (#36997): @devscheffer
• Remove superfluous @Sentry.enrich_errors (#37002): @jkramer-ginkgo @Lee-W
  Linked issues:
  • fix task_instance can not get task attribute correctly when turning on sentry (#36968)
  • Scheduler exceptions when Sentry and schedule_after_task_execution are enabled (#36957)
• Use execution_date= param as a backup to base date for grid view (#37018): @hterik @bbovenzi
  Linked issues:
  • Base date for fetching dag grid view must include selected run_id (#34887)
• Upgrade yarn stack in pre-commits (#37048): @potiuk
• Bugfix Triggering DAG with parameters is mandatory when show_trigger_form_if_no_params is enabled (#37063): @jscheffl @csp33
  Linked issues:
  • Triggering DAG with parameters is mandatory when show_trigger_form_if_no_params is enabled (#36843)
• Bump aiohttp min version to avoid CVE-2024-23829 and CVE-2024-23334 (#37110): @hussein-awala
• Switch instance type to m7g (#37127): @potiuk
• Allow running airflow against sqlite in-memory DB for tests (#37144): @potiuk @ashb
  Linked issues:
  • Better error message when sqlite URL uses relative path (#36774)
• Revert the sequence of initializing configuration defaults (#37155): @potiuk
• Avoid pendulum.from_timestamp usage (#37160): @Taragolis
• treeview - deterministic and new getter (#37162): @raphaelauv
• fix: PythonVirtualenvOperator crashes if any python_callable function is defined in the same source as DAG (#37165): @Felix-neko @rawwar
  Linked issues:
  • PythonVirtualenvOperator crashes if any python_callable function is defined in the same source as DAG (#35529)
• Fix Airflow serialization for namedtuple (#37168): @aru-trackunit @Joffreybvn
  Linked issues:
  • DatabricksSQLOperator struggles with parsing the data (#36839)
• Fix collapsed grid width and match selected row between grid/gantt (#37205): @bbovenzi
• Resolve handler close race condition at triggerer shutdown (#37206): @dstandish
• Adjust graph node layout (#37207): @bbovenzi
• Avoid log-groomer container failures when log persistence is enabled (#37222): @arovira @ckljohn
  Linked issues:
  • Empty log folders are not removed when clean up (#33162)
  • log-groomer container crashing with .Values.logs.persistence enabled (#37220)
• Upgrade to FAB 4.3.11 (#37233): @potiuk
• Upgrade to just released pip 24.0 (#37236): @potiuk
• Improve task names in graph view (#37254): @bbovenzi
• Fix the value of dateTimeAttrFormat constant (#37285): @Taragolis @Satoshi-Sh
  Linked issues:
  • Momento Warning (#37281)
• Fix permission check on DAGs when access_entity is specified (#37290): @vincbeck
• Fix permissions of parent folders for log file handler (#37310): @matejmurin01 @potiuk
  Linked issues:
  • Permissions of new log directory not being set to parent folders. (#37200)
• Limit universal-pathlib to < 0.2.0 (#37311): @potiuk
• Remove unused pageSize method. (#37319): @potiuk
• Turn Pydantic into an optional dependency (#37320): @potiuk
• Add more-itertools as dependency of common-sql (#37359): @potiuk
• Remove flakiness introduced by cleanup in configuration test (#37375): @potiuk
  Linked issues:
  • Turn Pydantic into an optional dependency (#37320)
• Include airflow_pre_installed_providers.txt into sdist distribution (#37388): @Taragolis
• Fix bug with clicking url-unsafe tags (#37395): @patgarz
• Remove markdown from core dependencies (#37396): @Taragolis
• Fix intermittent failures coming from buildkit parallel builds (#37403): @potiuk
• Add Python 3.12 exclusions in providers/pyproject.toml (#37404): @potiuk
  Linked issues:
  • Python3.12 (#36755)
• Uninstall twine before installing it in CI (#37415): @potiuk
• Fix typo (copy&paste victim) introduced in twine fix (#37428): @potiuk
  Linked issues:
  • Uninstall twine before installing it in CI (#37415)
• Replace other Python 3.11 and 3.12 deprecations (#37478): @Taragolis
• Revoking audit_log permission from all users except admin (#37501): @amoghrajesh
• Change margin to padding so first task can be selected (#37527): @vchiapaikeo
• Fix wrong test after fixing audit log permission (#37547): @amoghrajesh @potiuk
  Linked issues:
  • Revoking audit_log permission from all users except admin (#37501)
• Fix failing home view test after changing audit log permissions (#37550): @amoghrajesh @potiuk
  Linked issues:
  • Revoking audit_log permission from all users except admin (#37501)
• Revert "Fix future DagRun rarely triggered by race conditions when max_active_runs reached its upper limit. (#31414)" (#37596): @ephraimbuddy
• Fix regression on trigger form error display on form validation: @jscheffl

Thanks to all who contributed to the release (probably not a complete list!):
@devscheffer @kn3609571 @Joffreybvn @arovira @im-perativa @Taragolis @potiuk @bolkedebruin @ephraimbuddy @jedcunningham @patgarz @bbovenzi @Lee-W @SamWheating @koushik-rout-samsung @rawwar @raphaelauv @vlieven @maiconkkl @ByPort @vincbeck @amoghrajesh @Adaverse @dstandish @csp33 @jkramer-ginkgo @ckljohn @jsgh7276 @ashb @anteverse @Rurushu13 @sktemkar16 @aru-trackunit @tobiaszorzetto @jscheffl @avkirilishin @barrysteyn @Felix-neko @matejmurin01 @andylamp @hterik @Narendra-Neerukonda @Tenzer @aritra24 @hussein-awala @vchiapaikeo @charleschang0531 @Satoshi-Sh @ryandutton

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[patgarz]

#37395 confirmed

[jscheffl]

#37063 confirmed.

But I was expecting #37497 also be in 2.8.2 but I miss this in the RC. Mistake or by intend? (then milestone label is not correct)

[andylamp]

#32867 seems to work as expected.

[vchiapaikeo]

#37527 is fine

[Satoshi-Sh]

#37285 confirmed.

[Satoshi-Sh]

In case, someone doesn't not know how to run this version. (I asked Yarek how to do it.)

https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md#verify-the-release-candidate-by-contributors

[amoghrajesh]

Tested my changes

• Fixing status icon alignment for various views #36804 is not working as expected, not sure why. The changes are present here already https://github.com/apache/airflow/blob/v2-8-stable/airflow/www/static/js/dag/StatusBox.tsx
• Revoking audit_log permission from all users except admin #37501 needs the FAB provider changes, so we can probably only test it with that. How we do approach such cases @ephraimbuddy? Same applies for Fix failing home view test after changing audit log permissions #37550 and Fix wrong test after fixing audit log permission (Fix wrong test after fixing audit log permission #37547)

[ephraimbuddy]

Tested my changes

• Fixing status icon alignment for various views #36804 is not working as expected, not sure why. The changes are present here already https://github.com/apache/airflow/blob/v2-8-stable/airflow/www/static/js/dag/StatusBox.tsx
• Revoking audit_log permission from all users except admin #37501 needs the FAB provider changes, so we can probably only test it with that. How we do approach such cases @ephraimbuddy? Same applies for Fix failing home view test after changing audit log permissions #37550 and Fix wrong test after fixing audit log permission (Fix wrong test after fixing audit log permission #37547)

@amoghrajesh , the #37501 was cherry-picked differently by @potiuk so it works in 2.8.2 without the provider, you can test it. Please check that the fix works. The same applies to the related PRs.

#36804 is not a blocker. It's minor

[Lee-W]

#37002 confirmed

[amoghrajesh]

@ephraimbuddy Fresh installed the RC again and tested my changes after resetting the DB.

• Revoking audit_log permission from all users except admin #37501 works as expected, a viewer user is not able to see audit logs. (related PRs too)
• Fixing status icon alignment for various views #36804 also works as expected

Btw, this was also cherry picked: #37507 and it works as expected.

[aritra24]

#36692: Confirmed, works as expected now.

[anteverse]

#36855

Tested with 3.11, 3.10, 3.9 & 3.8. Works like a charm.

[potiuk]

Tested and marked all mine (and few others). Checked that release process works now well without flaws (#37633) - found a few small things to correct in verification process - but they are irrelvant for the relase #37656

All looks good!

[kn3609571]

#36538 (related to #36437, #36343) seems to work as expected.

[im-perativa]

#36855 and #36941 Confirmed 🚀

[jscheffl]

#37063 confirmed.

But I was expecting #37497 also be in 2.8.2 but I miss this in the RC. Mistake or by intend? (then milestone label is not correct)

Tested and all PRs labelled are not correctly in RC2 - but found (late) a small UI regression. Created PR #37672 as a fix.

[potiuk]

I guess "are not correctly in RC2" -> "are now correctly" in RC2 :) ?

I guess the regression is small enough to not require RC3?

[jscheffl]

Tested RC3 now and can confirm all stuff is working (wohooo), also #37675

[potiuk]

Tested rc3 - looks good. Also tested that Docker Image for rc3 contains latest docker binary (added in #37651 )

[vatsrahul1001]

Tested RC3, LGTM :)
Below is the list of items I have tested:

1. Tested Regression DAG with Celery and k8s executor.
2. Tested issue 37497 372027 37254 37205
3. UI sanity.

[jsgh7276]

Tested issue 37497 and worked as I expected!
Thanks a lot to @jscheffl and all related contributors.

[Lee-W]

#37002 confirmed on rc3

[ephraimbuddy]

Apache Airflow 2.8.2 has been released. Thank you all for testing this release

[potiuk]

2.8.2 to be precise :) . From RC3

[ephraimbuddy]

2.8.2 to be precise :) . From RC3

Ah, mistake. Correcting that