Status of testing of Apache Airflow 2.5.2rc2

[pierrejeambrun]

We have a kind request for all the contributors to the latest Apache Airflow RC 2.5.2rc2.

Could you please help us to test the RC versions of Airflow?

Please let us know in the comment if the issue is addressed in the latest RC.

• Fix inconsitencies in checking edit permissions for a DAG (#20346): @kaxil
• logging poke info when external dag is not none and task_id and task_ids are none (#28097): @surabathini
• Migrate remaining core sensors tests to pytest (#28204): @potiuk @Taragolis
  Linked issues:
  • Migrate remaining core sensors tests to pytest (#28177)
  • Revert "Migrate remaining core sensors tests to pytest (#28177)" (#28196)
• Fix manual task trigger failing for k8s. (#28394): @snjypl
• Fix bugs introduced when moving the code of issue generation to breeze (#28683): @potiuk
  Linked issues:
  • Move provider issue generation to breeze (#28352)
• introduce dag processor job fix #27140 (#28799): @farhan0syakir
• Throttle streaming log reads (#28818): @dstandish
• Bump swagger-ui-dist from 3.52.0 to 4.1.3 in /airflow/www (#28824): @dependabot[bot]
• Annotate KubeExecutor pods that we don't delete (#28844): @jedcunningham
• KubenetesExecutor sends state even when successful (#28871): @jedcunningham
• Add dep context description for better log message (#28875): @dstandish
• Be more selective when adopting pods with KubernetesExecutor (#28899): @jedcunningham
• Allow URI without authority and host blocks in airflow connections add (#28922): @Taragolis
• Update how PythonSensor returns values from python_callable (#28932): @SoxMax
• Resolve all variables in pickled XCom iterator (#28982): @uranusjr
• Write action log to DB when DAG run is trigged via API (#28998): @kolfild26
• Adding dataset to be discovered by static analyzers (#29007): @bharat99k
• Fix leak sensitive field via V1EnvVar on exception (#29016): @ecerulm
• Sanitize url_for arguments before they are passed (#29039): @potiuk
• Clarify graph view dynamic task labels (#29042): @bbovenzi
• Fix kerberos authentication for the REST API. (#29054): @potiuk
• Handling error on cluster policy itself (#29056): @okayhooni
• Check for run_id url param when linking to graph/gantt views (#29066): @bbovenzi
• Emit dagrun duration when it fails because of a timeout (#29076): @yangguoaws @hussein-awala
  Linked issues:
  • Metrics dagrun.duration.failed.<dag_id> not updated when the dag run failed due to timeout (#29013)
• Remove upper bound limitation for pytest (#29086): @auvipy
• Fix warning in migrations about old config. (#29092): @ashb
• Make options output of breeze command consistent and complete (#29097): @potiuk
• Fix to run tasks on Dask Worker for Windows systems (#29107): @casra-developers
• Fix access to context in functions decorated by task.sensor (#29146): @Thomas-McKanna
  Linked issues:
  • Fix access to context in functions decorated by task.sensor (#29137)
• Bump ua-parser-js from 0.7.31 to 0.7.33 in /airflow/www (#29172): @dependabot[bot]
• Fixing Task Duration view in case of manual DAG runs only (#22015) (#29195): @duke8585
• Fix Trigger DAG w/config origin url (#29212): @hussein-awala @enriqueayala
  Linked issues:
  • Trigger DAG w/config raising error from task detail views (#29197)
• Fix dag run trigger with a note. (#29228): @vedantlodha @ttben
  Linked issues:
  • Bad request when triggering dag run with note in payload (#28825)
• Clarify description of worker control config (#29247): @potiuk
• Increase max height for grid view elements (#29367): @bbovenzi
• Fix grid logs for large logs (#29390): @bbovenzi
• Fix validation of date-time field in API and Parameter schemas (#29395): @potiuk
• Persist DAG and task doc values in TaskFlow API if explicitly set (#29399): @josh-fell
• Ensure Serialized DAG is deleted (#29407): @michaelmicheal
• Fix Rest API update user output (#29409): @vincbeck
• Edgemodifier refactoring w/ labels in TaskGroup edge case (#29410): @vchiapaikeo
• datasets, next_run_datasets, remove unnecessary timestamp filter (#29441): @tseruga @michaelmicheal
  Linked issues:
  • Dataset Next Trigger Modal Not Populating Latest Update (#26892)
• Fix nested fields rendering in mapped operators (#29451): @pshrivastava27 @hussein-awala @bdsoha
  Linked issues:
  • Jinja templating doesn't work with container_resources when using dymanic task mapping with Kubernetes Pod Operator (#29432)
  • Mapped KubernetesPodOperater not rendering nested templates (#25588)
• Make prev_logical_date variable offset-aware (#29454): @akrava
• Use newer setuptools v67.2.0 (#29465): @lewijw @arjunanan6
  Linked issues:
  • Require newer version of pypi/setuptools to remove security scan issue (CVE-2022-40897) (#29428)
• Require explicit flag to remove MyPy cache when running breeze stop (#29493): @potiuk
  Linked issues:
  • Also remove mypy cache when stopping breeze (#29184)
• Change permissions of config/password files created by airflow (#29495): @potiuk
• Do not show version/node in UI traceback for unauthenticated user (#29501): @potiuk
• Remove section check from get-value command (#29541): @Limess @potiuk
  Linked issues:
  • Docker image fails to start if celery config section is not defined (#29537)
• Don't display empty rendered attrs in Task Instance Details page (#29545): @josh-fell @lorenzomagni
  Linked issues:
  • Hide non-used docs attributes from Task Instance Detail (#29515)
• Change expose_hostname default to false (#29547): @jedcunningham
• Bump undici from 5.9.1 to 5.19.1 in /airflow/www (#29583): @dependabot[bot]
• fix clear dag run openapi spec responses by adding additional return type (#29600): @aliotta
• convert moment with timezone to UTC instead of raising an exception (#29606): @Gollum999 @hussein-awala
  Linked issues:
  • DateTimeSensorAsync breaks if target_time is timezone-aware (#29576)
• Fix Scheduler crash when clear a previous run of a normal task that is now a mapped task (#29645): @ephraimbuddy
• Move extra links position in grid view (#29703): @bbovenzi
• Fix on_failure_callback when task receives a SIGTERM (#29743): @ephraimbuddy
• DAG list sorting lost when switching page (#29756): @alejandrofm @amoghrajesh
  Linked issues:
  • DAG list, sorting lost when switching page (#29322)
• Improve DAG loading in Graph view when using Dynamic Tasks (#29791): @Josersanvil @jose-workpath
  Linked issues:
  • DAG loading very slow in Graph view when using Dynamic Tasks (#27483)
• Add a check for not templateable fields (#29821): @hussein-awala @cosinequanon
  Linked issues:
  • DAG fails serialization if template_field contains execution_timeout (#29819)
• Updating dag run endpoint to 404 if requested dag is inactive (#29860): @SamWheating
• Update min version of python-deamon to fix containerd file limits (#29916): @antonio-antuan @potiuk
  Linked issues:
  • high memory leak, cannot start even webserver (#29841)
• Limit importlib-metadata backport to < 5.0.0 (Limit importlib-metadata backport to < 5.0.0 #29924, Also limit importlib on Python 3.9 #30069) @potiuk

Thanks to all who contributed to the release (probably not a complete list!):
@Gollum999 @snjypl @auvipy @amoghrajesh @antonio-antuan @bharat99k @vchiapaikeo @SamWheating @jose-workpath @tseruga @akrava @Josersanvil @Thomas-McKanna @ashb @SoxMax @bdsoha @vedantlodha @lewijw @hussein-awala @arjunanan6 @dependabot[bot] @yangguoaws @kaxil @Limess @surabathini @jedcunningham @dstandish @pshrivastava27 @ecerulm @ttben @casra-developers @josh-fell @duke8585 @ephraimbuddy @michaelmicheal @farhan0syakir @okayhooni @lorenzomagni @bbovenzi @kolfild26 @potiuk @aliotta @alejandrofm @cosinequanon @enriqueayala @uranusjr @vincbeck @Taragolis

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[auvipy]

how should I report?

[amoghrajesh]

@pierrejeambrun what is the best way to test this?

[pierrejeambrun]

@auvipy you can directly report in this issue your findings. And if the change (PR) you tested is working as expected or not. (Or any other issue you noticed in the release candidate)

@amoghrajesh The easiest way is to install the release candidate in a fresh virtualenv, then run airflow in standalone mode and test changes there. (If you have a dev environment for airflow you can also upgrade there and see if everything is working as expected)

[vchiapaikeo]

#29410 is still looking good.

We did see an unrelated issue from #29821 but that seems to be a user error on our end that we need to correct. Thank you @hussein-awala for that fix!

[amoghrajesh]

#29756 looks good on the RC build. Performed small multiple tests, one snippet attached to this comment. @pierrejeambrun

RC_testing.mov

[potiuk]

• Migrate remaining core sensors tests to pytest #28204, Fix bugs introduced when moving the code of issue generation to breeze #28683, Make options output of breeze command consistent and complete #29097. Require explicit flag to remove MyPy cache when running breeze stop #29493, - were just CI/test modifications
• Sanitize url_for arguments before they are passed #29039 looks good (sanitization is there)
• Clarify description of worker control config #29247 - updated description is there
• Fix kerberos authentication for the REST API. #29054 - I'd appreciate @BMFH if you could check if it works with your kerberos installation

To be continued (have to hurry now)

[potiuk]

• Fix validation of date-time field in API and Parameter schemas #29395 - correctly validates badly formatted time
• Change permissions of config/password files created by airflow #29495 - is good:
  Both standard config and standalone password file have user-only permissions:

^Xroot@f8d4a86cf3ef:~/airflow# ls -la airflow.cfg
-rw------- 1 root root 51764 Mar 12 11:36 airflow.cfg

⌁ ⌂23.88 [jarek:~/airflow] [airflow-astroid] % ls -la standalone_admin_password.txt
-rw-------  1 jarek  staff  16 Mar 12 12:48 standalone_admin_password.txt

• Do not show version/node in UI traceback for unauthenticated user #29501 - the details are redacted when user is not authenticated

• Remove section check from get-value command #29541 works fine when no celery section is defined in cfg we get now:

> airflow config get-value celery broker_url
redis://redis:6379/0

rather than exit 1 and:

> airflow config get-value celery broker_url
The section [celery] is not found in config.

• Update min version of python-deamon to fix containerd file limits #29916 - I've checked that python-daemon installed is 3.0.1 which is good (also to note 3.0.0 that fixed the problem had been yanked by the maintainer as it did not work without setuptools (and it uses packaging now instead). But that should keep the fix released in 3.0.0. @antonio-antuan maybe you could check if the 2.5.2.rc1 fixes the memory issue in your setup (I have no access to my test machine I set it up for testin and travelling for the next few days).

[potiuk]

All looks good for my fixes - except the two issues where it "looks" good but I cannot test it easily and I would love to get confirmation :)

[hussein-awala]

I've tested #29076, #29212, #29451, #29606, #29821 and they all look good.

[BMFH]

I've checked #29054. It works fine.

[Josersanvil]

Tested #29791 in the 2.5.2rc1 docker tag and it looks good 🙂

[josh-fell]

#29399 and #29545 look good!

[bbovenzi]

#29066, #29703, #29367, #29042, and #29600 look good

[SamWheating]

Tested #29860 in breeze after pulling the 2.5.2rc1 tag, looks like it works as expected.

Tried to create a DAGRun from an inactive DAG, and got a 404 rather than a 500:

curl -X POST localhost:8080/api/v1/dags/data_warehouse_dag_o259owoo3s/dagRuns -d '{"dag_run_id":"string2","logical_date":"2019-08-24T14:15:24Z","execution_date":"2019-08-24T14:15:24Z","conf":{},"note":"strings"}'  -H 'Content-Type: application/json'

{
  "detail": "DAG with dag_id: 'data_warehouse_dag_o259owoo3s' not found",
  "status": 404,
  "title": "DAG not found", 
  "type": "https://airflow.apache.org/docs/apache-airflow/2.5.2/stable-rest-api-ref.html#section/Errors/NotFound"                       
}

[pierrejeambrun]

A bug was found in rc1 that necessitate an rc2 (constraints). I think we can safely assume that what was tested for rc1 is still working in rc2. Feel free to do more testing. The issue has been updated to target 2.5.2rc2

[akrava]

Tested #29454 in 2.5.2rc2, looks good.

[okayhooni]

Tested #29056 (minor fix for handling error on cluster policy itself) and looks good.

But i think it's better to use repr(e) than str(e) for showing error class name on web UI message.

• https://github.com/apache/airflow/blob/2.5.2rc2/airflow/models/dagbag.py#L439

self.import_errors[dag.fileloc] = str(e) -> self.import_errors[dag.fileloc] = repr(e)

could I modify this line now?

• tiny 1-line fix MR: Add exception class name to DAG-parsing error message #30105

[pierrejeambrun]

Thanks to everyone that helped testing 2.5.2rc1 and 2.5.2rc2. 🙏

Closing as 2.5.2 has been released based on 2.5.2rc2 🎉

[josh-fell]

🎉 Thanks for organizing @pierrejeambrun!