Status of testing of Apache Airflow 2.2.4rc1

[jedcunningham]

Body

We have a kind request for all the contributors to the latest Apache Airflow RC 2.2.4rc1.

Could you please help us to test the RC versions of Airflow?

Please let us know in the comment if the issue is addressed in the latest RC.

• Fixing ses email backend (#18042): @ignaski @kaushikparanjpe @ecerulm @satishmittal111
  Linked issues:
  • Add from_{address,name} configuration parameters (#18200)
  • Emails notifications with AWS SES not working due to missing "from:" field (#17930)
  • Fixing SES backend - source (mail_from) must be string. (#16166)
  • Airflow module for email backend using AWS SES not resolving mail_from parameter from airflow config and is defaulted as None (#16778)
• Fix mismatch in generated run_id and logical date of DAG run (#18707): @davidcaron
• Use XCom to correctly display the execution date from triggered DAG in TriggerDagOperator's extra link (#19410): @o-nikolas @ljades
  Linked issues:
  • TriggerDagRunOperator External Link doesn't link with execution date to match to triggered DAG (#19264)
• Enhance multiple_outputs inference of dict typing (#19608): @josh-fell
  Linked issues:
  • TaskFlow API multiple_outputs inference not handling all flavors of dict typing (#19538)
• upgrade celery 5.2.3 (#19703): @auvipy
• Deprecate some functions in the experimental API (#19931): @ephraimbuddy
• Add possibility to create user in the Remote User mode Auth. (#19963): @lwyszomi
• Fix: Use seconds for schedule_delay metric when using dogstatsd (#19973): @lostkamp @coinzerge
  Linked issues:
  • Use seconds for schedule_delay metric (#18622)
  • datadog parsing error for dagrun.schedule_delay since it is not passed in float type (#18599)
• Adds retry on taskinstance retrieval lock (#20030): @potiuk @smartnewsdingli
  Linked issues:
  • Deadlock in worker pod (#19832)
• Lift off upper bound for MarkupSafe (#20113): @tsingh2k15
  Linked issues:
  • removing upper bound for markupsafe (#19761)
• Deprecate smart sensors (#20151): @jedcunningham
• Fix slow DAG deletion due to missing dag_id index for job table (#20282): @kushsharma
• Fix arrow styling in DAG dependencies view (#20303): @jon-fearer @BasPH
  Linked issues:
  • Arrows in DAG dependencies view are not consistent (#19989)
• Remove references to deepcopied object attributes (#20318): @BasPH
• Un-ignore DeprecationWarning (#20322): @uranusjr
  Linked issues:
  • Lazy Jinja2 context (#20217)
  • Prevent unexpected warnings in the code base (#19788)
• Fix Scheduler crash when executing task instances of missing DAG (#20349): @ephraimbuddy
• Remove unnecssary logging in experimental API (#20356): @kaxil
• fix(standalone): remove hardcoded webserver-port (#20429): @boredland
• Avoid calling DAG.following_schedule() for TaskInstance.get_template_context() (#20486): @lionheart106008 @uranusjr
  Linked issues:
  • Custom Timetable error with CeleryExecutor (#20475)
• Bump croniter from <1.1 to <1.2 (#20489): @ashwin153
• Removed duplicated dag_run join in Dag.get_task_instances() (#20591): @hubert-pietron
• Bump flask-appbuilder to >=3.3.4 (#20628): @jakubczaplicki
• bugfix: deferred tasks does not cancel when DAG is marked fail (#20649): @dungdm93
  Linked issues:
  • Triggers are not terminated when DAG is mark failed (#20580)
• Allow Viewing DagRuns and TIs if a user has DAG "read" perms (#20663): @kaxil
• Add Roles from Azure OAUTH Response in internal Security Manager (#20707): @Datkros
• Avoid unintentional data loss when deleting DAGs (#20758): @SamWheating
• Switch to non-vendored latest connexion library (#20910): @potiuk
• Fix session usage in /rendered-k8s view (#21006): @uranusjr @microhuang
  Linked issues:
  • is not bound to a Session; lazy load operation of attribute 'dag_model' (#20534)
• Fix the incorrect scheduling time for the first run of dag (#21011): @wanlce
• Fix opening "Task Instance Details" page in web UI unintentionally resets TaskInstance.max_tries (#21018): @yuqian90
  Linked issues:
  • Opening "Task Instance Details" page in web UI unintentionally resets TaskInstance.max_tries (#21017)
• OpenApiSpec - DAGRun - make dag_id in body not required (#21024): @DrFaust92
• Fix running airflow dags test results in error when run twice (#21031): @chenglongyan @howardyoo
  Linked issues:
  • Running airflow dags test <dag_id> <execution_dt> results in error when run twice. (#21023)
• Fix 'airflow dags backfill --reset-dagruns' errors when run twice (#21062): @uplsh580
• Use compat data interval shim in log handlers (#21289): @uranusjr @collinmcnulty
  Linked issues:
  • Elasticsearch remote log will not fetch task logs from manual dagruns before 2.2 upgrade (#20991)
• Adding missing login provider related methods from Flask-Appbuilder (#21294): @aa3pankaj
• Show task status only for running dags or only for the last finished dag (#21352): @alete89 @avkirilishin
  Linked issues:
  • Recent Tasks mixes status for current and old dag runs (#21036)
• Avoid deadlock when rescheduling task (#21362): @easontm @potiuk @stablum
  Linked issues:
  • Airflow crashes with a psycopg2.errors.DeadlockDetected exception (#19957)
  • Tasks fail and do not log due to backend DB (dead?)lock (#16982)
• Alt: Add a session backend to store session data in the database (#21478): @blag @jedcunningham
  Linked issues:
  • Add a session backend to store session data in the database (#21167)
• Pin Markupsafe until we are able to upgrade Flask/Jinja (#21664): @potiuk

Thanks to all who contributed to the release (probably not a complete list!):
@auvipy @uranusjr @kushsharma @SamWheating @lwyszomi @tsingh2k15 @potiuk @dungdm93 @aa3pankaj @davidcaron @alete89 @BasPH @wanlce @hubert-pietron @ashwin153 @smartnewsdingli @yuqian90 @chenglongyan @blag @jakubczaplicki @lostkamp @ljades @easontm @DrFaust92 @jon-fearer @howardyoo @stablum @josh-fell @Datkros @ignaski @kaushikparanjpe @avkirilishin @o-nikolas @jedcunningham @coinzerge @kaxil @microhuang @collinmcnulty @ecerulm @uplsh580 @boredland @lionheart106008 @ephraimbuddy @satishmittal111

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[jon-fearer]

#20303 Looks good in the RC version

[hubert-pietron]

#20591 Looks good too :)

[wanlce]

#21011 Looks good too

[aa3pankaj]

#21294 Looks good

[lwyszomi]

#19963 works fine

[josh-fell]

#19608 looks good for what it was designed to fix but I did find some other scenarios that the inference does not handle properly. Mainly when the @task decorator is "called"/has an argument passed directly in the decorator (e.g. @task(task_id="some_task") -> Dict: ... or @task(on_failure_callback=some_func) -> Dict: ... ) the inference does not work.

Although I have verified that this did not work in Airflow 2.2.3 but does function properly on main.

A workaround is to explicitly use multiple_outputs=True in any case. On the main branch, the inference was refactored and no longer uses the function signature with inspect but uses typing_extensions and seems to work much better.

@jedcunningham This isn't a regression because the inference functionality doesn't work with directly passing args to the @task decorator currently. However, do you think logging an issue is warranted here for visibility? I think some unit tests should be added as well to handle this set of scenarios. WDYT?

[jedcunningham]

Yeah, adding some unit tests to main to make sure it doesn't regress in the future sounds good to me.

I assume cherry-picking the refactoring won't be trivial, so it might not be worth an issue for visibility though.

[jedcunningham]

Thanks everyone, 2.2.4 has been released 🎉

[uplsh580]

#21062 LGTM 👍 (I'm sorry for being late.)

[jedcunningham]

No worries, thanks!

[zachliu]

there is a 🐛 in #18042, sending alert emails using aws ses still doesn't work 😢
gonna open an issue now

issue opened: #21878