Airflow 2.3.0 Support

[c-thiel]

Checks

• I have checked for existing issues.
• This report is about the User-Community Airflow Helm Chart.

Motivation

The Chart is currently not compatible with Airflow 2.3.0 due to changes in the Database Configuration:
https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#database-configuration-moved-to-new-section-22284

Implementation

No response

Are you willing & able to help?

• I am able to submit a PR!
• I can help test the feature!

[c-thiel]

When updating the image to 2.3.0 I get the following Error during CheckDB:

Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/__init__.py", line 35, in <module>
    from airflow import settings
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/settings.py", line 35, in <module>
    from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf  # NOQA F401
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/configuration.py", line 1345, in <module>
    conf.validate()
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/configuration.py", line 293, in validate
    self._validate_config_dependencies()
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/configuration.py", line 387, in _validate_config_dependencies
    raise AirflowConfigException(f"error: cannot use sqlite with the {self.get('core', 'executor')}")
airflow.exceptions.AirflowConfigException: error: cannot use sqlite with the KubernetesExecutor

Actually, acording to https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#database-configuration-moved-to-new-section-22284, it should just throw a deprecation warning and not fail. Not sure what I am missing.

[c-thiel]

I traced the broken backward compatibility over to Airflow and created an Issue: apache/airflow#23408

Nevertheless, we should use AIRFLOW__DATABASE__SQL_ALCHEMY_CONN_CMD instead of AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD.

[c-thiel]

Until this is fixed, you can use the following workaround:

airflow:
  config:
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN_CMD: "bash -c 'eval \"$DATABASE_SQLALCHEMY_CMD\"'"

[martinrw]

Thanks for this @c-thiel - your suggestion helped.
My application is now up and running version 2.3.0 using the helm chart and seems to be working fine but I have one pod called "sync users" that keeps restarting.

It appears to be trying to run this script:
/mnt/scripts/sync_users.py

and eventually fails with this error:
sqlalchemy.exc.InvalidRequestError: Multiple classes found for path "Permission" in the registry of this declarative base. Please use a fully module-qualified path.

So maybe there is something more that also needs to be changed?
I am using Oauth for user logins so maybe that's why I'm not affected by this

[Linux-oiD]

Could you also add support for the separate DagProcessorManager deployment that was introduced in 2.3.0?
https://airflow.apache.org/blog/airflow-2.3.0/#dagprocessormanager-as-standalone-process-aip-43
Or should I create a separate issue for that?

[denysivanov]

Really need this one!

[kvin007]

Looking forward to this feature

[OlexandrRudenko]

It will be great

[mr-wolf-rsh]

stuck with this too, hoping for this feature to be released soon

[Zakolesnik]

Waiting for this feature

[mmeza09]

Waiting for this feature

[PamelaSofiaCastillo]

i hope this will be released soon

[JoHermoza]

Hoping for this feature

[CarrascoEr]

Waiting for this feature

[diego-santamaria]

This will be really helpful.

[rociofc0312]

Looking forward to this feature

[chrisalo97]

Will be great to have this feature

[Alexsander00]

Waiting for this feature too

[GianFNoguni]

Waiting for this feature!

[emilymitacc]

Waiting to start to use this feature

[gmontero06]

Waiting for the release of this feature 🙏

[LuisEspinoza212]

Really need this one

[thesuperzapper]

Hey all,

I plan to release chart version 8.6.1 with a fix for the main issues preventing Airflow 2.3 from working, allowing people to choose an Airflow 2.3.0 image if they want.

However, I think we should wait until chart version 8.7.0 before making Airflow 2.3.X the default version.
This will give us time to implement support for Airflow 2.3 specific features, and wait for airflow itself to find and patch any critical problems with 2.3.0 before encouraging half the world to update.

• Support the new Standalone DagProcessorManager
• Fully support ARM/M1 chips out-of-box (most stuff already works, but we need to update the embedded Postgres/Redis with cross-compiled ARM images)

PS: If anyone knows of any other Airflow 2.3 features, we currently don't support, please comment them here.

[Valenzione]

@thesuperzapper That's great, thank you for your work! Do you have any time estimates in mind for the 2.3.0 support release date?

[thesuperzapper]

@thesuperzapper That's great, thank you for your work! Do you have any time estimates in mind for the 2.3.0 support release date?

@Valenzione I expect to release 8.6.1 in the next few days, which will allow using airflow 2.3.0 images but not make it default.

Track progress on the 8.6.1 milestone. (NOTE: I haven't added the 2.3.0 issue fixes to that milestone yet)

---

Regarding the 8.7.0 release, I want at least one large feature (in addition to making 2.3.0 the default) so that people have a reason to update.

Right now, my hope is to finish the new task-aware celery autoscaler which allows safe and configurable up/downscaling of CeleryExecutor workers (including selectively removing the inactive worker Pods when downscaling). The prototype is coming along, but I haven't had much time to put the final polish on yet.

---

PS: the reason 8.6.1 is not quite ready yet was that in the last week I focused on improving the docs, the best change is that we now provide sample starting points of custom-values.yaml for each executor type CeleryExecutor, KubernetesExecutor, and CeleryKubernetesExecutor.

But I also made large updates to Quickstart Guide, How to manage airflow connections?, How to load DAG definitions?, How to persist airflow logs?.

[thesuperzapper]

Hey All, I am very sorry about the delay, I have been quite bogged down this week and last (Airflow Summit and the like).

However, we now have resolutions for the main problems caused by 2.3.0, so should be ready to cut 8.6.1:

• Removal of AIRFLOW__DATABASE__SQL_ALCHEMY_CONN_CMD
  • (fixed by fix: some breaking changes from airflow 2.3.0 #592)
  • NOTE: airflow 2.3.1, actually fixes this issue by reverting the incorrect remoal
• Airflow now uses airflow.www.fab_security.sqla.model rather than flask_appbuilder.security.sqla.models
  • (fixed by fix: some breaking changes from airflow 2.3.0 #592)
• timeout=0 breaks the airflow.utils.db.check_migrations()
  • (fixed by fix: wait-for-db-migrations in airflow 2.3.0 #576)

If anyone knows of other issues from 2.3+ not listed above please comment ASAP, so we can get them fixed in 8.6.1!

[jurovee]

@thesuperzapper I think we should give it a go, let's test that in our environments and provide a feedback if necessary.

Disclaimer: We struggle a little bit with one of the bugs in Airflow 2.2.* which should be fixed in 2.3.1 hopefully, so no pressure! 😄😄

[karakanb]

@jurovee not sure if it helps, but I have managed to upgrade from v2.2.5 to v2.3.2 with the chart version v8.6.0 without any changes in my values file. The breaking changes that were introduced in v2.3.0 seem to be fixed in the subsequent versions, so the upgrade itself works fine from this chart's perspective.

[jurovee]

@karakanb good to know! 👏 Thanks for the info. I think that 8.6.1 is almost ready and it should contain a fix for one of my issues with logs and external volumeMounts, so I think I'll just wait to have everything together + 2.3.2 in a single package 👍

[thesuperzapper]

@karakanb @jurovee there is still one issue with the sync-users script in airflow 2.3.2 which will require the 8.6.1 release (as they actually moved some of the imports).

NOTE: this is fixed by #592

[thesuperzapper]

Hey all, the long-awaited 8.6.1 is out now with support for Airflow 2.3.0!

I would love it if you tested it out, and shared feedback!

[martinrw]

Just gave it a test...
Set my dockerfile to use apache/airflow:2.3.2-python3.9 as the base and helm chart version 8.6.1
it works perfectly. Thanks !!!