Upgrade to Airflow 2.0

[nguyenkien1402]

Has anyone had successfully upgrade to Airflow 2.0 yet?
If yes, what you need to alter from the chart?

I have been trying to alter the files under the template folder and values.yaml but the best I got so far was to be able to see WebServer and Scheduler pods are running but not the DAG.

I trigger the DAG and quickly got the error telling me Failed to log action with (sqlite3.OperationalError) no such table....

The weird thing is I use an external Postgres database and I can confirm there are >20 tables are created by the scheduler but somehow I still get error related tosqlite3.OperationalError.

[thesuperzapper]

I will be getting the chart ready for 2.0 soon as I describe here: #52 (comment)

[nguyenkien1402]

Hi @thesuperzapper , that is very good news.
I will be the first one to try once the chart ready for 2.0
Thank you 🥇

[stijndehaes]

@thesuperzapper If you can use my help just ping me. I am also thinking shouldn't we support the changed k8s executor config supported in airflow 1.10.14 for example first before we fully use airflow 2.0. That way users can first upgrade to that version and make sure that all the other stuff is supported before pulling the trigger on 2.0

[illyaSlobozhanin]

Up.
For the proper launch of airlfow 2.0 with this chart need to manually update sts/deployments.
Almost all issues are connected with commands renames.
e.g. airflow worker -> airflow celery worker; airflow upgradedb -> airflow db upgrade, etc.

[thesuperzapper]

For those watching, I am working on this, it's just taking a while to do properly between my many other responsibilities.

Hopefully I will get it done this week.

[dmerrick]

We're rooting for you!

[jherrmannNetfonds]

If you need a "beta tester", please let me know.

[thesuperzapper]

Small update for those watching, the 8.0.0 version of the chart is coming along (in my local development), probably a few days away now.

Trust me, the wait will be worth it!

[jdotjdot]

@thesuperzapper is there a publicly available branch of 8.0.0 we could use to get started, even if it's not formally released yet? I would absolutely get started with it now + test it. The only thing stopping me from deploying Airflow 2.0 on Kubernetes right now is this—I would have written my own chart, but seems silly to do so when you say you're just days away!

[dmerrick]

What he is saying is his branch is almost ready for testing, not that 8.0.0 is almost ready for release

[xdoomdog]

Is there a way to prevent execution of this during a helm installation:
airflow variables -i /home/airflow/variables-pools/variables.json

this file doesn't even exists as far as I can tell. I could then try, if it makes sense, to run this command:
airflow variables import /home/airflow/variables-pools/variables.json
in an entrypoint.sh script, and hopefully the command dies gracefully if the file is missing. It probably shouldn't be a fatal error since there are other ways of exporting variables.

Disabling the loading of sample dags is also a problem.

[aramkarapetian]

@thesuperzapper Could you tell where the branch is? I would like to try id out.

[xdoomdog]

@thesuperzapper Could you tell where the branch is? I would like to try id out.

I wish I knew, I'm just using the formal release one. I don't think they've made that public yet, I could be wrong.

[vs4vijay]

Hi, Might be a silly question, but why can't we use the existing helm chart from previous version? If we just update the docker image of airflow and point helm chart to updated airflow image, wouldn't that work? Are there any other changes required for Airflow 2?

[yoavo-datricks]

Hi, Might be a silly question, but why can't we use the existing helm chart from previous version? If we just update the docker image of airflow and point helm chart to updated airflow image, wouldn't that work? Are there any other changes required for Airflow 2?

They changed some CLI parameters causing the start script to fail (among other changes)

[Yinabled]

In the meantime, is there a way to explicitly specify an older version of Airflow (e.g. 1.10.14) in the chart? Right now, Airflow simply isn't working for us because the chart is defaulting to the latest available image (i.e. 2.0.1).

[bensta]

@Yinabled the chart is defaulting to version 1.10.12, not version 2.0.1. You can check the default values file to double check that.
If you see that your helm tries to install version 2.0.1, double check your values file.

[vprus]

I see that Airflow git repository now also has a Helm chart, at https://github.com/apache/airflow/tree/master/chart - and looks different from this one, and possibly a copy of Astronomer's helm chart. I am a bit at a loss which one is to use.

[bensta]

@vprus the helm chart you see in the airflow repo is not officially released yet. So I would expect it to change a bit until it is.
You can follow this issue for updates apache/airflow#10752

[thesuperzapper]

Im really sorry for taking so long with this, but I am still working on it, hopefully in the next few days it will be finished.

NOTE: I already have it working with 2.0, but I want to actually support all the new features (and remove 1.10 specific features of the chart)

[dmerrick]

Ty for your work, @thesuperzapper !

[denysivanov]

@thesuperzapper thanks a lot for your work!

[gmiretti]

@thesuperzapper did you push your working branch so we can start to test it before the super-duper complete release?
The current helm chart working with 2.0 that you mentioned would already help us a lot to accelerate our adoption of airflow 2.
Thx for your work!

[jdotjdot]

@thesuperzapper just want to reiterate that we all are happy to beta-test the chart even if it's in draft form, submit PRs to get it the rest of the way, etc. We just need you to be willing to release the code you currently have, even if it's unfinished! If you push the code to a feature branch (even if it literally doesn't work yet) I'd gladly do work on it to help it get over the finish line!

[kapopken]

Any updates on this? I would like to beta test it or help in anyway as well.

[thesuperzapper]

I will post a PR tonight, but it will need a few days to get merged.

[rolanddb]

Hi @thesuperzapper, just to repeat the question asked by many: is it possible to push your branch (with the understanding that it is Work In Progress) to a public repo? It seems that people (including me) are eager to dive in, beta test and help finish your work.

[joshi95]

Very correct !!

Hi @thesuperzapper, just to repeat the question asked by many: is it possible to push your branch (with the understanding that it is Work In Progress) to a public repo? It seems that people (including me) are eager to dive in, beta test and help finish your work.

Yes please

[thesuperzapper]

I know everyone is waiting eagerly, and trust me I am working on it, I want it done too!

But I don't want to encourage people to use a beta version, especially since there are still some significant changes I have yet to make (and that's just a bad experience for users).

Version 8.0.0 will be a very significant upgrade and therefore would not be very useful without the upgrade guide (which I am still writing).

Also, as a teaser of new features:

• the docs have been re-written (and formatted much better)
• we have moved to the official kubernetes git-sync container
• extra pip packages are now installed in an init container (removing the possibility of a timeout)
• KubernetesExecutor /CeleryKubernetesExecutor will work out of the box (with a dynamically generated pod_template.yaml)
• many of the confusing values have been removed or refactored
• many more things...

In terms of when I expect to push out an initial version of 8.0.0, I can't 100% confirm it, but since I have more time over the next week, I expect it will be ready in a couple of days.

[fengsi]

A key feature I've been waiting for is the ability to have the git sync sidecar only running with the scheduler. We have an EFS config for DAGs such that all pods share the same DAGs from a single EFS share, and only one pod is needed to sync the DAG repo.

In the past we had our own chart based on this one with modified templates for above requirement (as the sidecar config in this chart is for all pods). Now we are evaluating Bitnami Airflow chart that supports such use case (dedicated sidecar for scheduler/worker/web etc.).

I was wondering if the upcoming 8.0.0 supports such feature or not, as we are going to make a decision very soon whether to switch or not, in order not having to maintain our own chart.

Thanks.

[thesuperzapper]

@fengsi the chart (even in its current form) allows you to store DAGs on a PVC dags.persistence.enabled however the git-sync sidecar doesn't work with this enabled as they would all fight.

I think an easy solution is for me to add a new "git-sync" Deployment (with a single replica), which is only deployed if the user enables dags.persistence.enabled and dags.gitSync.enabled.

[fengsi]

@fengsi the chart (even in its current form) allows you to store DAGs on a PVC dags.persistence.enabled however the git-sync sidecar doesn't work with this enabled as they would all fight.

I think an easy solution is for me to add a new "git-sync" Deployment (with a single replica), which is only deployed if the user enables dags.persistence.enabled and dags.gitSync.enabled.

Right – ideally if all requirements could be fulfilled by just tweaking the values.yaml file then that'll be awesome. Another example is the envFrom.secretRef config that is totally missing in this chart, leaving us no choice but making our own chart to add that feature.

[olivierpilotte]

@thesuperzapper

All we want is to have some base to work with. We've actually considered creating our own chart, but the "it's gonna be done this week" as been going for over a month... so we are waiting.

We are totally OK with the chart not being 100% finished. Please push what you have done so far so we can at least move forward?

Thanks!

[thesuperzapper]

I know this is getting ridiculous at this point, but I have been slammed by Kubeflow work (@davidspek, lol).

I am VERY CLOSE to getting this merged, just some minor stuff to finish.

So I expect 8.0.0 to be released in the next 48 hours.

<3

[davidspek]

I'd like to apologize to everybody here that's been waiting on the Airflow update for so long. The past week I've been keeping @thesuperzapper up until 6 a.m. fixing the new Dockerfiles for Kubeflow. Any moment I wasn't keeping him busy he was working on this Helm chart and from what he's told me it sounds like an amazing update. From what I've been able to see the chart is nearly finished :)

[Nijanthan-Hadoop]

I'm using MacBook Air M1, getting below issues while using Airflow 2.0 latest docker compose. Appreciate any help to fix this issue. Docker compose

irflow-init_1 | qemu: uncaught target signal 4 (Illegal instruction) - core dumped
airflow-init_1 | /entrypoint: line 192: 27 Illegal instruction airflow db upgrade
airflow-init_1 |
airflow-init_1 | airflow command error: argument GROUP_OR_COMMAND: airflow upgradedb command, has been removed, please use airflow db upgrade, see help above.
airflow-init_1 | usage: airflow [-h] GROUP_OR_COMMAND ...

airflow-init_1 | /entrypoint: line 129: 54 Illegal instruction airflow users create --username "${_AIRFLOW_WWW_USER_USERNAME="admin"}" --firstname "${_AIRFLOW_WWW_USER_FIRSTNAME="Airflow"}" --lastname "${_AIRFLOW_WWW_USER_LASTNME="Admin"}" --email "${_AIRFLOW_WWW_USER_EMAIL="airflowadmin@example.com"}" --role "${_AIRFLOW_WWW_USER_ROLE="Admin"}" --password "${local_password}"
airflow-init_1 |
airflow-init_1 | airflow command error: argument GROUP_OR_COMMAND: airflow create_user command, has been removed, please use airflow users create, see help above.
airflow-init_1 | usage: airflow [-h] GROUP_OR_COMMAND ...
airflow-init_1 |

[kapopken]

@Nijanthan-Hadoop the api has changed for airflow. The start up command should be airflow db migrate. I don't think this is the right place for this comment however given your working with the docker compose and this is specifically for the helm chart.
https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html

[thesuperzapper]

Hey all, I am pleased to say I have released version 8.0.0 of the chart!!!

I am looking forwards to hearing feedback/ideas from you guys!

PS: I want to encourage people who have been following closely to read the Request for Contributions section and contribute if you are willing/able.