[incubator/airflow] Airflow Helm Chart

[gsemet]

Hello,

Here is the port of the Helm chart I am maintaining on https://github.com/gsemet/kube-airflow. It allows starting Airflow on a Kubernetes cluster using the Celery Backend.

Fix: #2591
Reference: gsemet/kube-airflow#15

This Pull Request is still a Work in Progress, even if it runs in our production server, there have been some recent changes that needs to be stabilized.

[k8s-ci-robot]

Hi @gsemet. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[unguiculus]

/assign

[jpds]

By the way, I'm doing work at https://github.com/bloomberg/airflow/pull/46 which may impact options in this chart (for instance, flower isn't required with the Kubernetes executor).

[gsemet]

Hello. Yes I follow actively your fork and activity. I guess there will be 2 charts at the end, one on the celery executor and one on the Kube execute since they cannot work at the same time (that's a pitty). By the way, I already talk with daniel (I guess) about the KubeOperator, I would be so pleased to be able to use it from the Celery Executor (for example to offload some work to Kubernetes Jobs easily for instance).
But if merge of both charts are easy to do, I'll be happy to let user choose between the celery executor or the kubeexecutor.

[gsemet]

/assign @unguiculus
Patch is now ready to be reviewed

[gsemet]

@unguiculus @jpds can you review or test this chart?

[gsemet]

/joke

[k8s-ci-robot]

@gsemet: What do you call a troublesome Canadian high schooler? A poutine.

In response to this:

/joke

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[unguiculus]

Please make sure you follow our best practices. Please read the guidelines:
https://github.com/kubernetes/charts/blob/master/REVIEW_GUIDELINES.md

[barney-s]

Thanks for the helm chart for airflow. We could get on a quick call to take this forward. Iam interested in using a mysql operator as an option( vs postgres) and redis operator. that could be a subsequent revision.

[maver1ck]

@gsemet
Did you test it with Nginx ingress ?

[gsemet]

What problem do you have ? I use traefik ingress and it only requires a bunch of annotations

[k8s-ci-robot]

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
• If you have done the above and are still having issues with the CLA being reported as unsigned, please email the CNCF helpdesk: helpdesk@rt.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[gsemet]

Hi. I have rebased and followed your recommendation, @unguiculus ! Thanks for the review !

[gsemet]

@kppullin if you are motivated to get this feature working with this airflow chart that would be great to work! I was thinking it was easier to maintain 2 charts but if differences are not that huge, yes, supporting the two executors would be great (celery is pretty damned stable, so it seems like a must have for the moment)!

[kppullin]

@gsemet - The chart and associated airflow config require just a few config changes to get it working with the current state of this chart (basically disabling redis, postgres, and the workers and setting the executor to Kubernetes and setting some airflow properties).

So the differences are quite minor IMHO and I'd prefer having a single chart for the community to maintain and keep up to date.

Also thanks for all you've done on this!!

[kppullin]

In case it helps others, I had to recreate my airflow deployment due to the changes made to the label names. Maybe there's another way to fix this up but a purge and reinstall way simplest on my end.

I should probably also mention that this may delete other resources that you want to keep around (PVCs, potentially the postgres DB, etc) so be careful!

[gsemet]

@unguiculus all done :) thanks for the review

[unguiculus]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gsemet, unguiculus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [unguiculus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gsemet]

Thanks a lot ! I really hope this will be useful for many people ! And also thanks a lot to all contributors that increased a lot the stability and features of this chart :)

[luthes]

I'm having an issue with this where I'm getting not getting the output from the NOTES.txt, looks like the templated values aren't populating.

Congratulations. You have just deployed Apache Airflow
   export POD_NAME=$(kubectl get pods --namespace airflow1 -l "component=" -o jsonpath="{.items[0].metadata.name}")
   echo http://127.0.0.1:
   kubectl port-forward --namespace airflow1 $POD_NAME :

2. Open Airflow in your web browser

When running the kubectl command by itself I get this:

Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template:
        template was:
                {.items[0].metadata.name}
        object given to jsonpath engine was:
                map[string]interface {}{"kind":"List", "apiVersion":"v1", "metadata":map[string]interface {}{"selfLink":"", "resourceVersion":""}, "items":[]interface {}{}}

error: error executing jsonpath "{.items[0].metadata.name}": array index out of bounds: index 0, length 0

I can get the first one to populate this command: kubectl get pods --namespace airflow1 -L "component" -o jsonpath="{.items[0].metadata.name}"

Unless I'm mistaken, .Values.airflow.name doesn't exist in the values.yaml, if that's where it should come from. Is there some updating that needs done to the NOTES.txt file?

I'm running this on AWS EKS, so k8s 1.10.3.

Thanks.

[gsemet]

Hi. There have been a recent change around here just before fix, if there is a mismatch do not hesitate to fix it. We might probably want to use the template airflow.fullname.