Quickstart Helm Chart fails post-install #16176
Comments
|
Thanks for opening your first issue here! Be sure to follow the issue template! |
|
Can you provide logs of |
|
I will try to see if I can reproduce this issue. |
|
I could not reproduce this issue. Consider closing it @kaxil ...
helm install $RELEASE_NAME apache-airflow/airflow --namespace $NAMESPACE --debug...
NOTES:
Thank you for installing Apache Airflow 2.0.2!
Your release is named airflow-release.
You can now access your dashboard(s) by executing the following command(s) and visiting the corresponding port at localhost in your browser:
Airflow Webserver: kubectl port-forward svc/airflow-release-webserver 8080:8080 --namespace airflow-namespace
Flower dashboard: kubectl port-forward svc/airflow-release-flower 5555:5555 --namespace airflow-namespace
Default Webserver (Airflow UI) Login credentials:
username: admin
password: admin
Default Postgres connection credentials:
username: postgres
password: postgres
port: 5432
You can get Fernet Key value by running the following:
echo Fernet Key: $(kubectl get secret --namespace airflow-namespace airflow-release-fernet-key -o jsonpath="{.data.fernet-key}" | base64 --decode) |
|
|
@stephsamson can you delete the namespace and recreate it. Then run |
|
@ephraimbuddy thanks that worked! |
|
Hi everyone, I have run into the same issue on a fresh microk8s cluster. I have tried deleting the namespace and updating the repo several times. Anyone running into the same issues? |
|
I have not been able to get airflow installed. I've tried several times, deleting the cluster on each attempt. I'm following these instructions https://marclamberti.com/blog/airflow-on-kubernetes-get-started-in-10-mins/. I'm working on a 2021 M1 Mac Air under Big Sur. |
|
On another attempt, this gets added to the output: |
|
@kaxil @Dr-Denzy please re-open this issue as multiple people are reporting it. I am able to recreate it intermittently myself. You can follow the notes here: https://github.com/zambien/tf-eks-airflow/blob/tf_eks_extended/notes.md deploy airflow on k8s using helm without packaged dbTo keep everything simple we use the default namespace Get your charts Run postgres helm install db \
--set postgresqlPassword=secretpassword,postgresqlDatabase=airflow \
bitnami/postgresqlRun airflow without the included db: helm install airflow apache-airflow/airflow --debug \
-f terraform/kind/airflow-values.yaml \
--set 'env[0].name=AIRFLOW__CORE__LOAD_EXAMPLES,env[0].value=True'Sometimes this works, other times it does not. It seems that the catalyst may be the separate database. Here is the issue I see: helm install --debug airflow apache-airflow/airflow \ ✔ 5624 06:57:56
-f terraform/kind/airflow-values.yaml \
--set 'env[0].name=AIRFLOW__CORE__LOAD_EXAMPLES,env[0].value=True'
install.go:178: [debug] Original chart version: ""
install.go:199: [debug] CHART PATH: /home/adam/.cache/helm/repository/airflow-1.2.0.tgz
client.go:299: [debug] Starting delete for "airflow-broker-url" Secret
client.go:128: [debug] creating 1 resource(s)
client.go:299: [debug] Starting delete for "airflow-fernet-key" Secret
client.go:128: [debug] creating 1 resource(s)
client.go:299: [debug] Starting delete for "airflow-redis-password" Secret
client.go:128: [debug] creating 1 resource(s)
client.go:128: [debug] creating 27 resource(s)
client.go:299: [debug] Starting delete for "airflow-run-airflow-migrations" Job
client.go:128: [debug] creating 1 resource(s)
client.go:528: [debug] Watching for changes to Job airflow-run-airflow-migrations with timeout of 5m0s
client.go:556: [debug] Add/Modify event for airflow-run-airflow-migrations: ADDED
client.go:595: [debug] airflow-run-airflow-migrations: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:556: [debug] Add/Modify event for airflow-run-airflow-migrations: MODIFIED
client.go:595: [debug] airflow-run-airflow-migrations: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
Error: INSTALLATION FAILED: failed post-install: timed out waiting for the condition
helm.go:88: [debug] failed post-install: timed out waiting for the condition
INSTALLATION FAILED
main.newInstallCmd.func2
helm.sh/helm/v3/cmd/helm/install.go:127
github.com/spf13/cobra.(*Command).execute
github.com/spf13/cobra@v1.2.1/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
github.com/spf13/cobra@v1.2.1/command.go:974
github.com/spf13/cobra.(*Command).Execute
github.com/spf13/cobra@v1.2.1/command.go:902
main.main
helm.sh/helm/v3/cmd/helm/helm.go:87
runtime.main
runtime/proc.go:225
runtime.goexit
runtime/asm_amd64.s:1371
|
|
@zambien #18776 should allow a disabling a Helm Hooks which might fix issue for you. Can you try it out on your local machine or dev cluster by running the following commands:
|
|
@kaxil, i am receiving similar err (see below) - wonder if i should also try the above commands relating to 1.3.0rc1, or whether it requires different kind of fix - thanks. |
|
Up, facing the same problem. |
|
up, I have the same issue |
|
I have the same issue. |
|
I got the same issue, but installing older version worked for me |
|
Confirming this. |
|
Can someone please open a new issue with all the details please? This is a closed issue. Likely different reason. Commenting on a closed issue from May will not resurrect it. Even if symptoms might be similiar, it is likely a different issue. |
|
We really need more details - values, configurtions, detaile logs from the "wait-for-migrations" jobs etc |
|
Hi |
|
Perhaps someone can install with "--timeout 10m0s" option. It worked for me when I use the official apache helm chart. |
|
I'm facing the same issue, even with timeout option it's not working. |
Please open a detailed issue about this with more details (logs and describing what you experience). It might likely be a differetn issues The comment "I have the same issue" on a closed issue does not help in any meaningful way in diagnosing the issue. |
I run this command: [root@localhost ~]# helm upgrade --install airflow apache-airflow/airflow --namespace airflow --create-namespace --debug --timeout 10m0s client.go:128: [debug] creating 1 resource(s) |
|
can you please use |
|
K9s will allow you to monitor more logs in your deployment and likely find the right problem - just explore your installation with it. |
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 142, in _do_get │ |
|
So you have a problem with connecting to postgres then |
how can I resolve connection issue? |
|
No idea. You have to debug it. |
|
install.go:173: [debug] Original chart version: "" client.go:290: [debug] Starting delete for "airflow-broker-url" Secret |
|
I am facing the exact same error, as per official documentation the postgres db is itself being created in a container, so connection issue should not be there, it just gets stuck at airflow-run-airflow-migrations. Any resolution will be highly appreciated. |
More details as mentioned, are the only way any help can be given to you (or rather yourself looking at the logs of migration job will likely find the reason). Without those details we are not able to help you. Stating " I have the same problem" without providing any additional details helps no-one to find the root cause. If you state "I have the same problem" you need to provide more detailed logs to bring any value to the discussion here @Abhinav1598 |
Tried using K9s still dont see any log output, most of the time it just says |
|
kubectl ? How else are you debugging other charts? Just do the same. |
|
It’s solved, I was inside my companies vpn, so I was unable to pull the images from docker. I pulled and pushed the images to my remote repo, and it started working. :) |
|
I had the same issue there were multiple reasons, so thought of sharing
|
|
I was able to resolve this for my single-node testing cluster. Checking the |
|
@noah-gil Do you have the configuration to get the correct cluster to run Airflow? I'm using the last helm chart of Airflow, and I used this configuration for the cluster... but I get the same error :( |
|
I'm getting the same error. Currently debugging but will open a separate issue if I get to a standstill [ sorry to add on to the comments of this closed issue ] I added the update step after installing the apache-airflow helm chart. Not sure how to proceed. Any suggestions welcome, thank you! |
|
I tried deleting/uninstalling the airflow deployment and also wiped clean the airflow repo from local helm. Still facing same issue. Will post my progress in case if i'm able to fix it. Command: `sudo helm install airflow apache-airflow/airflow --namespace airflow --debug Helm Version |
|
Without any details while the migration job failed, I am afraid commenting on closed issue will not help. You need to see the logs of the job that failed and post it (ideally as a new issue as this might be completelty different issue). |
Apologies @potiuk, i'll create a new issue with more detailed instructions to reproduce the issue. Cheers!! |
|
To anyone else who has stumbled upon this thread, for me the issue was Docker Desktop not having enough memory. I increased this to 9gb with a Swap of 2gb and repeated the helm install process and all was fine |
|
My issue was fixed when I cleared more of system space |
These are the commands I'm using the check the logs of the failed job: Looks like my issue is similar to the one described above by @Abhinav1598... but unsure which image is the one that is causing the failure. |
|
You myst check your logs on K8S - this is absolutely normal for you as someone who manages k8s installation to fix any problems and be able to diagnose this. You have to learn it I am afraid @beascar. Various tools (kubectl, helm, k9s) are useful for that and your job is basically to master them. You chose k8s as your deployment, so you need to understand how to diagnose various problems there as a consequence. I cannot solve and diagnose your k8s installation for you, but If you are not familiar with using kubectl (you should eventually), one useful tool to use is Good luck with the diagnoses. |
|
I have spectated, that in my case postgres pod were in pending state due to claim - hope it helps some of you above
|
|
for windows after increasing WSL memory, it worked!! https://learn.microsoft.com/en-us/windows/wsl/wsl-config#configure-global-options-with-wslconfig |
|
Hi Same issue. Helm output Migration job log shows all done but never gets "success" state? |
|
Increase timeout (look at help of helm), or increase memory (check your resources settings. Or if you use Argo or Similar look at our docs for chart https://airflow.apache.org/docs/helm-chart/stable/index.html#installing-the-chart-with-argo-cd-flux-rancher-or-terraform |
|
Thanks @potiuk Can you by any chance elaborate on why does one need to for Argo, Rancher etc? As in: why without this (or with this? I'm confused now) the migrations will not be run? |
|
I think it's the question to Argo and Rancher. The current way works with standard Helm - they seem to use the hooks in a non-standard way, but maybe you can help developing better ways. We are open-source projects so we aim to support standards, not commercial solutions that somewhat modified it. But if you use such a solution and want to help with making it better supported - cool. Some of the initial reasoning was described here #17447 but if someone (you?) find a better way of supporting Argo/Rancher that's cool. We are happy to accept contributions to make it easier/better. I personally don't use Argo, so I am not able to comment more other than - this is the way someone at some point found as working solution. But if someone else finds a better way and can confirm it works (and keeps it working for regular Helm Chart - this is even cooler). Airflow is created by > 2600 contributors - and often people who miss something or find it confusing, spend time to fix it better and contribute back. So - if you think you can help with analysing and providing a better fix - cool. |
and others
Apache Airflow version: 2.0.2
Kubernetes version (if you are using kubernetes) (use
kubectl version): 1.19Environment:
kindlocallyuname -a): Darwin MacBook-Pro 19.6.0 Darwin Kernel Version 19.6.0: Mon Apr 12 20:57:45 PDT 2021; root:xnu-6153.141.28.1~1/RELEASE_X86_64 x86_64What happened:
Helm chart does not successfully deploy to a kind cluster despite following the Quick Start. Repeatedly tried multiple times and the flower, postgres, redis and statsd services run fine but it fails at the run-airflow-migrations service with a
CrashLoopBackoff:What you expected to happen:
Successful Helm deployment.
How to reproduce it:
kind create cluster --image kindest/node:v1.18.15helm repo add apache-airflow https://airflow.apache.orgkubectl create namespace airflowhelm install airflow apache-airflow/airflow --namespace airflow --debugThe text was updated successfully, but these errors were encountered: