[stable/airflow] web-pod can't connect to postgres (password authentication failed)

[Kyrremann]

Describe the bug
When using the stable chart, my web-pod can't connect to the postgres-pod. I've also tried to exec into the postgres-pod, but the same result there.

Version of Helm and Kubernetes:
kubernetes: v1.16.7
helm: v3.2.3

Which chart:
stable/airflow

What happened:
airflow-web logs:

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  password authentication failed for user "postgres"

airflow-postgres logs:

2020-07-03 11:43:19.358 GMT [196] FATAL:  password authentication failed for user "postgres"
2020-07-03 11:43:19.358 GMT [196] DETAIL:  Password does not match for user "postgres".
	Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"

What you expected to happen:
Expected it to get up and running

How to reproduce it (as minimally and precisely as possible):
helm install airflow stable/airflow

Anything else we need to know:

[ioluwayo]

I am facing this exact same issue.

[thesuperzapper]

@Kyrremann @ioluwayo

1. is anything working at all, (can you see the webui)?
2. are you using a custom values.yaml?
3. are you sure there was no residual resources in your default namespace from some other airflow install?

Give an install a try in a new namespace:

kubectl create namespace airflow

helm install stable/airflow \
  --version "X.X.X" \
  --name "airflow" \
  --namespace "airflow" \
  --values ./custom-values.yaml

[Kyrremann]

Tried it in a new blank namespace (airflow) and it works. So at least latest version is working. I'll inspect our cluster and test some more.

I'm running jupyterhub and Dask in the other namespace, strange if that will stop it. Also worth mentioning that I'm using Helm 3.

[Kyrremann]

Still no luck, can't even login to postgres

$ k exec -it airflow-postgresql-0 bash
I have no name!@airflow-postgresql-0:/$ echo $POSTGRES_DB
airflow
I have no name!@airflow-postgresql-0:/$ echo $POSTGRES_PASSWORD
airflow
I have no name!@airflow-postgresql-0:/$ echo $POSTGRES_USER
postgres
I have no name!@airflow-postgresql-0:/$ PGPASSWORD=airflow psql -d airflow -U postgres -w
psql: FATAL:  password authentication failed for user "postgres"

Works on the clean namespace. So my guess is that something is not working when the postgres-pod is starting up 🤔

[Kyrremann]

Another discovery, using postgres as password, I managed to get a new error!

I have no name!@airflow-postgresql-0:/$ psql -d airflow -U postgres
Password for user postgres:
psql: FATAL:  database "airflow" does not exist

So it looks like the start up script is not run.

[Kyrremann]

After deleting the namespace, and recreating the helm-charts that where already there, I finally got Airflow working! Have no idea what solved, but there where something wrong with that specific namespace.

[Kyrremann]

Ended up with this problem again, and I found out what the issue was! When you run helm uninstall it doesn't remove the PVC that the postgres-database uses. So the old database and password is still there when you run helm install again! So remember to delete the PVC!

[imorti]

Totally nailed this. Great call. Thank you.

[thesuperzapper]

@Kyrremann or @imorti can you raise as an issue on the new helm chart repo?

https://github.com/airflow-helm/charts/tree/main/charts/airflow