helm_release can not properly deploy airflow while helm cli does

[andormarkus]

Terraform, Provider, Kubernetes and Helm Versions

Terraform version: 0.15.1
Provider version: 2.1.2
Kubernetes version: 1.19
Helm version: 3.5.0

Affected Resource(s)

• helm_release

Terraform Configuration Files

resource "helm_release" "airflow" {
  repository = "https://airflow-helm.github.io/charts"
  chart      = "airflow"
  name       = "airflow"
  version    = "8.0.9"
  namespace  = "ns-airflow"
  values     = [file("${path.root}/helm/airflow.yml")]
}

helm/airflow.yml

aiflow:
  executor: KubernetesExecutor
  fernetKey: "CTFNGXwQMB2olDn_xz4OUWMgdTVENNQq0dnHTEyCj9A="
flower:
  enabled: false
redis:
  enabled: false

Bash command

helm repo add airflow-stable https://airflow-helm.github.io/charts
helm repo update
helm install airflow airflow-stable/airflow --namespace ns-airflow --version 8.0.9 --values ./helm/airflow.yml

Debug Output

Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/db_command.py", line 54, in check_migrations
    db.check_migrations(timeout=args.migration_wait_timeout)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/db.py", line 598, in check_migrations
    raise TimeoutError(f"There are still unapplied migrations after {ticker} seconds.")
TimeoutError: There are still unapplied migrations after 60 seconds.

Steps to Reproduce

1. terraform apply
2. kubectl -n airflow get all
3. kubectl -n airflow logs airflow-web wait-for-db-migrations

Expected Behavior

helm_release does provide the same result as helm cli

Actual Behavior

helm_release could not deploy airflow, it will stick on wait-for-db-migrations, while helm cli could deploy it without problem.

References

Other members from the community has the same problem
https://forum.astronomer.io/t/run-airflow-migration-and-wait-for-airflow-migrations/1189

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[andormarkus]

Relates to #683

[jensgrnb]

Hey @andormarkus, I'm facing same issue with my cluster, I'm however not using terraform for helm charts, do you know of a native way of skipping the looping init container?

[andormarkus]

Hi @jensgrnb

Why do you want to skip the init containers?
If you are running Airflow on Kubernetes the two init containers check-db and wait-for-db-migrations is mandatory.

• check-db is checking the connection to the backend database. This can be on the kubernetes cluster or a external one like AWS RDS

• wait-for-db-migrations checks is the schema of the database matches the expected schema. If wait-for-db-migrations fails that means the airflow db init command never was executed or the backend database does not match the required schema.

[jensgrnb]

Well yes, but if the db upgrade job succeeds, and the wait-for-db-migrations still isn't satisfied, then I don't see other option than to skip it and file an airflow bug report?

[andormarkus]

Hi @jensgrnb

For me the db-upgrade never run until init containers succeeded . Do you have the same issue if you run agains a new/clean database?

[jensgrnb]

Yeah that's unfortunally the real world with existing dbs that people like to carry over...(

Shame the logs are not more informative these == clauses without informative logs are just a nightmare for users.

[andormarkus]

I have raised logs not verbose on airlfow side apache/airflow#15650

I would suggests the following solution to keeping the data.

1. Create a new database for airflow and don't let people to put other stuff into it. AWS RDS is like 30$ per month.
2. Do a database migration from the old to the new one while you are transforming the data into the new structure
3. Delete old DB

[jensgrnb]

Yup sounds like a plan forward! Let's see if I can talk sysops into that...

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!