CrashLoopBackOff for most pods due to DB migration failing when upgrading to chart version 8.6.1 and Airflow version 2.3.0.

[conchaox]

Hello,

We've been using this Helm chart for a little bit more that a year now, to deploy it we are using the Terraform helm_release resource. Right now, I'm trying to update one release from chart version 8.5.2 to 8.6.1 and Airflow version from 2.1.x to 2.3.0, I'm getting the following error in the db-migrations container:

/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:470: DeprecationWarning: The auth_backend option in [api] has been renamed to auth_backends - the old setting has been used, but please update your config.
  option = self._get_environment_variables(deprecated_key, deprecated_section, key, section)
[2022-11-15 22:13:17,877] {db.py:702} INFO - Waiting for migrations... 0 second(s)
[2022-11-15 22:13:17,878] {db_migrations.py:53} WARNING - there are unapplied db migrations, triggering apply...
[2022-11-15 22:13:17,878] {db_migrations.py:45} INFO - -------- START - APPLY DB MIGRATIONS --------
[2022-11-15 22:13:18,217] {db.py:1410} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade f9da662e7089 -> e655c0453f75, Add ``map_index`` column to TaskInstance to identify task-mapping,
and a ``task_map`` table to track mapping values from XCom.
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_context
    self.dialect.do_execute(
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 716, in do_execute
    cursor.execute(statement, parameters)
  File "/home/airflow/.local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/home/airflow/.local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/home/airflow/.local/lib/python3.8/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1091, "Can't DROP 'task_reschedule_ti_fkey'; check that column/key exists")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/scripts/db_migrations.py", line 78, in <module>
    main(sync_forever=True)
  File "/mnt/scripts/db_migrations.py", line 54, in main
    apply_db_migrations()
  File "/mnt/scripts/db_migrations.py", line 46, in apply_db_migrations
    upgradedb()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 71, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/db.py", line 1411, in upgradedb
    command.upgrade(config, revision=to_revision or 'heads')
  File "/home/airflow/.local/lib/python3.8/site-packages/alembic/command.py", line 320, in upgrade
    script.run_env()
  File "/home/airflow/.local/lib/python3.8/site-packages/alembic/script/base.py", line 563, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/home/airflow/.local/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
    module = load_module_py(module_id, path)
  File "/home/airflow/.local/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/migrations/env.py", line 107, in <module>
    run_migrations_online()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/migrations/env.py", line 101, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/home/airflow/.local/lib/python3.8/site-packages/alembic/runtime/environment.py", line 851, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/home/airflow/.local/lib/python3.8/site-packages/alembic/runtime/migration.py", line 620, in run_migrations
    step.migration_fn(**kw)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/migrations/versions/0100_e655c0453f75_add_taskmap_and_map_id_on_taskinstance.py", line 49, in upgrade
    batch_op.drop_index("idx_task_reschedule_dag_task_run")
  File "/usr/local/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/home/airflow/.local/lib/python3.8/site-packages/alembic/operations/base.py", line 374, in batch_alter_table
    impl.flush()
  File "/home/airflow/.local/lib/python3.8/site-packages/alembic/operations/batch.py", line 108, in flush
    fn(*arg, **kw)
  File "/home/airflow/.local/lib/python3.8/site-packages/alembic/ddl/mysql.py", line 153, in drop_constraint
    super(MySQLImpl, self).drop_constraint(const)
  File "/home/airflow/.local/lib/python3.8/site-packages/alembic/ddl/impl.py", line 336, in drop_constraint
    self._exec(schema.DropConstraint(const))
  File "/home/airflow/.local/lib/python3.8/site-packages/alembic/ddl/impl.py", line 193, in _exec
    return conn.execute(construct, multiparams)
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1200, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/sql/ddl.py", line 77, in _execute_on_connection
    return connection._execute_ddl(
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1290, in _execute_ddl
    ret = self._execute_context(
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1748, in _execute_context
    self._handle_dbapi_exception(
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1929, in _handle_dbapi_exception
    util.raise_(
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_context
    self.dialect.do_execute(
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 716, in do_execute
    cursor.execute(statement, parameters)
  File "/home/airflow/.local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/home/airflow/.local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/home/airflow/.local/lib/python3.8/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1091, "Can't DROP 'task_reschedule_ti_fkey'; check that column/key exists")
[SQL: ALTER TABLE task_reschedule DROP FOREIGN KEY task_reschedule_ti_fkey]
(Background on this error at: http://sqlalche.me/e/14/e3q8)

Clients are using the following Dockerfile to create the custom image:

FROM apache/airflow:2.3.0-python3.8
COPY requirements.txt /tmp
RUN pip install --no-cache-dir --user \
    -c https://raw.githubusercontent.com/apache/airflow/constraints-2.3.0/constraints-3.8.txt \
    -r /tmp/requirements.txt

requirements.txt content:

apache-airflow-providers-apache-beam==3.3.0
apache-airflow-providers-google==6.8.0
apache-airflow-providers-pagerduty==2.1.3   # Constrained to v2.1.3 in Airflow2.3.0-Python3.8
apache-airflow-providers-http==2.1.2        # Constrained to v2.1.2 in Airflow2.3.0-Python3.8
google-apitools==0.5.32
google-cloud-firestore==2.3.4
pydantic==1.9.0
grpcio==1.44.0

it seems is trying to drop something but not sure what it its, this instance is running a lot of DAGs and it is possible that some tasks were running at the time of the upgrade, is there any previous steps or precautions that we need to take before updating the chart version or Airflow version? could you please help us to shed some light on this?

Please let me know if you might need further information.

Thanks!

[thesuperzapper]

I highly recommend against using Airflow 2.3.0, as there were many bugs. You might want to try using Airflow 2.3.4, and see if that fixes your issue.

https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#airflow-2-3-4-2022-08-23

[conchaox]

@thesuperzapper Haven't commented here in a while, I guess the main question would be if there's a recommended way for doing Airflow version upgrades with this Chart. I'm currently using the Terraform Helm release for managing the deployments and I always see errors when upgrading version, mostly related to DB migration issues, does't really matter which version I'm trying to migrate to, right now I need to jump from 2.1.x to 2.3.x since one of the clients needs to use triggers.

[conchaox]

trying to upgrade to 2.3.4 getting the same error pasted above:

sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1091, "Can't DROP 'task_reschedule_ti_fkey'; check that column/key exists")
[SQL: ALTER TABLE task_reschedule DROP FOREIGN KEY task_reschedule_ti_fkey]
(Background on this error at: http://sqlalche.me/e/14/e3q8)