Scheduler was dead when database service failover.

[robintian001]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.7.1

What happened?

Scheduler was dead when database service failover.

What you think should happen instead?

No response

How to reproduce

The mysql service failover is triggered manually.

Operating System

Linux (rhel 9.2) RedHat

Versions of Apache Airflow Providers

Airflow 2.7.1

Deployment

Other

Deployment details

deploment on azure vm, the database is MySQL Flexible Server of Azure.

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[aritra24]

@robintian001 could you share some logs for the same. Also if my understanding is right when your mysql service is failing over it causes the scheduler to exit, if so that seems reasonable since it expects an active connection to be available for it to run? cc: @potiuk

[robintian001]

The Scheduler's logs

2024-03-16 17:11:08,194 INFO - Adopting or resetting orphaned tasks for active dag runs
2024-03-16 17:11:22,864 ERROR - Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3371, in _wrap_pool_connect
return fn()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 327, in connect
return _ConnectionFairy._checkout(self)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 894, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 498, in checkout
rec.checkin_failed(err, fairy_was_created=False)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit
compat.raise(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise
raise exception
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
dbapi_connection = rec.get_connection()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 630, in get_connection
self.__connect()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 691, in connect
pool.logger.debug("Error on connect(): %s", e)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit
compat.raise(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise
raise exception
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 686, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 574, in connect
return dialect.connect(*cargs, **cparams)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 598, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/init.py", line 121, in Connect
return Connection(*args, **kwargs)
File "/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/connections.py", line 193, in init
super().init(*args, **kwargs2)
MySQLdb.OperationalError: (2003, "Can't connect to MySQL server on 'apec-mysql-flexible.mysql.database.azure.com:3306' (111)")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py", line 845, in _execute
self._run_scheduler_loop()
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py", line 979, in _run_scheduler_loop
num_queued_tis = self._do_scheduling(session)
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py", line 1053, in _do_scheduling
self._create_dagruns_for_dags(guard, session)
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/utils/retries.py", line 91, in wrapped_function
for attempt in run_with_db_retries(max_retries=retries, logger=logger, **retry_kwargs):
File "/home/airflow-user/.local/lib/python3.9/site-packages/tenacity/init.py", line 347, in iter
do = self.iter(retry_state=retry_state)
File "/home/airflow-user/.local/lib/python3.9/site-packages/tenacity/init.py", line 325, in iter
raise retry_exc.reraise()
File "/home/airflow-user/.local/lib/python3.9/site-packages/tenacity/init.py", line 158, in reraise
raise self.last_attempt.result()
File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/utils/retries.py", line 100, in wrapped_function
return func(*args, **kwargs)
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py", line 1120, in _create_dagruns_for_dags
query, dataset_triggered_dag_info = DagModel.dags_needing_dagruns(session)
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/models/dag.py", line 3615, in dags_needing_dagruns
for x in session.execute(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1716, in execute
conn = self._connection_for_bind(bind)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1555, in _connection_for_bind
return self._transaction._connection_for_bind(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 750, in _connection_for_bind
conn = bind.connect()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/future/engine.py", line 406, in connect
return super(Engine, self).connect()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3325, in connect
return self._connection_cls(self, close_with_result=close_with_result)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 96, in init
else engine.raw_connection()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3404, in raw_connection
return self._wrap_pool_connect(self.pool.connect, _connection)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3374, in _wrap_pool_connect
Connection.handle_dbapi_exception_noconnection(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2208, in handle_dbapi_exception_noconnection
util.raise(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise
raise exception
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3371, in _wrap_pool_connect
return fn()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 327, in connect
return _ConnectionFairy._checkout(self)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 894, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 498, in checkout
rec.checkin_failed(err, fairy_was_created=False)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit
compat.raise(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise
raise exception
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
dbapi_connection = rec.get_connection()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 630, in get_connection
self.__connect()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 691, in connect
pool.logger.debug("Error on connect(): %s", e)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit
compat.raise(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise
raise exception
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 686, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 574, in connect
return dialect.connect(*cargs, **cparams)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 598, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/init.py", line 121, in Connect
return Connection(*args, **kwargs)
File "/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/connections.py", line 193, in init
super().init(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (2003, "Can't connect to MySQL server on 'apec-mysql-flexible.mysql.database.azure.com:3306' (111)")
(Background on this error at: https://sqlalche.me/e/14/e3q8)
2024-03-16 17:11:22,868 INFO - Shutting down LocalExecutor; waiting for running tasks to finish. Signal again if you don't want to wait.
2024-03-16 17:11:22,910 INFO - Sending Signals.SIGTERM to group 1732789. PIDs of all processes in the group: []
2024-03-16 17:11:22,910 INFO - Sending the signal Signals.SIGTERM to group 1732789
2024-03-16 17:11:22,910 INFO - Sending the signal Signals.SIGTERM to process 1732789 as process group is missing.
2024-03-16 17:11:22,910 INFO - Exited execute loop
2024-03-16 17:11:23,161 ERROR - Exception when running scheduler job
Traceback (most recent call last):
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3371, in _wrap_pool_connect
return fn()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 327, in connect
return _ConnectionFairy._checkout(self)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 894, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 498, in checkout
rec.checkin_failed(err, fairy_was_created=False)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit
compat.raise(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise
raise exception
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
dbapi_connection = rec.get_connection()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 630, in get_connection
self.__connect()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 691, in connect
pool.logger.debug("Error on connect(): %s", e)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit
compat.raise(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise
raise exception
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 686, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 574, in connect
return dialect.connect(*cargs, **cparams)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 598, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/init.py", line 121, in Connect
return Connection(*args, **kwargs)
File "/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/connections.py", line 193, in init
super().init(*args, **kwargs2)
MySQLdb.OperationalError: (2003, "Can't connect to MySQL server on 'apec-mysql-flexible.mysql.database.azure.com:3306' (111)")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/job.py", line 289, in run_job
return execute_job(job, execute_callable=execute_callable)
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/job.py", line 318, in execute_job
ret = execute_callable()
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py", line 845, in _execute
self._run_scheduler_loop()
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py", line 979, in _run_scheduler_loop
num_queued_tis = self._do_scheduling(session)
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py", line 1053, in _do_scheduling
self._create_dagruns_for_dags(guard, session)
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/utils/retries.py", line 91, in wrapped_function
for attempt in run_with_db_retries(max_retries=retries, logger=logger, **retry_kwargs):
File "/home/airflow-user/.local/lib/python3.9/site-packages/tenacity/init.py", line 347, in iter
do = self.iter(retry_state=retry_state)
File "/home/airflow-user/.local/lib/python3.9/site-packages/tenacity/init.py", line 325, in iter
raise retry_exc.reraise()
File "/home/airflow-user/.local/lib/python3.9/site-packages/tenacity/init.py", line 158, in reraise
raise self.last_attempt.result()
File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/utils/retries.py", line 100, in wrapped_function
return func(*args, **kwargs)
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py", line 1120, in _create_dagruns_for_dags
query, dataset_triggered_dag_info = DagModel.dags_needing_dagruns(session)
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/models/dag.py", line 3615, in dags_needing_dagruns
for x in session.execute(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1716, in execute
conn = self._connection_for_bind(bind)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1555, in _connection_for_bind
return self._transaction._connection_for_bind(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 750, in _connection_for_bind
conn = bind.connect()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/future/engine.py", line 406, in connect
return super(Engine, self).connect()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3325, in connect
return self._connection_cls(self, close_with_result=close_with_result)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 96, in init
else engine.raw_connection()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3404, in raw_connection
return self._wrap_pool_connect(self.pool.connect, _connection)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3374, in _wrap_pool_connect
Connection.handle_dbapi_exception_noconnection(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2208, in handle_dbapi_exception_noconnection
util.raise(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise
raise exception
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3371, in _wrap_pool_connect
return fn()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 327, in connect
return _ConnectionFairy._checkout(self)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 894, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 498, in checkout
rec.checkin_failed(err, fairy_was_created=False)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit
compat.raise(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise
raise exception
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
dbapi_connection = rec.get_connection()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 630, in get_connection
self.__connect()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 691, in connect
pool.logger.debug("Error on connect(): %s", e)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit
compat.raise(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise
raise exception
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 686, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 574, in connect
return dialect.connect(*cargs, **cparams)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 598, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/init.py", line 121, in Connect
return Connection(*args, **kwargs)
File "/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/connections.py", line 193, in init
super().init(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (2003, "Can't connect to MySQL server on 'apec-mysql-flexible.mysql.database.azure.com:3306' (111)")
(Background on this error at: https://sqlalche.me/e/14/e3q8)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3371, in _wrap_pool_connect
return fn()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 327, in connect
return _ConnectionFairy._checkout(self)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 894, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 498, in checkout
rec.checkin_failed(err, fairy_was_created=False)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit
compat.raise(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise
raise exception
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
dbapi_connection = rec.get_connection()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 630, in get_connection
self.__connect()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 691, in connect
pool.logger.debug("Error on connect(): %s", e)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit
compat.raise(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise
raise exception
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 686, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 574, in connect
return dialect.connect(*cargs, **cparams)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 598, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/init.py", line 121, in Connect
return Connection(*args, **kwargs)
File "/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/connections.py", line 193, in init
super().init(*args, **kwargs2)
MySQLdb.OperationalError: (2003, "Can't connect to MySQL server on 'apec-mysql-flexible.mysql.database.azure.com:3306' (111)")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/cli/commands/scheduler_command.py", line 47, in _run_scheduler_job
run_job(job=job_runner.job, execute_callable=job_runner._execute)
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 77, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/job.py", line 291, in run_job
job.complete_execution(session=session)
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 74, in wrapper
return func(*args, **kwargs)
File "/home/airflow-user/.local/lib/python3.9/site-packages/airflow/jobs/job.py", line 237, in complete_execution
session.merge(self)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3056, in merge
return self._merge(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3136, in _merge
merged = self.get(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 2853, in get
return self._get_impl(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 2975, in _get_impl
return db_load_fn(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 530, in load_on_pk_identity
session.execute(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1716, in execute
conn = self._connection_for_bind(bind)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1555, in _connection_for_bind
return self._transaction._connection_for_bind(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 750, in _connection_for_bind
conn = bind.connect()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/future/engine.py", line 406, in connect
return super(Engine, self).connect()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3325, in connect
return self._connection_cls(self, close_with_result=close_with_result)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 96, in init
else engine.raw_connection()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3404, in raw_connection
return self._wrap_pool_connect(self.pool.connect, _connection)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3374, in _wrap_pool_connect
Connection.handle_dbapi_exception_noconnection(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2208, in handle_dbapi_exception_noconnection
util.raise(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise
raise exception
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3371, in _wrap_pool_connect
return fn()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 327, in connect
return _ConnectionFairy._checkout(self)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 894, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 498, in checkout
rec.checkin_failed(err, fairy_was_created=False)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit
compat.raise(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise
raise exception
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
dbapi_connection = rec.get_connection()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 630, in get_connection
self.__connect()
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 691, in connect
pool.logger.debug("Error on connect(): %s", e)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit
compat.raise(
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise
raise exception
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 686, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 574, in connect
return dialect.connect(*cargs, **cparams)
File "/home/airflow-user/.local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 598, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/init.py", line 121, in Connect
return Connection(*args, **kwargs)
File "/home/airflow-user/.local/lib/python3.9/site-packages/MySQLdb/connections.py", line 193, in init
super().init(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (2003, "Can't connect to MySQL server on 'apec-mysql-flexible.mysql.database.azure.com:3306' (111)")
(Background on this error at: https://sqlalche.me/e/14/e3q8)

---

this message from microsoft support team

It can be seen from the background connection log that your standby server restarts at 17:00 UTC on 3/16 for regular maintenance, and restarts at 17:07 UTC on 3/16 for about 7 minutes. In this case, your standby server will be failover to the new primary server for you to connect. At the same time, your original primary server restarts at 17:11 UTC on 3/16 and completes the restart at 17:19 UTC on 3/16, which takes about 8 minutes. From the connection log, your initial standby server restarts before the initial primary server, so your standby server that has restarted can connect before your initial primary server restarts.

From the figure below about the conversion of HA server, you can see that at 17:12 UTC on 3/16, your HA server has been switched, which means that at this time your original standby server has become your new primary server available for connection. As you can see from the following figure, your total failover time is about 2 minutes. If you feel that you are not connected for about 20 minutes, it may be related to the retry mechanism of your client.