Airflow 2.2.3 with Schedulers HA Failing on postgres 13

[thsubramani]

Apache Airflow version

2.2.3 (latest released)

What happened

airflow scheduler POD with HA connecting to same postgres DB encounters below error even though use_row_level_locking=true dy defualt/explicitly setting as env.

airflow-c-qa4-64956645c-tr4zm airflow-scheduler [2022-01-05 17:18:26,109] {scheduler_job.py:721} INFO - Exited execute loop
airflow-c-qa4-64956645c-tr4zm airflow-scheduler Traceback (most recent call last):
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
airflow-c-qa4-64956645c-tr4zm airflow-scheduler cursor, statement, parameters, context
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
airflow-c-qa4-64956645c-tr4zm airflow-scheduler cursor.execute(statement, parameters)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler psycopg2.errors.LockNotAvailable: could not obtain lock on row in relation "slot_pool"
airflow-c-qa4-64956645c-tr4zm airflow-scheduler
airflow-c-qa4-64956645c-tr4zm airflow-scheduler
airflow-c-qa4-64956645c-tr4zm airflow-scheduler The above exception was the direct cause of the following exception:
airflow-c-qa4-64956645c-tr4zm airflow-scheduler
airflow-c-qa4-64956645c-tr4zm airflow-scheduler Traceback (most recent call last):
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/bin/airflow", line 8, in
airflow-c-qa4-64956645c-tr4zm airflow-scheduler sys.exit(main())
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/main.py", line 40, in main
airflow-c-qa4-64956645c-tr4zm airflow-scheduler args.func(args)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/cli_parser.py", line 48, in command
airflow-c-qa4-64956645c-tr4zm airflow-scheduler return func(*args, **kwargs)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/cli.py", line 91, in wrapper
airflow-c-qa4-64956645c-tr4zm airflow-scheduler return f(*args, **kwargs)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/commands/scheduler_command.py", line 70, in scheduler
airflow-c-qa4-64956645c-tr4zm airflow-scheduler job.run()
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/base_job.py", line 245, in run
airflow-c-qa4-64956645c-tr4zm airflow-scheduler self._execute()
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 694, in _execute
airflow-c-qa4-64956645c-tr4zm airflow-scheduler self._run_scheduler_loop()
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 787, in _run_scheduler_loop
airflow-c-qa4-64956645c-tr4zm airflow-scheduler num_queued_tis = self._do_scheduling(session)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 926, in _do_scheduling
airflow-c-qa4-64956645c-tr4zm airflow-scheduler num_queued_tis = self._critical_section_execute_task_instances(session=session)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 550, in _critical_section_execute_task_instances
airflow-c-qa4-64956645c-tr4zm airflow-scheduler queued_tis = self._executable_task_instances_to_queued(max_tis, session=session)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/session.py", line 67, in wrapper
airflow-c-qa4-64956645c-tr4zm airflow-scheduler return func(*args, **kwargs)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 307, in _executable_task_instances_to_queued
airflow-c-qa4-64956645c-tr4zm airflow-scheduler pools = models.Pool.slots_stats(lock_rows=True, session=session)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/session.py", line 67, in wrapper
airflow-c-qa4-64956645c-tr4zm airflow-scheduler return func(*args, **kwargs)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/pool.py", line 107, in slots_stats
airflow-c-qa4-64956645c-tr4zm airflow-scheduler pool_rows: Iterable[Tuple[str, int]] = query.all()
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3319, in all
airflow-c-qa4-64956645c-tr4zm airflow-scheduler return list(self)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3481, in iter
airflow-c-qa4-64956645c-tr4zm airflow-scheduler return self._execute_and_instances(context)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
airflow-c-qa4-64956645c-tr4zm airflow-scheduler result = conn.execute(querycontext.statement, self._params)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
airflow-c-qa4-64956645c-tr4zm airflow-scheduler return meth(self, multiparams, params)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
airflow-c-qa4-64956645c-tr4zm airflow-scheduler return connection._execute_clauseelement(self, multiparams, params)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1139, in execute_clauseelement
airflow-c-qa4-64956645c-tr4zm airflow-scheduler distilled_params,
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1324, in execute_context
airflow-c-qa4-64956645c-tr4zm airflow-scheduler e, statement, parameters, cursor, context
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1515, in handle_dbapi_exception
airflow-c-qa4-64956645c-tr4zm airflow-scheduler util.raise(newraise, with_traceback=exc_info[2], from=e)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise
airflow-c-qa4-64956645c-tr4zm airflow-scheduler raise exception
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
airflow-c-qa4-64956645c-tr4zm airflow-scheduler cursor, statement, parameters, context
airflow-c-qa4-64956645c-tr4zm airflow-scheduler File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
airflow-c-qa4-64956645c-tr4zm airflow-scheduler cursor.execute(statement, parameters)
airflow-c-qa4-64956645c-tr4zm airflow-scheduler AttributeError: 'PGExecutionContext_psycopg2' object has no attribute '_stan_scope'

What you expected to happen

airflow scheduler should not crash

How to reproduce

bring uo airflow scheduler 2 pods connecting to same postgres DB

Operating System

docker

Versions of Apache Airflow Providers

2.1.0,2.1.1,2.1.3

Deployment

Other Docker-based deployment

Deployment details

k8s

Anything else

nothing much

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

I think this problem is because you are not using proper installation method. Old version of postgres libraries apparently exhibit this problem but later version should be fine (at least that's why quick search returned)

Follow the instructions here: https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html