Airflow 2.8.3 in K8s | Scheduler In crashloop

[sats268842]

helm chart version: 1.13.1
executor: LocalExecutor

Airflow Scheduler is crashlooping. Can you help me on solving this issue

[2024-05-22T01:19:37.678+0000] {task_context_logger.py:63} INFO - Task context logging is enabled
[2024-05-22T01:19:37.679+0000] {executor_loader.py:115} INFO - Loaded executor: LocalExecutor
2024-05-22T01:19:37.780074946Z [2024-05-22 01:19:37 +0000] [85] [INFO] Starting gunicorn 21.2.0
[2024-05-22 01:19:37 +0000] [85] [INFO] Listening at: http://[::]:8793 (85)
[2024-05-22 01:19:37 +0000] [85] [INFO] Using worker: sync
[2024-05-22 01:19:37 +0000] [86] [INFO] Booting worker with pid: 86
[2024-05-22 01:19:37 +0000] [87] [INFO] Booting worker with pid: 87
[2024-05-22T01:19:37.946+0000] {scheduler_job_runner.py:808} INFO - Starting the scheduler
[2024-05-22T01:19:37.948+0000] {scheduler_job_runner.py:815} INFO - Processing each file at most -1 times
[2024-05-22T01:19:38.276+0000] {manager.py:169} INFO - Launched DagFileProcessorManager with pid: 224
[2024-05-22T01:19:38.279+0000] {scheduler_job_runner.py:1608} INFO - Adopting or resetting orphaned tasks for active dag runs
[2024-05-22T01:19:38.288+0000] {settings.py:60} INFO - Configured default timezone UTC
[2024-05-22T01:19:38.323+0000] {settings.py:541} INFO - Loaded airflow_local_settings from /opt/airflow/config/airflow_local_settings.py .
[2024-05-22 01:20:22 +0000] [87] [INFO] Worker exiting (pid: 87)
[2024-05-22 01:20:22 +0000] [86] [INFO] Worker exiting (pid: 86)
[2024-05-22 01:20:22 +0000] [85] [INFO] Handling signal: term
[2024-05-22T01:20:22.524+0000] {scheduler_job_runner.py:258} INFO - Exiting gracefully upon receiving signal 15
[2024-05-22 01:20:22 +0000] [85] [INFO] Shutting down: Master
[2024-05-22T01:20:23.528+0000] {process_utils.py:131} INFO - Sending Signals.SIGTERM to group 224. PIDs of all processes in the group: [224]
[2024-05-22T01:20:23.529+0000] {process_utils.py:86} INFO - Sending the signal Signals.SIGTERM to group 224
[2024-05-22T01:20:23.983+0000] {process_utils.py:79} INFO - Process psutil.Process(pid=224, status='terminated', exitcode=0, started='01:19:37') (224) terminated with exit code 0
[2024-05-22T01:20:23.985+0000] {local_executor.py:402} INFO - Shutting down LocalExecutor; waiting for running tasks to finish.  Signal again if you don't want to wait.
[2024-05-22T01:20:23.986+0000] {scheduler_job_runner.py:878} ERROR - Exception when executing Executor.end
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job_runner.py", line 855, in _execute
    self._run_scheduler_loop()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job_runner.py", line 1001, in _run_scheduler_loop
    next_event = timers.run(blocking=False)
  File "/usr/local/lib/python3.8/sched.py", line 151, in run
    action(*argument, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/event_scheduler.py", line 40, in repeat
    action(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 79, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job_runner.py", line 1589, in _emit_pool_metrics
    pools = Pool.slots_stats(session=session)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 76, in wrapper
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/pool.py", line 177, in slots_stats
    pool_rows = session.execute(query)
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1716, in execute
    conn = self._connection_for_bind(bind)
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1555, in _connection_for_bind
    return self._transaction._connection_for_bind(
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 750, in _connection_for_bind
    conn = bind.connect()
2024-05-22T01:20:24.001914492Z   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/future/engine.py", line 412, in connect
    return super(Engine, self).connect()
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3325, in connect
    return self._connection_cls(self, close_with_result=close_with_result)
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
    else engine.raw_connection()
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3404, in raw_connection
    return self._wrap_pool_connect(self.pool.connect, _connection)
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3371, in _wrap_pool_connect
    return fn()
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 327, in connect
    return _ConnectionFairy._checkout(self)
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 995, in _checkout
    del fairy
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 928, in _checkout
    result = pool._dialect.do_ping(fairy.dbapi_connection)
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 873, in do_ping
    cursor.execute(self._dialect_specific_select_one)
  File "/usr/local/lib/python3.8/encodings/utf_8.py", line 15, in decode
    def decode(input, errors='strict'):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job_runner.py", line 261, in _exit_gracefully
    sys.exit(os.EX_OK)
SystemExit: 0

2024-05-22T01:20:24.002081355Z During handling of the above exception, another exception occurred:

2024-05-22T01:20:24.002091415Z Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job_runner.py", line 876, in _execute
    self.job.executor.end()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/local_executor.py", line 406, in end
    self.impl.end()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/local_executor.py", line 350, in end
    self.queue.put((None, None))
  File "<string>", line 2, in put
  File "/usr/local/lib/python3.8/multiprocessing/managers.py", line 834, in _callmethod
    conn.send((self._id, methodname, args, kwds))
  File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
2024-05-22T01:20:24.002185927Z BrokenPipeError: [Errno 32] Broken pipe
[2024-05-22T01:20:24.001+0000] {process_utils.py:131} INFO - Sending Signals.SIGTERM to group 224. PIDs of all processes in the group: []
[2024-05-22T01:20:24.002+0000] {process_utils.py:86} INFO - Sending the signal Signals.SIGTERM to group 224
[2024-05-22T01:20:24.003+0000] {process_utils.py:100} INFO - Sending the signal Signals.SIGTERM to process 224 as process group is missing.
[2024-05-22T01:20:24.003+0000] {scheduler_job_runner.py:884} INFO - Exited execute loop