Airflow worker pod not able to fetch connections information during its execution

[kardamk]

Facing an issue while upgrading from Airflow 2.11.x to 3.1.8 hosted on OpenShift

I keep getting the error, "httpx.HTTPStatusError: Server error '503 Service Unavailable' for url 'http://airflow-api-service:8080/execution/connections/aws_default" when Airflow DAG is executed within a pod.

I tested the connectivity from worker pod <-> api-server pod using curl and was able to connect successfully to the API server ruling out network connectivity issues.

As suggested here #55380 (comment), I already tried configuring the environment variable AIRFLOW__CORE__EXECUTION_API_SERVER_URL=http://airflow-api-service:8080/execution to no avail.

Requesting for any inputs from the community to analyze and debug the issue further.

Deployment Information

Apache Airflow 3.1.8 deployed on OpenShift version 4.18.41 using official Airflow Helm chart version 1.19.0. Currently, migrating from 2.11.x to 3.1.8. Remote logging is enabled with an S3 compatible storage. Metadatabase is a Postgres crunchy cluster hosted on OpenShift within namespace.

Apache Airflow

version | 3.1.8
executor | KubernetesExecutor
task_logging_handler | airflow.utils.log.file_task_handler.FileTaskHandler
sql_alchemy_conn | postgresql+psycopg2://xxxx
dags_folder | /opt/airflow/dags/repo/dags
plugins_folder | /opt/airflow/plugins
base_log_folder | /opt/airflow/logs
remote_base_log_folder | s3://xxx/airflow/yyyy/

System info

OS | Linux
architecture | x86_64
uname | uname_result(system='Linux', node='xxx, release='5.14.0-427.125.1.el9_4.x86_64', version='#1 SMP PREEMPT_DYNAMIC Sat May 9 17:31:37 EDT 2026', machine='x86_64')
locale | ('C', 'UTF-8')
python_version | 3.12.13 (main, Mar 11 2026, 20:29:10) [GCC 12.2.0]
python_location | /usr/python/bin/python3.12

Tools info

git | git version 2.39.5
ssh | OpenSSH_9.2p1 Debian-2+deb12u7, OpenSSL 3.0.18 30 Sep 2025
kubectl | NOT AVAILABLE
gcloud | NOT AVAILABLE
cloud_sql_proxy | NOT AVAILABLE
mysql | mysql Ver 15.1 Distrib 10.11.16-MariaDB, for debian-linux-gnu (x86_64) using EditLine wrapper
sqlite3 | 3.40.1 2022-12-28 14:03:47 df5c253c0b3dd24916e4ec7cf77d3db5294cc9fd45ae7b9c5e82ad8197f3alt1
psql | psql (PostgreSQL) 18.3 (Debian 18.3-1.pgdg12+1)

Providers info

apache-airflow-providers-amazon | 9.22.0
apache-airflow-providers-celery | 3.17.0
apache-airflow-providers-cncf-kubernetes | 10.8.2
apache-airflow-providers-common-compat | 1.14.0
apache-airflow-providers-common-io | 1.7.1
apache-airflow-providers-common-messaging | 2.0.2
apache-airflow-providers-common-sql | 1.30.4
apache-airflow-providers-docker | 4.5.2
apache-airflow-providers-elasticsearch | 6.5.0
apache-airflow-providers-fab | 3.4.0
apache-airflow-providers-ftp | 3.14.1
apache-airflow-providers-git | 0.2.4
apache-airflow-providers-google | 20.0.0
apache-airflow-providers-grpc | 3.9.2
apache-airflow-providers-hashicorp | 4.5.0
apache-airflow-providers-http | 6.0.0
apache-airflow-providers-microsoft-azure | 13.0.0
apache-airflow-providers-microsoft-mssql | 4.4.1
apache-airflow-providers-mysql | 6.5.0
apache-airflow-providers-odbc | 4.12.0
apache-airflow-providers-openlineage | 2.11.0
apache-airflow-providers-postgres | 6.6.0
apache-airflow-providers-redis | 4.4.2
apache-airflow-providers-samba | 4.12.2
apache-airflow-providers-sendgrid | 4.2.1
apache-airflow-providers-sftp | 5.7.0
apache-airflow-providers-slack | 9.7.0
apache-airflow-providers-smtp | 2.4.2
apache-airflow-providers-snowflake | 6.9.0
apache-airflow-providers-ssh | 4.3.1
apache-airflow-providers-standard | 1.12.0

Detailed error logs

{"timestamp":"2026-07-10T19:31:45.305110Z","level":"info","event":"Connecting to server:","server":"http://airflow-api-service:8080/execution","logger":"__main__","filename":"execute_workload.py","lineno":64}
{"timestamp":"2026-07-10T19:31:45.334611Z","level":"debug","event":"Connecting to execution API server","server":"http://airflow-api-service:8080/execution","logger":"supervisor","filename":"supervisor.py","lineno":1964}
{"timestamp":"2026-07-10T19:31:45.338203Z","level":"debug","event":"connect_tcp.started host='internal-api-host' port=8080 local_address=None timeout=5.0 socket_options=None","logger":"httpcore.connection","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:45.345337Z","level":"debug","event":"connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7f11d8165b20>","logger":"httpcore.connection","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:45.345626Z","level":"debug","event":"send_request_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:45.345986Z","level":"debug","event":"send_request_headers.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:45.346139Z","level":"debug","event":"send_request_body.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:45.346304Z","level":"debug","event":"send_request_body.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:45.346415Z","level":"debug","event":"receive_response_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:45.348000Z","level":"debug","event":"receive_response_headers.complete return_value=(b'HTTP/1.0', 503, b'Service Unavailable', [(b'Connection', b'close')])","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:45.348756Z","level":"debug","event":"response_closed.started","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:45.348951Z","level":"debug","event":"response_closed.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:45.349252Z","level":"warning","event":"Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it.","logger":"airflow.sdk.api.client","filename":"before.py","lineno":40}
{"timestamp":"2026-07-10T19:31:46.350088Z","level":"debug","event":"connect_tcp.started host='internal-api-host' port=8080 local_address=None timeout=5.0 socket_options=None","logger":"httpcore.connection","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:46.357830Z","level":"debug","event":"connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7f11d81664b0>","logger":"httpcore.connection","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:46.358111Z","level":"debug","event":"send_request_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:46.358405Z","level":"debug","event":"send_request_headers.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:46.358538Z","level":"debug","event":"send_request_body.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:46.358684Z","level":"debug","event":"send_request_body.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:46.358789Z","level":"debug","event":"receive_response_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:46.360397Z","level":"debug","event":"receive_response_headers.complete return_value=(b'HTTP/1.0', 503, b'Service Unavailable', [(b'Connection', b'close')])","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:46.360815Z","level":"debug","event":"response_closed.started","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:46.360991Z","level":"debug","event":"response_closed.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:46.361226Z","level":"warning","event":"Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it.","logger":"airflow.sdk.api.client","filename":"before.py","lineno":40}
{"timestamp":"2026-07-10T19:31:48.214846Z","level":"debug","event":"connect_tcp.started host='internal-api-host' port=8080 local_address=None timeout=5.0 socket_options=None","logger":"httpcore.connection","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:48.222447Z","level":"debug","event":"connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7f11d8166ea0>","logger":"httpcore.connection","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:48.222715Z","level":"debug","event":"send_request_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:48.223010Z","level":"debug","event":"send_request_headers.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:48.223155Z","level":"debug","event":"send_request_body.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:48.223300Z","level":"debug","event":"send_request_body.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:48.223642Z","level":"debug","event":"receive_response_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:48.225096Z","level":"debug","event":"receive_response_headers.complete return_value=(b'HTTP/1.0', 503, b'Service Unavailable', [(b'Connection', b'close')])","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:48.225478Z","level":"debug","event":"response_closed.started","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:48.225640Z","level":"debug","event":"response_closed.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:48.225821Z","level":"warning","event":"Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it.","logger":"airflow.sdk.api.client","filename":"before.py","lineno":40}
{"timestamp":"2026-07-10T19:31:51.414937Z","level":"debug","event":"connect_tcp.started host='internal-api-host' port=8080 local_address=None timeout=5.0 socket_options=None","logger":"httpcore.connection","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:51.421926Z","level":"debug","event":"connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7f11d8166540>","logger":"httpcore.connection","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:51.422180Z","level":"debug","event":"send_request_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:51.422440Z","level":"debug","event":"send_request_headers.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:51.422570Z","level":"debug","event":"send_request_body.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:51.422712Z","level":"debug","event":"send_request_body.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:51.422818Z","level":"debug","event":"receive_response_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:51.424488Z","level":"debug","event":"receive_response_headers.complete return_value=(b'HTTP/1.0', 503, b'Service Unavailable', [(b'Connection', b'close')])","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:51.424870Z","level":"debug","event":"response_closed.started","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:51.425034Z","level":"debug","event":"response_closed.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:51.425239Z","level":"warning","event":"Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it.","logger":"airflow.sdk.api.client","filename":"before.py","lineno":40}
{"timestamp":"2026-07-10T19:31:55.267800Z","level":"debug","event":"connect_tcp.started host='internal-api-host' port=8080 local_address=None timeout=5.0 socket_options=None","logger":"httpcore.connection","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:55.277167Z","level":"debug","event":"connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7f11d8165af0>","logger":"httpcore.connection","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:55.277425Z","level":"debug","event":"send_request_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:55.277703Z","level":"debug","event":"send_request_headers.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:55.277836Z","level":"debug","event":"send_request_body.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:55.278030Z","level":"debug","event":"send_request_body.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:55.278178Z","level":"debug","event":"receive_response_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:55.280298Z","level":"debug","event":"receive_response_headers.complete return_value=(b'HTTP/1.0', 503, b'Service Unavailable', [(b'Connection', b'close')])","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:55.280656Z","level":"debug","event":"response_closed.started","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
{"timestamp":"2026-07-10T19:31:55.280816Z","level":"debug","event":"response_closed.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47}
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/execute_workload.py", line 125, in
main()
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/execute_workload.py", line 121, in main
execute_workload(workload)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/execute_workload.py", line 66, in execute_workload
supervise(
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1972, in supervise
logger, log_file_descriptor = _configure_logging(log_path, client)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1887, in _configure_logging
with _remote_logging_conn(client):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/contextlib.py", line 137, in enter
return next(self.gen)
^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 899, in _remote_logging_conn
conn = _fetch_remote_logging_conn(conn_id, client)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 863, in _fetch_remote_logging_conn
conn = client.connections.get(conn_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 361, in get
resp = self.client.get(f"connections/{conn_id}")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1053, in get
return self.request(
^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/init.py", line 331, in wrapped_f
return copy(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/init.py", line 470, in call
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/init.py", line 371, in iter
result = action(retry_state)
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/init.py", line 413, in exc_check
raise retry_exc.reraise()
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/init.py", line 184, in reraise
raise self.last_attempt.result()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/init.py", line 473, in call
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 887, in request
return super().request(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 825, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
response = self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
response = self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 999, in _send_handling_redirects
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 982, in _send_handling_redirects
hook(response)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 186, in raise_on_4xx_5xx_with_note
return get_json_error(response) or response.raise_for_status()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_models.py", line 829, in raise_for_status
raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Server error '503 Service Unavailable' for url 'http://airflow-api-service:8080/execution/connections/aws_default'