HELM CHART over ARGOCD on OpenShift -> Scheudular disconnections

[m3t41]

Apache Airflow version

2.3.4

What happened

I am an SRE, and i have a Problem to get the Helm Chart up und Running on an OpenShift Cluster.

changes i made to the chart

Executor = KubernetesExecutor
SecurityContext disable in all Stages
Set Limits in all Stages
Redis = disable

that's it

and then i get these errors
�[34m2022-09-13 05:52:43,424�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c477efc70>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True�[0m
[�[34m2022-09-13 05:52:43,424�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839f0a60>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue�[0m
[�[34m2022-09-13 05:54:54,496�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839f0f40>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue�[0m
[�[34m2022-09-13 05:54:54,496�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c477efe80>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True�[0m
[�[34m2022-09-13 05:57:05,568�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c47808040>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True�[0m
[�[34m2022-09-13 05:57:05,568�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839f0df0>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue�[0m

What you think should happen instead

`[38;5;6mairflow-scheduler �[38;5;5m05:50:17.69 �[0m
�[38;5;6mairflow-scheduler �[38;5;5m05:50:17.69 �[0m�[1mWelcome to the Bitnami airflow-scheduler container�[0m
�[38;5;6mairflow-scheduler �[38;5;5m05:50:17.69 �[0mSubscribe to project updates by watching �[1mhttps://github.com/bitnami/containers�[0m
�[38;5;6mairflow-scheduler �[38;5;5m05:50:17.70 �[0mSubmit issues and feature requests at �[1mhttps://github.com/bitnami/containers/issues�[0m
�[38;5;6mairflow-scheduler �[38;5;5m05:50:17.70 �[0m
�[38;5;6mairflow-scheduler �[38;5;5m05:50:17.75 �[0m�[38;5;2mINFO �[0m ==> Enabling non-root system user with nss_wrapper
�[38;5;6mairflow-scheduler �[38;5;5m05:50:17.76 �[0m�[38;5;2mINFO �[0m ==> ** Starting Airflow setup **
�[38;5;6mairflow-scheduler �[38;5;5m05:50:17.87 �[0m�[38;5;2mINFO �[0m ==> No injected configuration file found. Creating default config file
�[38;5;6mairflow-scheduler �[38;5;5m05:50:19.65 �[0m�[38;5;2mINFO �[0m ==> Configuring Airflow database
�[38;5;6mairflow-scheduler �[38;5;5m05:50:19.67 �[0m�[38;5;2mINFO �[0m ==> Trying to connect to the database server
�[38;5;6mairflow-scheduler �[38;5;5m05:50:23.37 �[0m�[38;5;2mINFO �[0m ==> Waiting for Airflow Webserser to be up
�[38;5;6mairflow-scheduler �[38;5;5m05:50:23.45 �[0m�[38;5;2mINFO �[0m ==> ** Airflow setup finished! **

�[38;5;6mairflow-scheduler �[38;5;5m05:50:23.47 �[0m�[38;5;2mINFO �[0m ==> ** Starting Airflow **

---

____ |( )_______ / /________ __
____ /| |_ /__ / / __ / __ _ | /| / /
___ ___ | / _ / _ / _ / / // / |/ |/ /
// |// // // // _/____/|__/
[�[34m2022-09-13 05:50:31,573�[0m] {�[34mscheduler_job.py:�[0m708} INFO�[0m - Starting the scheduler�[0m
[�[34m2022-09-13 05:50:31,573�[0m] {�[34mscheduler_job.py:�[0m713} INFO�[0m - Processing each file at most -1 times�[0m
[�[34m2022-09-13 05:50:31,578�[0m] {�[34mexecutor_loader.py:�[0m105} INFO�[0m - Loaded executor: KubernetesExecutor�[0m
[�[34m2022-09-13 05:50:31,675�[0m] {�[34mkubernetes_executor.py:�[0m520} INFO�[0m - Start Kubernetes executor�[0m
[�[34m2022-09-13 05:50:31,856�[0m] {�[34mkubernetes_executor.py:�[0m128} INFO�[0m - Event: and now my watch begins starting at resource_version: 0�[0m
[�[34m2022-09-13 05:50:31,875�[0m] {�[34mkubernetes_executor.py:�[0m469} INFO�[0m - Found 0 queued task instances�[0m
[�[34m2022-09-13 05:50:31,957�[0m] {�[34mmanager.py:�[0m160} INFO�[0m - Launched DagFileProcessorManager with pid: 124�[0m
[�[34m2022-09-13 05:50:31,959�[0m] {�[34mscheduler_job.py:�[0m1233} INFO�[0m - Resetting orphaned tasks for active dag runs�[0m
[�[34m2022-09-13 05:50:31,965�[0m] {�[34msettings.py:�[0m55} INFO�[0m - Configured default timezone Timezone('UTC')�[0m
[�[34m2022-09-13 05:50:31,969�[0m] {�[34mscheduler_job.py:�[0m1256} INFO�[0m - Marked 1 SchedulerJob instances as failed�[0m
[�[34m2022-09-13 05:52:43,424�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c477efc70>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True�[0m
[�[34m2022-09-13 05:52:43,424�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839f0a60>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue�[0m
[�[34m2022-09-13 05:54:54,496�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839f0f40>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue�[0m
[�[34m2022-09-13 05:54:54,496�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c477efe80>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True�[0m
[�[34m2022-09-13 05:57:05,568�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c47808040>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True�[0m
[�[34m2022-09-13 05:57:05,568�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839f0df0>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue�[0m
[�[34m2022-09-13 05:59:16,640�[0m] {�[34mkubernetes_executor.py:�[0m111} ERROR�[0m - Unknown error in KubernetesJobWatcher. Failing�[0m
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
conn.connect()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 358, in connect
self.sock = conn = self._new_conn()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f9c47808190>: Failed to establish a new connection: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 102, in run
self.resource_version = self._run(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 147, in _run
for event in list_worker_pods():
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/watch/watch.py", line 163, in stream
resp = func(*args, **kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15697, in list_namespaced_pod
return self.list_namespaced_pod_with_http_info(namespace, **kwargs) # noqa: E501
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15812, in list_namespaced_pod_with_http_info
return self.api_client.call_api(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/rest.py", line 240, in GET
return self.request("GET", url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/rest.py", line 213, in request
r = self.pool_manager.request(method, url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/request.py", line 74, in request
return self.request_encode_url(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/request.py", line 96, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/poolmanager.py", line 376, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
retries = retries.increment(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='172.26.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c47808190>: Failed to establish a new connection: [Errno 110] Connection timed out'))
[�[34m2022-09-13 05:59:16,642�[0m] {�[34mscheduler_job.py:�[0m768} ERROR�[0m - Exception when executing SchedulerJob._run_scheduler_loop�[0m
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
conn.connect()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 358, in connect
self.sock = conn = self._new_conn()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f9c839d7430>: Failed to establish a new connection: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 751, in _execute
self._run_scheduler_loop()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 806, in _run_scheduler_loop
self.adopt_or_reset_orphaned_tasks()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/session.py", line 71, in wrapper
return func(*args, session=session, **kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1236, in adopt_or_reset_orphaned_tasks
for attempt in run_with_db_retries(logger=self.log):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/tenacity/init.py", line 382, in iter
do = self.iter(retry_state=retry_state)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/tenacity/init.py", line 349, in iter
return fut.result()
File "/opt/bitnami/python/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/opt/bitnami/python/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1281, in adopt_or_reset_orphaned_tasks
to_reset = self.executor.try_adopt_task_instances(tis_to_reset_or_adopt)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 713, in try_adopt_task_instances
self._adopt_completed_pods(kube_client)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 760, in _adopt_completed_pods
pod_list = kube_client.list_namespaced_pod(namespace=self.kube_config.kube_namespace, **kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15697, in list_namespaced_pod
return self.list_namespaced_pod_with_http_info(namespace, **kwargs) # noqa: E501
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15812, in list_namespaced_pod_with_http_info
return self.api_client.call_api(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/rest.py", line 240, in GET
return self.request("GET", url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/rest.py", line 213, in request
r = self.pool_manager.request(method, url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/request.py", line 74, in request
return self.request_encode_url(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/request.py", line 96, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/poolmanager.py", line 376, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
retries = retries.increment(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='172.26.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839d7430>: Failed to establish a new connection: [Errno 110] Connection timed out'))
[�[34m2022-09-13 05:59:16,650�[0m] {�[34mkubernetes_executor.py:�[0m821} INFO�[0m - Shutting down Kubernetes executor�[0m
Process KubernetesJobWatcher-3:
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
conn.connect()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 358, in connect
self.sock = conn = self._new_conn()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f9c47808190>: Failed to establish a new connection: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 102, in run
self.resource_version = self._run(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 147, in _run
for event in list_worker_pods():
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/watch/watch.py", line 163, in stream
resp = func(*args, **kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15697, in list_namespaced_pod
return self.list_namespaced_pod_with_http_info(namespace, **kwargs) # noqa: E501
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15812, in list_namespaced_pod_with_http_info
return self.api_client.call_api(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/rest.py", line 240, in GET
return self.request("GET", url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/rest.py", line 213, in request
r = self.pool_manager.request(method, url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/request.py", line 74, in request
return self.request_encode_url(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/request.py", line 96, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/poolmanager.py", line 376, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
retries = retries.increment(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='172.26.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c47808190>: Failed to establish a new connection: [Errno 110] Connection timed out'))
[�[34m2022-09-13 05:59:17,755�[0m] {�[34mprocess_utils.py:�[0m125} INFO�[0m - Sending Signals.SIGTERM to group 124. PIDs of all processes in the group: [124]�[0m
[�[34m2022-09-13 05:59:17,755�[0m] {�[34mprocess_utils.py:�[0m80} INFO�[0m - Sending the signal Signals.SIGTERM to group 124�[0m
[�[34m2022-09-13 05:59:18,393�[0m] {�[34mprocess_utils.py:�[0m75} INFO�[0m - Process psutil.Process(pid=124, status='terminated', exitcode=0, started='05:50:31') (124) terminated with exit code 0�[0m
[�[34m2022-09-13 05:59:18,393�[0m] {�[34mscheduler_job.py:�[0m780} INFO�[0m - Exited execute loop�[0m
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
conn.connect()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 358, in connect
self.sock = conn = self._new_conn()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f9c839d7430>: Failed to establish a new connection: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/bin/airflow", line 8, in
sys.exit(main())
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/main.py", line 38, in main
args.func(args)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 51, in command
return func(*args, **kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/cli.py", line 99, in wrapper
return f(*args, **kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/cli/commands/scheduler_command.py", line 75, in scheduler
_run_scheduler_job(args=args)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/cli/commands/scheduler_command.py", line 46, in _run_scheduler_job
job.run()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 244, in run
self._execute()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 751, in _execute
self._run_scheduler_loop()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 806, in _run_scheduler_loop
self.adopt_or_reset_orphaned_tasks()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/session.py", line 71, in wrapper
return func(*args, session=session, **kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1236, in adopt_or_reset_orphaned_tasks
for attempt in run_with_db_retries(logger=self.log):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/tenacity/init.py", line 382, in iter
do = self.iter(retry_state=retry_state)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/tenacity/init.py", line 349, in iter
return fut.result()
File "/opt/bitnami/python/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/opt/bitnami/python/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1281, in adopt_or_reset_orphaned_tasks
to_reset = self.executor.try_adopt_task_instances(tis_to_reset_or_adopt)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 713, in try_adopt_task_instances
self._adopt_completed_pods(kube_client)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 760, in _adopt_completed_pods
pod_list = kube_client.list_namespaced_pod(namespace=self.kube_config.kube_namespace, **kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15697, in list_namespaced_pod
return self.list_namespaced_pod_with_http_info(namespace, **kwargs) # noqa: E501
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15812, in list_namespaced_pod_with_http_info
return self.api_client.call_api(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/rest.py", line 240, in GET
return self.request("GET", url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/rest.py", line 213, in request
r = self.pool_manager.request(method, url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/request.py", line 74, in request
return self.request_encode_url(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/request.py", line 96, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/poolmanager.py", line 376, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
retries = retries.increment(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='172.26.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839d7430>: Failed to establish a new connection: [Errno 110] Connection timed out'))`

How to reproduce

i am Using ArgoCD to deploy the chart to OpenShift.

Operating System

OpenShift

Versions of Apache Airflow Providers

No response

Deployment

Other 3rd-party Helm chart

Deployment details

bitnami helm chart 13.1.3

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

I think you should either raise it to bitnami (this community has nothing to do with it - they should support it), or switch to the official helm chart which is managed by the community https://airflow.apache.org/docs/helm-chart/stable/index.html

Converting it to a discussion in case more information is needed.