The task should renew the JWT token.
scheduler-1 | 10:47:16 scheduler.1 | 2026-05-28T10:47:16.387387Z logLevel=INFO logger=airflow.models.taskinstance - Marking task as FAILED. dag_id=sleep_dag, task_id=sleep_180_minutes, run_id=manual__2026-05-28T08:58:44.271878+00:00, logical_date=20260528T085843, start_date=20260528T085845, end_date=20260528T104716
api-server-1 | 10:47:17 webserver.1 | 2026-05-28T10:47:17.112735Z logLevel=WARNING logger=airflow.api_fastapi.execution_api.security - Failed to validate JWT token=eyJ*** correlation_id=019e6e32-1772-766e-b111-bde2b336bd43
api-server-1 | 10:47:17 webserver.1 | Traceback (most recent call last):
api-server-1 | 10:47:17 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/execution_api/security.py", line 125, in __call__
api-server-1 | 10:47:17 webserver.1 | claims = await validator.avalidated_claims(creds.credentials, dict(self.required_claims))
api-server-1 | 10:47:17 webserver.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server-1 | 10:47:17 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/tokens.py", line 331, in avalidated_claims
scheduler-1 | 10:47:17 scheduler.1 | 2026-05-28T10:47:17.116922Z logLevel=WARNING logger=supervisor - Failed to send heartbeat. Will be retried failed_heartbeats=2 ti_id=UUID('019e6dce-b6bf-7d77-964e-5dd0f488cb53') max_retries=3
scheduler-1 | 10:47:17 scheduler.1 | Traceback (most recent call last):
scheduler-1 | 10:47:17 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1180, in _send_heartbeat_if_needed
scheduler-1 | 10:47:17 scheduler.1 | self.client.task_instances.heartbeat(self.id, pid=self._process.pid)
scheduler-1 | 10:47:17 scheduler.1 | self.client.put(f"task-instances/{id}/heartbeat", content=body.model_dump_json()) line 274, in heartbeat
scheduler-1 | 10:47:17 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1181, in put
scheduler-1 | 10:47:17 scheduler.1 | return self.request(
scheduler-1 | 10:47:17 scheduler.1 | ^^^^^^^^^^^^^
scheduler-1 | 10:47:17 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 331, in wrapped_f
scheduler-1 | 10:47:17 scheduler.1 | return copy(f, *args, **kw)
scheduler-1 | 10:47:17 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:17 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 470, in __call__
scheduler-1 | 10:47:17 scheduler.1 | do = self.iter(retry_state=retry_state)
scheduler-1 | 10:47:17 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:17 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 371, in iter
scheduler-1 | 10:47:17 scheduler.1 | result = action(retry_state)
scheduler-1 | 10:47:17 scheduler.1 | ^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:17 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 393, in <lambda>
scheduler-1 | 10:47:17 scheduler.1 | self._add_action_func(lambda rs: rs.outcome.result())
scheduler-1 | 10:47:17 scheduler.1 | ^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:17 scheduler.1 | File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449, in result
scheduler-1 | 10:47:17 scheduler.1 | return self.__get_result()
scheduler-1 | 10:47:17 scheduler.1 | ^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:17 scheduler.1 | File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
scheduler-1 | 10:47:17 scheduler.1 | raise self._exception
scheduler-1 | 10:47:17 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 473, in __call__
scheduler-1 | 10:47:17 scheduler.1 | result = fn(*args, **kwargs)
scheduler-1 | 10:47:17 scheduler.1 | ^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:17 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 985, in request
scheduler-1 | 10:47:17 scheduler.1 | return super().request(*args, **kwargs)
scheduler-1 | 10:47:17 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:17 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 825, in request
scheduler-1 | 10:47:17 scheduler.1 | return self.send(request, auth=auth, follow_redirects=follow_redirects)
scheduler-1 | 10:47:17 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:17 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
scheduler-1 | 10:47:17 scheduler.1 | response = self._send_handling_auth(
scheduler-1 | 10:47:17 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:17 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
scheduler-1 | 10:47:17 scheduler.1 | response = self._send_handling_redirects(
scheduler-1 | 10:47:17 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:17 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 999, in _send_handling_redirects
scheduler-1 | 10:47:17 scheduler.1 | raise exc
scheduler-1 | 10:47:17 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 982, in _send_handling_redirects
scheduler-1 | 10:47:17 scheduler.1 | hook(response)
scheduler-1 | 10:47:17 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 190, in raise_on_4xx_5xx_with_note
scheduler-1 | 10:47:17 scheduler.1 | return get_json_error(response) or response.raise_for_status()
scheduler-1 | 10:47:17 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:17 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 180, in get_json_error
scheduler-1 | 10:47:17 scheduler.1 | raise err
scheduler-1 | 10:47:17 scheduler.1 | airflow.sdk.api.client.ServerResponseError: Invalid auth token: Signature has expired
scheduler-1 | 10:47:17 scheduler.1 | Correlation-id=019e6e32-1772-766e-b111-bde2b336bd43
api-server-1 | 10:47:17 webserver.1 | claims = jwt.decode(
api-server-1 | 10:47:17 webserver.1 | ^^^^^^^^^^^
api-server-1 | 10:47:17 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 365, in decode
api-server-1 | 10:47:17 webserver.1 | decoded = self.decode_complete(
api-server-1 | 10:47:17 webserver.1 | ^^^^^^^^^^^^^^^^^^^^^
api-server-1 | 10:47:17 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 272, in decode_complete
api-server-1 | 10:47:17 webserver.1 | self._validate_claims(
api-server-1 | 10:47:17 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 405, in _validate_claims
api-server-1 | 10:47:17 webserver.1 | self._validate_exp(payload, now, leeway)
api-server-1 | 10:47:17 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 508, in _validate_exp
api-server-1 | 10:47:17 webserver.1 | raise ExpiredSignatureError("Signature has expired")
api-server-1 | 10:47:17 webserver.1 | jwt.exceptions.ExpiredSignatureError: Signature has expired
api-server-1 | 10:47:17 webserver.1 | 2026-05-28T10:47:17.114135Z logLevel=WARNING logger=airflow.api_fastapi.execution_api.app - JWT reissue middleware failed to refresh token error='Signature has expired'
api-server-1 | 10:47:17 webserver.1 | Traceback (most recent call last):
api-server-1 | 10:47:17 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/execution_api/app.py", line 143, in dispatch
api-server-1 | 10:47:17 webserver.1 | claims = await validator.avalidated_claims(token, {})
api-server-1 | 10:47:17 webserver.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server-1 | 10:47:17 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/tokens.py", line 331, in avalidated_claims
api-server-1 | 10:47:17 webserver.1 | claims = jwt.decode(
api-server-1 | 10:47:17 webserver.1 | ^^^^^^^^^^^
api-server-1 | 10:47:17 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 365, in decode
api-server-1 | 10:47:17 webserver.1 | decoded = self.decode_complete(
api-server-1 | 10:47:17 webserver.1 | ^^^^^^^^^^^^^^^^^^^^^
api-server-1 | 10:47:17 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 272, in decode_complete
api-server-1 | 10:47:17 webserver.1 | self._validate_claims(
api-server-1 | 10:47:17 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 405, in _validate_claims
api-server-1 | 10:47:17 webserver.1 | self._validate_exp(payload, now, leeway)
api-server-1 | 10:47:17 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 508, in _validate_exp
api-server-1 | 10:47:17 webserver.1 | raise ExpiredSignatureError("Signature has expired")
api-server-1 | 10:47:17 webserver.1 | jwt.exceptions.ExpiredSignatureError: Signature has expired
api-server-1 | 10:47:17 webserver.1 | 2026-05-28T10:47:17.115568Z logLevel=INFO logger=http.access - request finished method=PUT path=/task-exec/execution/task-instances/019e6dce-b6bf-7d77-964e-5dd0f488cb53/heartbeat query= status_code=403 duration_us=6205 client_addr=172.19.0.8:56688
scheduler-1 | 10:47:18 scheduler.1 | 2026-05-28T10:47:18.415876Z logLevel=INFO logger=airflow.models.dagrun.DagRun - Marking run <DagRun sleep_dag @ 2026-05-28 08:58:43+00:00: manual__2026-05-28T08:58:44.271878+00:00, state:running, queued_at: 2026-05-28 08:58:44.278783+00:00. run_type: manual> failed
scheduler-1 | 10:47:18 scheduler.1 | 2026-05-28T10:47:18.416776Z logLevel=INFO logger=airflow.models.dagrun.DagRun - DagRun Finished: dag_id=sleep_dag, logical_date=2026-05-28 08:58:43+00:00, run_id=manual__2026-05-28T08:58:44.271878+00:00, run_start_date=2026-05-28 08:58:45.319171+00:00, run_end_date=2026-05-28 10:47:18.415970+00:00, run_duration=6513.096799, state=failed, run_type=manual, data_interval_start=2026-05-28 08:58:43+00:00, data_interval_end=2026-05-28 08:58:43+00:00,
scheduler-1 | 10:47:18 dag_processor.1 | 2026-05-28T10:47:18.456733Z logLevel=INFO logger=airflow.dag_processing.bundles.manager.DagBundlesManager - DAG bundles loaded: dags-folder
scheduler-1 | 10:47:18 dag_processor.1 | 2026-05-28T10:47:18.543408Z logLevel=INFO logger=airflow.serialization.definitions.dag - Bulk-writing dags to db count=1
scheduler-1 | 10:47:18 dag_processor.1 | 2026-05-28T10:47:18.545103Z logLevel=INFO logger=airflow.dag_processing.collection - Getting latest run for non-partitioned Dag dag_id=sleep_dag
scheduler-1 | 10:47:18 dag_processor.1 | 2026-05-28T10:47:18.546317Z logLevel=INFO logger=airflow.dag_processing.collection - got latest run dag_id=sleep_dag logical_date='2021-01-01 00:00:00+00:00' partition_key=None
scheduler-1 | 10:47:18 dag_processor.1 | 2026-05-28T10:47:18.547857Z logLevel=INFO logger=airflow.serialization.definitions.dag - get next_dagrun_info_v2 last_automated_run_info=DagRunInfo(run_after=DateTime(2021, 1, 1, 0, 0, 0, tzinfo=Timezone('UTC')), data_interval=DataInterval(start=DateTime(2021, 1, 1, 0, 0, 0, tzinfo=Timezone('UTC')), end=DateTime(2021, 1, 1, 0, 0, 0, tzinfo=Timezone('UTC'))), partition_date=None, partition_key=None) next_info=None
scheduler-1 | 10:47:18 dag_processor.1 | 2026-05-28T10:47:18.547937Z logLevel=INFO logger=airflow.models.dag - setting next dagrun info next_dagrun=None next_dagrun_create_after=None next_dagrun_data_interval_start=None next_dagrun_data_interval_end=None next_dagrun_partition_key=None next_dagrun_partition_date=None
scheduler-1 | 10:47:20 dag_processor.1 | 2026-05-28T10:47:20.582901Z logLevel=INFO logger=airflow.dag_processing.manager.DagFileProcessorManager - Not time to refresh bundle dags-folder
api-server-1 | 10:47:22 webserver.1 | 2026-05-28T10:47:22.131563Z logLevel=WARNING logger=airflow.api_fastapi.execution_api.security - Failed to validate JWT token=eyJ*** correlation_id=019e6e32-2b10-7725-b598-40c99ab7a3d8
api-server-1 | 10:47:22 webserver.1 | Traceback (most recent call last):
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/execution_api/security.py", line 125, in __call__
api-server-1 | 10:47:22 webserver.1 | claims = await validator.avalidated_claims(creds.credentials, dict(self.required_claims))
api-server-1 | 10:47:22 webserver.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/tokens.py", line 331, in avalidated_claims
scheduler-1 | 10:47:22 scheduler.1 | 2026-05-28T10:47:22.134338Z logLevel=WARNING logger=supervisor - Failed to send heartbeat. Will be retried failed_heartbeats=3 ti_id=UUID('019e6dce-b6bf-7d77-964e-5dd0f488cb53') max_retries=3
scheduler-1 | 10:47:22 scheduler.1 | Traceback (most recent call last):
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1180, in _send_heartbeat_if_needed
scheduler-1 | 10:47:22 scheduler.1 | self.client.task_instances.heartbeat(self.id, pid=self._process.pid)
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 274, in heartbeat
scheduler-1 | 10:47:22 scheduler.1 | self.client.put(f"task-instances/{id}/heartbeat", content=body.model_dump_json())
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1181, in put
scheduler-1 | 10:47:22 scheduler.1 | return self.request(
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 331, in wrapped_f
scheduler-1 | 10:47:22 scheduler.1 | return copy(f, *args, **kw)
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 470, in __call__
scheduler-1 | 10:47:22 scheduler.1 | do = self.iter(retry_state=retry_state)
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 371, in iter
scheduler-1 | 10:47:22 scheduler.1 | result = action(retry_state)
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 393, in <lambda>
scheduler-1 | 10:47:22 scheduler.1 | self._add_action_func(lambda rs: rs.outcome.result())
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449, in result
scheduler-1 | 10:47:22 scheduler.1 | return self.__get_result()
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
scheduler-1 | 10:47:22 scheduler.1 | raise self._exception
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 473, in __call__
scheduler-1 | 10:47:22 scheduler.1 | result = fn(*args, **kwargs)
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 985, in request
scheduler-1 | 10:47:22 scheduler.1 | return super().request(*args, **kwargs)
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 825, in request
scheduler-1 | 10:47:22 scheduler.1 | return self.send(request, auth=auth, follow_redirects=follow_redirects)
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
scheduler-1 | 10:47:22 scheduler.1 | response = self._send_handling_auth(
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
scheduler-1 | 10:47:22 scheduler.1 | response = self._send_handling_redirects(
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 999, in _send_handling_redirects
scheduler-1 | 10:47:22 scheduler.1 | raise exc
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 982, in _send_handling_redirects
scheduler-1 | 10:47:22 scheduler.1 | hook(response)
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 190, in raise_on_4xx_5xx_with_note
scheduler-1 | 10:47:22 scheduler.1 | return get_json_error(response) or response.raise_for_status()
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 180, in get_json_error
scheduler-1 | 10:47:22 scheduler.1 | raise err
scheduler-1 | 10:47:22 scheduler.1 | airflow.sdk.api.client.ServerResponseError: Invalid auth token: Signature has expired
scheduler-1 | 10:47:22 scheduler.1 | Correlation-id=019e6e32-2b10-7725-b598-40c99ab7a3d8
scheduler-1 | 10:47:22 scheduler.1 | 2026-05-28T10:47:22.135267Z logLevel=ERROR logger=supervisor - Too many failed heartbeats; terminating process failed_heartbeats=3
scheduler-1 | 10:47:22 scheduler.1 | 2026-05-28 10:47:22.136255+00:00 logLevel=INFO logger=airflow.task.hooks.airflow.providers.standard.hooks.subprocess.SubprocessHook - Sending SIGTERM signal to process group
scheduler-1 | 10:47:22 scheduler.1 | 2026-05-28 10:47:22.137296+00:00 logLevel=INFO logger=airflow.task.hooks.airflow.providers.standard.hooks.subprocess.SubprocessHook - Command exited with return code -15
scheduler-1 | 10:47:22 scheduler.1 | 2026-05-28 10:47:22.138419+00:00 logLevel=ERROR logger=task - Task failed with exception error_detail=[{'exc_type': 'AirflowException', 'exc_value': 'Bash command failed. The command returned a non-zero exit code -15.', 'exc_notes': [], 'syntax_error': None, 'is_cause': False, 'frames': [{'filename': '/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py', 'lineno': 1263, 'name': 'run'}, {'filename': '/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py', 'lineno': 1678, 'name': '_execute_task'}, {'filename': '/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py', 'lineno': 443, 'name': 'wrapper'}, {'filename': '/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/bash.py', 'lineno': 226, 'name': 'execute'}], 'is_group': False, 'exceptions': []}]
scheduler-1 | 10:47:22 scheduler.1 | 2026-05-28T10:47:22.145081Z logLevel=INFO logger=supervisor - Process exited pid=1173 exit_code=0 signal_sent=SIGTERM
api-server-1 | 10:47:22 webserver.1 | claims = jwt.decode(
api-server-1 | 10:47:22 webserver.1 | ^^^^^^^^^^^
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 365, in decode
api-server-1 | 10:47:22 webserver.1 | decoded = self.decode_complete(
scheduler-1 | 10:47:22 scheduler.1 | 2026-05-28T10:47:22.149939Z logLevel=ERROR logger=airflow.executors.local_executor.LocalExecutor - Task execution failed.
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/local_executor.py", line 105, in _run_worker
scheduler-1 | 10:47:22 scheduler.1 | _execute_work(log, workload, team_conf)
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/local_executor.py", line 144, in _execute_work
scheduler-1 | 10:47:22 scheduler.1 | supervise(
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 2107, in supervise
scheduler-1 | 10:47:22 scheduler.1 | exit_code = process.wait()
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1070, in wait
scheduler-1 | 10:47:22 scheduler.1 | self.update_task_state_if_needed()
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1084, in update_task_state_if_needed
scheduler-1 | 10:47:22 scheduler.1 | self.client.task_instances.finish(
api-server-1 | 10:47:22 webserver.1 | ^^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 241, in finish
scheduler-1 | 10:47:22 scheduler.1 | self.client.patch(f"task-instances/{id}/state", content=body.model_dump_json())
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1218, in patch
scheduler-1 | 10:47:22 scheduler.1 | return self.request(
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 331, in wrapped_f
scheduler-1 | 10:47:22 scheduler.1 | return copy(f, *args, **kw)
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 470, in __call__
scheduler-1 | 10:47:22 scheduler.1 | do = self.iter(retry_state=retry_state)
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 371, in iter
scheduler-1 | 10:47:22 scheduler.1 | result = action(retry_state)
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 393, in <lambda>
scheduler-1 | 10:47:22 scheduler.1 | self._add_action_func(lambda rs: rs.outcome.result())
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449, in result
scheduler-1 | 10:47:22 scheduler.1 | return self.__get_result()
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
scheduler-1 | 10:47:22 scheduler.1 | raise self._exception
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 473, in __call__
scheduler-1 | 10:47:22 scheduler.1 | result = fn(*args, **kwargs)
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 985, in request
scheduler-1 | 10:47:22 scheduler.1 | return super().request(*args, **kwargs)
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 825, in request
scheduler-1 | 10:47:22 scheduler.1 | return self.send(request, auth=auth, follow_redirects=follow_redirects)
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
scheduler-1 | 10:47:22 scheduler.1 | response = self._send_handling_auth(
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
scheduler-1 | 10:47:22 scheduler.1 | response = self._send_handling_redirects(
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 999, in _send_handling_redirects
scheduler-1 | 10:47:22 scheduler.1 | raise exc
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 982, in _send_handling_redirects
scheduler-1 | 10:47:22 scheduler.1 | hook(response)
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 190, in raise_on_4xx_5xx_with_note
scheduler-1 | 10:47:22 scheduler.1 | return get_json_error(response) or response.raise_for_status()
scheduler-1 | 10:47:22 scheduler.1 | ^^^^^^^^^^^^^^^^^^^^^^^^
scheduler-1 | 10:47:22 scheduler.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 180, in get_json_error
scheduler-1 | 10:47:22 scheduler.1 | raise err
scheduler-1 | 10:47:22 scheduler.1 | airflow.sdk.api.client.ServerResponseError: Invalid auth token: Signature has expired
scheduler-1 | 10:47:22 scheduler.1 | Correlation-id=019e6e32-2b21-717f-9876-c87b021bcc10
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 272, in decode_complete
api-server-1 | 10:47:22 webserver.1 | self._validate_claims(
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 405, in _validate_claims
api-server-1 | 10:47:22 webserver.1 | self._validate_exp(payload, now, leeway)
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 508, in _validate_exp
api-server-1 | 10:47:22 webserver.1 | raise ExpiredSignatureError("Signature has expired")
api-server-1 | 10:47:22 webserver.1 | jwt.exceptions.ExpiredSignatureError: Signature has expired
api-server-1 | 10:47:22 webserver.1 | 2026-05-28T10:47:22.132633Z logLevel=WARNING logger=airflow.api_fastapi.execution_api.app - JWT reissue middleware failed to refresh token error='Signature has expired'
api-server-1 | 10:47:22 webserver.1 | Traceback (most recent call last):
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/execution_api/app.py", line 143, in dispatch
api-server-1 | 10:47:22 webserver.1 | claims = await validator.avalidated_claims(token, {})
api-server-1 | 10:47:22 webserver.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/tokens.py", line 331, in avalidated_claims
api-server-1 | 10:47:22 webserver.1 | claims = jwt.decode(
api-server-1 | 10:47:22 webserver.1 | ^^^^^^^^^^^
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 365, in decode
api-server-1 | 10:47:22 webserver.1 | decoded = self.decode_complete(
api-server-1 | 10:47:22 webserver.1 | ^^^^^^^^^^^^^^^^^^^^^
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 272, in decode_complete
api-server-1 | 10:47:22 webserver.1 | self._validate_claims(
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 405, in _validate_claims
api-server-1 | 10:47:22 webserver.1 | self._validate_exp(payload, now, leeway)
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 508, in _validate_exp
api-server-1 | 10:47:22 webserver.1 | raise ExpiredSignatureError("Signature has expired")
api-server-1 | 10:47:22 webserver.1 | jwt.exceptions.ExpiredSignatureError: Signature has expired
api-server-1 | 10:47:22 webserver.1 | 2026-05-28T10:47:22.133532Z logLevel=INFO logger=http.access - request finished method=PUT path=/task-exec/execution/task-instances/019e6dce-b6bf-7d77-964e-5dd0f488cb53/heartbeat query= status_code=403 duration_us=3772 client_addr=172.19.0.8:56734
api-server-1 | 10:47:22 webserver.1 | 2026-05-28T10:47:22.147535Z logLevel=WARNING logger=airflow.api_fastapi.execution_api.security - Failed to validate JWT token=eyJ*** correlation_id=019e6e32-2b21-717f-9876-c87b021bcc10
api-server-1 | 10:47:22 webserver.1 | Traceback (most recent call last):
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/execution_api/security.py", line 125, in __call__
api-server-1 | 10:47:22 webserver.1 | claims = await validator.avalidated_claims(creds.credentials, dict(self.required_claims))
api-server-1 | 10:47:22 webserver.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/tokens.py", line 331, in avalidated_claims
api-server-1 | 10:47:22 webserver.1 | claims = jwt.decode(
api-server-1 | 10:47:22 webserver.1 | ^^^^^^^^^^^
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 365, in decode
api-server-1 | 10:47:22 webserver.1 | decoded = self.decode_complete(
api-server-1 | 10:47:22 webserver.1 | ^^^^^^^^^^^^^^^^^^^^^
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 272, in decode_complete
api-server-1 | 10:47:22 webserver.1 | self._validate_claims(
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 405, in _validate_claims
api-server-1 | 10:47:22 webserver.1 | self._validate_exp(payload, now, leeway)
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 508, in _validate_exp
api-server-1 | 10:47:22 webserver.1 | raise ExpiredSignatureError("Signature has expired")
api-server-1 | 10:47:22 webserver.1 | jwt.exceptions.ExpiredSignatureError: Signature has expired
api-server-1 | 10:47:22 webserver.1 | 2026-05-28T10:47:22.148310Z logLevel=WARNING logger=airflow.api_fastapi.execution_api.app - JWT reissue middleware failed to refresh token error='Signature has expired'
api-server-1 | 10:47:22 webserver.1 | Traceback (most recent call last):
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/execution_api/app.py", line 143, in dispatch
api-server-1 | 10:47:22 webserver.1 | claims = await validator.avalidated_claims(token, {})
api-server-1 | 10:47:22 webserver.1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/tokens.py", line 331, in avalidated_claims
api-server-1 | 10:47:22 webserver.1 | claims = jwt.decode(
api-server-1 | 10:47:22 webserver.1 | ^^^^^^^^^^^
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 365, in decode
api-server-1 | 10:47:22 webserver.1 | decoded = self.decode_complete(
api-server-1 | 10:47:22 webserver.1 | ^^^^^^^^^^^^^^^^^^^^^
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 272, in decode_complete
api-server-1 | 10:47:22 webserver.1 | self._validate_claims(
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 405, in _validate_claims
api-server-1 | 10:47:22 webserver.1 | self._validate_exp(payload, now, leeway)
api-server-1 | 10:47:22 webserver.1 | File "/home/airflow/.local/lib/python3.12/site-packages/jwt/api_jwt.py", line 508, in _validate_exp
api-server-1 | 10:47:22 webserver.1 | raise ExpiredSignatureError("Signature has expired")
api-server-1 | 10:47:22 webserver.1 | jwt.exceptions.ExpiredSignatureError: Signature has expired
api-server-1 | 10:47:22 webserver.1 | 2026-05-28T10:47:22.149087Z logLevel=INFO logger=http.access - request finished method=PATCH path=/task-exec/execution/task-instances/019e6dce-b6bf-7d77-964e-5dd0f488cb53/state query= status_code=403 duration_us=2955 client_addr=172.19.0.8:56734
scheduler-1 | 10:47:22 scheduler.1 | 2026-05-28T10:47:22.488752Z logLevel=INFO logger=airflow.jobs.scheduler_job_runner.SchedulerJobRunner - Received executor event with state failed for task instance TaskInstanceKey(dag_id='sleep_dag', task_id='sleep_180_minutes', run_id='manual__2026-05-28T08:58:44.271878+00:00', try_number=1, map_index=-1)
scheduler-1 | 10:47:22 scheduler.1 | 2026-05-28T10:47:22.493014Z logLevel=INFO logger=airflow.jobs.scheduler_job_runner.SchedulerJobRunner - TaskInstance Finished: dag_id=sleep_dag, task_id=sleep_180_minutes, run_id=manual__2026-05-28T08:58:44.271878+00:00, map_index=-1, run_start_date=2026-05-28 08:58:45.344802+00:00, run_end_date=2026-05-28 10:47:16.386918+00:00, run_duration=6511.042116, state=failed, executor=LocalExecutor(parallelism=2), executor_state=failed, try_number=1, max_tries=0, pool=default_pool, queue=default, priority_weight=1, operator=BashOperator, queued_dttm=2026-05-28 08:58:45.336438+00:00, scheduled_dttm=2026-05-28 08:58:45.326033+00:00,queued_by_job_id=86, pid=1173
api-server-1 | 10:47:24 webserver.1 | 2026-05-28T10:47:24.437988Z logLevel=INFO logger=http.access - request finished method=GET path=/task-exec/ui/grid/ti_summaries/sleep_dag query='run_ids=manual__2026-05-28T08%3A58%3A44.271878%2B00%3A00&run_ids=manual__2026-05-28T06%3A37%3A15.085544%2B00%3A00&run_ids=manual__2026-05-28T04%3A25%3A43.899944%2B00%3A00&run_ids=manual__2026-05-27T10%3A19%3A57.329793%2B00%3A00&run_ids=manual__2026-05-27T09%3A19%3A16.101098%2B00%3A00&run_ids=manual__2026-05-27T07%3A58%3A58.064377%2B00%3A00&run_ids=scheduled__2021-01-01T00%3A00%3A00%2B00%3A00' status_code=200 duration_us=23129 client_addr=172.19.0.4:54350
scheduler-1 | 10:4
Under which category would you file this issue?
Airflow Core
Apache Airflow version
3.2.2
What happened and how to reproduce it?
** Issue Description **
With the LocalExecutor having a DAG executing an task that runs e.g for 3 hours or more.
In about 100 minutes, the task is killed for missing / expired heartbeat.
The task itself fails to send the heartbeat because of expired JWT token.
Seems that the task does not refresh the JWT token.
If the EXECUTION_API__JWT_EXPIRATION_TIME is set to much longer value from the default 600s, the task will continue running.
** Steps to Reproduce: **
With similar setup as described in:
https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#concepts-task-instance-heartbeat-timeout
The default configuration:
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_SEC=0
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT=300
AIRFLOW__EXECUTION_API__JWT_EXPIRATION_TIME=600
Run the DAG:
dag_TEST_TIMEOUT.py:
What you think should happen instead?
The task should renew the JWT token.
Operating System
debian
Deployment
Other Docker-based deployment
Apache Airflow Provider(s)
No response
Versions of Apache Airflow Providers
apache-airflow-providers-standard==1.13.1
apache-airflow-providers-snowflake==6.13.0
apache-airflow-providers-amazon==9.29.0
apache-airflow-providers-google==21.3.0
apache-airflow-providers-microsoft-azure==13.3.0
apache-airflow-providers-odbc==4.12.2
apache-airflow-providers-postgres==6.7.0
apache-airflow-providers-databricks==7.15.0
apache-airflow-task-sdk==1.2.2
apache-airflow-providers-fab==3.6.4
apache-airflow-core[gunicorn]
Official Helm Chart version
Not Applicable
Kubernetes Version
No response
Helm Chart configuration
No response
Docker Image customizations
No response
Anything else?
Error log:
Are you willing to submit PR?
Code of Conduct