Airflow 3.1.7 - All tasks failing with Invalid auth token: Signature verification failed

[davidjfrickert]

Under which category would you file this issue?

Airflow Core

Apache Airflow version

3.1.7

What happened and how to reproduce it?

Not sure how to reproduce, but out of nowhere all DAGs started failing. We use KubernetesPodOperator exclusively.
Task pods were failing all with the same issue:

  2026-04-07T06:27:23.301527Z [info     ] setup plugin alembic.autogenerate.schemas
  [alembic.runtime.plugins] loc=plugins.py:37
  2026-04-07T06:27:23.301648Z [info     ] setup plugin alembic.autogenerate.tables
  [alembic.runtime.plugins] loc=plugins.py:37
  2026-04-07T06:27:23.301748Z [info     ] setup plugin alembic.autogenerate.types
  [alembic.runtime.plugins] loc=plugins.py:37
  2026-04-07T06:27:23.301813Z [info     ] setup plugin alembic.autogenerate.constraints
  [alembic.runtime.plugins] loc=plugins.py:37
  2026-04-07T06:27:23.301869Z [info     ] setup plugin alembic.autogenerate.defaults
  [alembic.runtime.plugins] loc=plugins.py:37
  2026-04-07T06:27:23.301923Z [info     ] setup plugin alembic.autogenerate.comments
  [alembic.runtime.plugins] loc=plugins.py:37
  {"timestamp":"2026-04-07T06:27:23.411801Z","level":"info","event":"Executing
  workload","workload":"ExecuteTask(token='<REDACTED_JWT>',
  ti=TaskInstance(id=UUID('<REDACTED_UUID>'), dag_version_id=UUID('<REDACTED_UUID>'),
  task_id='start_healthcheck', dag_id='<REDACTED_DAG_ID>',
  run_id='scheduled__2026-04-07T06:27:12.595187+00:00', try_number=1, map_index=-1,
  pool_slots=1, queue='default', priority_weight=4, executor_config=None,
  parent_context_carrier={}, context_carrier={}),
  dag_rel_path=PurePosixPath('local_dags/<REDACTED_DAG_ID>.py'),
  bundle_info=BundleInfo(name='dags-folder', version=None), log_path='dag_id=<REDACTED_DAG_ID>/r
  un_id=scheduled__2026-04-07T06:27:12.595187+00:00/task_id=start_healthcheck/attempt=1.log',
  type='ExecuteTask')","logger":"__main__","filename":"execute_workload.py","lineno":56}
  {"timestamp":"2026-04-07T06:27:23.412014Z","level":"info","event":"Connecting to
  server:","server":"http://<REDACTED_HOST>/execution/","logger":"__main__","filename":"execute_
  workload.py","lineno":64}
  {"timestamp":"2026-04-07T06:27:23.430813Z","level":"info","event":"Secrets backends loaded for
   worker","count":1,"backend_classes":["EnvironmentVariablesBackend"],"logger":"supervisor","fi
  lename":"supervisor.py","lineno":1975}
  {"timestamp":"2026-04-07T06:27:25.698677Z","level":"info","event":"Process
  exited","pid":14,"exit_code":-9,"signal_sent":"SIGKILL","logger":"supervisor","filename":"supe
  rvisor.py","lineno":710}
  Traceback (most recent call last):
    File "<frozen runpy>", line 198, in _run_module_as_main
    File "<frozen runpy>", line 88, in _run_code
    File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/execute_w
  orkload.py", line 125, in <module>
      main()
    File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/execute_w
  orkload.py", line 121, in main
      execute_workload(workload)
    File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/execute_w
  orkload.py", line 66, in execute_workload
      supervise(
    File
  "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py",
  line 1984, in supervise
      process = ActivitySubprocess.start(
                ^^^^^^^^^^^^^^^^^^^^^^^^^
    File
  "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py",
  line 955, in start
      proc._on_child_started(ti=what, dag_rel_path=dag_rel_path, bundle_info=bundle_info)
    File
  "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py",
  line 966, in _on_child_started
      ti_context = self.client.task_instances.start(ti.id, self.pid, start_date)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line
  215, in start
      resp = self.client.patch(f"task-instances/{id}/run", content=body.model_dump_json())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1218, in
  patch
      return self.request(
             ^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 338, in
  wrapped_f
      return copy(f, *args, **kw)
             ^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 477, in
  __call__
      do = self.iter(retry_state=retry_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 378, in
  iter
      result = action(retry_state)
               ^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 400, in
  <lambda>
      self._add_action_func(lambda rs: rs.outcome.result())
                                       ^^^^^^^^^^^^^^^^^^^
    File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449, in result
      return self.__get_result()
             ^^^^^^^^^^^^^^^^^^^
    File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
      raise self._exception
    File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 480, in
  __call__
      result = fn(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line
  887, in request
      return super().request(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 825, in
  request
      return self.send(request, auth=auth, follow_redirects=follow_redirects)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
      response = self._send_handling_auth(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in
  _send_handling_auth
      response = self._send_handling_redirects(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 999, in
  _send_handling_redirects
      raise exc
    File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 982, in
  _send_handling_redirects
      hook(response)
    File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line
  186, in raise_on_4xx_5xx_with_note
      return get_json_error(response) or response.raise_for_status()
             ^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line
  176, in get_json_error
      raise err
  airflow.sdk.api.client.ServerResponseError: Invalid auth token: Signature verification failed
  Correlation-id=<REDACTED_UUID>

Nothing really changed on the deployment. only thing i noticed was that one of the scheduler pods restarted recently.
API Server logs:

  2026-04-07T06:27:38.072406Z [info     ] Task instance state
  updated
  [airflow.api_fastapi.execution_api.routes.task_instances]
  loc=task_instances.py:424 new_state=failed rows_affected=1
  ti_id=<REDACTED_UUID>
  INFO:     <REDACTED_IP>:<PORT> - "PATCH
  /execution/task-instances/<REDACTED_UUID>/state HTTP/1.1" 204
   No Content
  INFO:     <REDACTED_IP>:<PORT> - "PUT
  /execution/task-instances/<REDACTED_UUID>/heartbeat HTTP/1.1"
   204 No Content
  INFO:     <REDACTED_IP>:<PORT> - "GET /api/v2/version
  HTTP/1.1" 200 OK
  INFO:     <REDACTED_IP>:<PORT> - "GET /api/v2/version
  HTTP/1.1" 200 OK
  INFO:     <REDACTED_IP>:<PORT> - "GET /admin/metrics/
  HTTP/1.1" 200 OK
  INFO:     <REDACTED_IP>:<PORT> - "GET /api/v2/version
  HTTP/1.1" 200 OK
  INFO:     <REDACTED_IP>:<PORT> - "GET /api/v2/version
  HTTP/1.1" 200 OK
  INFO:     <REDACTED_IP>:<PORT> - "PUT
  /execution/task-instances/<REDACTED_UUID>/heartbeat HTTP/1.1"
   204 No Content
  INFO:     <REDACTED_IP>:<PORT> - "PUT
  /execution/task-instances/<REDACTED_UUID>/heartbeat HTTP/1.1"
   204 No Content
  INFO:     <REDACTED_IP>:<PORT> - "GET /api/v2/version
  HTTP/1.1" 200 OK
  INFO:     <REDACTED_IP>:<PORT> - "GET /api/v2/version
  HTTP/1.1" 200 OK
  INFO:     <REDACTED_IP>:<PORT> - "GET /admin/metrics/
  HTTP/1.1" 200 OK
  INFO:     <REDACTED_IP>:<PORT> - "PUT
  /execution/task-instances/<REDACTED_UUID>/heartbeat HTTP/1.1"
   204 No Content
  INFO:     <REDACTED_IP>:<PORT> - "GET /api/v2/version
  HTTP/1.1" 200 OK
  INFO:     <REDACTED_IP>:<PORT> - "GET /api/v2/version
  HTTP/1.1" 200 OK
  INFO:     <REDACTED_IP>:<PORT> - "PUT
  /execution/task-instances/<REDACTED_UUID>/heartbeat HTTP/1.1"
   204 No Content
  INFO:     <REDACTED_IP>:<PORT> - "GET /admin/metrics/
  HTTP/1.1" 200 OK
  2026-04-07T06:28:21.716258Z [warning  ] Failed to validate
  JWT         [airflow.api_fastapi.execution_api.deps]
  loc=deps.py:86 token=<REDACTED_JWT>
  Traceback (most recent call last):
    File "/home/airflow/.local/lib/python3.12/site-packages/air
  flow/api_fastapi/execution_api/deps.py", line 83, in __call__
      claims = await
  validator.avalidated_claims(creds.credentials, validators)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/air
  flow/api_fastapi/auth/tokens.py", line 328, in
  avalidated_claims
      claims = jwt.decode(
               ^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/jwt
  /api_jwt.py", line 371, in decode
      decoded = self.decode_complete(
                ^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/jwt
  /api_jwt.py", line 268, in decode_complete
      decoded = self._jws.decode_complete(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/jwt
  /api_jws.py", line 244, in decode_complete
      self._verify_signature(signing_input, header, signature,
  key, algorithms)
    File "/home/airflow/.local/lib/python3.12/site-packages/jwt
  /api_jws.py", line 359, in _verify_signature
      raise InvalidSignatureError("Signature verification
  failed")
  jwt.exceptions.InvalidSignatureError: Signature verification
  failed
  2026-04-07T06:28:21.716787Z [warning  ] JWT reissue
  middleware failed to refresh token
  [airflow.api_fastapi.execution_api.app] error='Signature
  verification failed' loc=app.py:126
  Traceback (most recent call last):
    File "/home/airflow/.local/lib/python3.12/site-packages/air
  flow/api_fastapi/execution_api/app.py", line 115, in dispatch
      claims = await validator.avalidated_claims(token, {})
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/air
  flow/api_fastapi/auth/tokens.py", line 328, in
  avalidated_claims
      claims = jwt.decode(
               ^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/jwt
  /api_jwt.py", line 371, in decode
      decoded = self.decode_complete(
                ^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/jwt
  /api_jwt.py", line 268, in decode_complete
      decoded = self._jws.decode_complete(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/jwt
  /api_jws.py", line 244, in decode_complete
      self._verify_signature(signing_input, header, signature,
  key, algorithms)
    File "/home/airflow/.local/lib/python3.12/site-packages/jwt
  /api_jws.py", line 359, in _verify_signature
      raise InvalidSignatureError("Signature verification
  failed")
  jwt.exceptions.InvalidSignatureError: Signature verification
  failed
  INFO:     <REDACTED_IP>:<PORT> - "PATCH
  /execution/task-instances/<REDACTED_UUID>/run HTTP/1.1" 403
  Forbidden
  INFO:     <REDACTED_IP>:<PORT> - "GET /api/v2/version
  HTTP/1.1" 200 OK
  INFO:     <REDACTED_IP>:<PORT> - "GET /api/v2/version
  HTTP/1.1" 200 OK
  INFO:     <REDACTED_IP>:<PORT> - "GET /login/ HTTP/1.1" 200
  OK
  INFO:     <REDACTED_IP>:<PORT> - "PUT
  /execution/task-instances/<REDACTED_UUID>/heartbeat HTTP/1.1"
   204 No Content
  INFO:     <REDACTED_IP>:<PORT> - "GET /api/v2/version
  HTTP/1.1" 200 OK
  INFO:     <REDACTED_IP>:<PORT> - "GET /api/v2/version
  HTTP/1.1" 200 OK
  INFO:     <REDACTED_IP>:<PORT> - "PUT
  /execution/task-instances/<REDACTED_UUID>/heartbeat HTTP/1.1"
   204 No Content
  INFO:     <REDACTED_IP>:<PORT> - "GET /admin/metrics/
  HTTP/1.1" 200 OK
  INFO:     <REDACTED_IP>:<PORT> - "GET /api/v2/version
  HTTP/1.1" 200 OK
  INFO:     <REDACTED_IP>:<PORT> - "GET /api/v2/version
  HTTP/1.1" 200 OK
  2026-04-07T06:28:42.641659Z [warning  ] Failed to validate
  JWT         [airflow.api_fastapi.execution_api.deps]
  loc=deps.py:86 token=<REDACTED_JWT>
  Traceback (most recent call last):
    [... same JWT InvalidSignatureError traceback ...]
  jwt.exceptions.InvalidSignatureError: Signature verification
  failed
  2026-04-07T06:28:42.642204Z [warning  ] JWT reissue
  middleware failed to refresh token
  [airflow.api_fastapi.execution_api.app] error='Signature
  verification failed' loc=app.py:126
  Traceback (most recent call last):
    [... same JWT InvalidSignatureError traceback ...]
  jwt.exceptions.InvalidSignatureError: Signature verification
  failed
  INFO:     <REDACTED_IP>:<PORT> - "PATCH
  /execution/task-instances/<REDACTED_UUID>/run HTTP/1.1" 403
  Forbidden
  INFO:     <REDACTED_IP>:<PORT> - "GET /admin/metrics/
  HTTP/1.1" 200 OK
  2026-04-07T06:28:46.212078Z [warning  ] Failed to validate
  JWT         [airflow.api_fastapi.execution_api.deps]
  loc=deps.py:86 token=<REDACTED_JWT>
  Traceback (most recent call last):
    [... same JWT InvalidSignatureError traceback ...]
  jwt.exceptions.InvalidSignatureError: Signature verification
  failed
  2026-04-07T06:28:46.212700Z [warning  ] JWT reissue
  middleware failed to refresh token
  [airflow.api_fastapi.execution_api.app] error='Signature
  verification failed' loc=app.py:126
  Traceback (most recent call last):
    [... same JWT InvalidSignatureError traceback ...]
  jwt.exceptions.InvalidSignatureError: Signature verification
  failed
  INFO:     <REDACTED_IP>:<PORT> - "PATCH
  /execution/task-instances/<REDACTED_UUID>/run HTTP/1.1" 403
  Forbidden

I did a rollout restart of api-server, scheduler and dag-processor and this seems to have fixed it.

When I took JWT on pod logs and checked signature on jwt.io against the key in k8s secret airflow-jwt-secret it shows all OK, so I don't know why airflow suddenly failed to verify signatures...

What you think should happen instead?

No issues on running DAGs.

Operating System

Rocky Linux 8

Deployment

Official Apache Airflow Helm Chart

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

No response

Official Helm Chart version

1.18.0

Kubernetes Version

v1.30.14

Helm Chart configuration

No response

Docker Image customizations

Not applicable

Anything else?

It occurred out of nowhere.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[davidjfrickert]

Not 100% sure but likely related to jwtSecret being rotated. need to test if an helm upgrade rotates this secret, if so that may explain the issue. Also, seems like somewhere between chart 1.18.0 and 1.20.0 the jwtSecret checksum was added to api-server and scheduler, which would have prevented this issue.

[eladkal]

Also, seems like somewhere between chart 1.18.0 and 1.20.0

Specifically in 1.19.0 Add checksum for JWT secret in API server and scheduler deployments

@davidjfrickert I assume that upgrading to chart 1.19.0 solved your issue?

[laurenhall1526-netizen]

Good catch — that lines up with what I’m seeing. I haven’t tried upgrading yet. @davidjfrickert did moving to 1.19.0 fix it for you, or did you need anything else?

…

On Wed, Apr 8, 2026, 7:55 AM Elad Kalif ***@***.***> wrote: *eladkal* left a comment (apache/airflow#64826) <#64826 (comment)> Also, seems like somewhere between chart 1.18.0 and 1.20.0 Specifically in 1.19.0 Add checksum for JWT secret in API server and scheduler deployments <#60111> @davidjfrickert <https://github.com/davidjfrickert> I assume that upgrading to chart 1.19.0 solved your issue? — Reply to this email directly, view it on GitHub <#64826 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/B7IHA47ETW53KG55DEIV27T4UZD47AVCNFSM6AAAAACXPBYSNWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DEMBWGM3TKMRUGM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[davidjfrickert]

Also, seems like somewhere between chart 1.18.0 and 1.20.0

Specifically in 1.19.0 Add checksum for JWT secret in API server and scheduler deployments

@davidjfrickert I assume that upgrading to chart 1.19.0 solved your issue?

Not entirely, I upgraded to 1.20.0, but while upgrading the issue occured again.
I think because while on upgrade process the secret gets rotated but api-server and scheduler components have multiple replicas so they need a rollout. While rollout is ongoing DAGs may fail due to mismatch on scheduler / api-server versions.
I ended up extracting the secret from the helm chart so it doesn't rotate on upgrades.