[Airflow 3 / Task SDK] Heartbeat 404 "Task Instance not found" after ~12h run; supervisor SERVER_TERMINATED (SIGKILL); retry sees 409 while TI still running

[brigittatitisan]

Under which category would you file this issue?

Task SDK

Apache Airflow version

airflow 3.0.6

What happened and how to reproduce it?

During a long-running task (wall time on the order of ~12 hours in our case), the task process was terminated by the supervisor after the execution API stopped accepting heartbeats:

• PUT /execution/task-instances/<ti_uuid>/heartbeat returned 404 with Task Instance not found / reason: not_found.
• Supervisor logged that the server indicated the task should not keep running, then Task killed! with process SIGKILL (exit_code=-9), final_state=SERVER_TERMINATED.
• Supervisor also logged duration ~ 43214 s (~12 h) for that workload session.

In the same log window we also saw PATCH .../task-instances/<ti_uuid>/run return 409 Conflict with a message along the lines of cannot start task instance — invalid state, previous_state=running (exact try_number in our export was 2; we treat that as illustrative, not the core claim—we care about 404 + forced termination after a long running execute, regardless of attempt index).

DAG/operator architecture: The workload is a PythonOperator subclass whose execute() submits work to Snowflake and then blocks inside the same task on an imperative wait() (implemented on a BaseSensorOperator subclass: time.sleep + polling). There is no separate sensor task, so one TI can stay running for many hours.

What I need help with:

1. Under what conditions does the Task Execution API return 404 on heartbeat for a TI that was recently updating heartbeats successfully?
2. Whether long running tasks on Celery + Airflow 3 are expected to hit this, and how it relates to timeouts, TI lifecycle, or DAG run / span updates (we also see scheduler INFO such as Found span_status 'should_end', while updating state for dag_run ... on the same run).
3. Recommended patterns for multi-hour waits (deferrable operators, separate sensor task, etc.) under the Task SDK.

Representative log excerpts:
PUT /execution/task-instances/<ti_uuid>/heartbeat HTTP/1.1" 404 Not Found Task Instance not found ti_id=<uuid> Server indicated the task shouldn't be running anymore ... reason='not_found' message='Task Instance not found' Task killed! Task finished [supervisor] duration=43214.24... exit_code=-9 final_state=SERVER_TERMINATED

What you think should happen instead?

Heartbeat should keep returning 2xx until the TI is terminal or explicitly cleared; worker should not get 404 while still the active runner.

Operating System

No response

Deployment

MWAA, Celery

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

No response

Official Helm Chart version

Not Applicable

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.