DISCUSS Durable executor launch records + typed 409 for stale /run attempts (PR #70931, refs #69760)

[wolvery]

Hi all,

I'd like the list's input on which of two approaches to pursue for #69760
before either is hardened for review. I've prototyped both:

Minimal (KubernetesExecutor, equality check):
#69782
Durable redesign (all executors, RFC/Draft):
#70931
Issue: #69760

Problem

When a task instance transitions out of its original launch (scheduler
failover, pod adoption, stuck-queued reschedule, orphan reset, or a manual
clear), the executor originally assigned may still call the Execution API
/run for that task. Today the launch identity lives only on the mutable
TaskInstance row, so a late/stale launch surfaces as an opaque
404 not_found — indistinguishable from a genuinely unknown task. That
consumes a retry and/or blocks clean requeueing.

Two candidate designs

Both share the same goal — reject a stale/superseded launch at /run — but
differ in where the launch identity lives and how far the change reaches.

Option A — reuse external_executor_id, equality check (PR #69782)

• No schema change. Reuses the existing scheduler capability for executors
  that
  set pre_assigns_external_executor_id = True: at QUEUED the scheduler
  writes a
  fresh external_executor_id UUID to the task_instance row.
• Opts KubernetesExecutor into that path, threads the token through the K8s
  workload lifecycle (preserves it across queued events so the scheduler
  doesn't
  overwrite it with the job id, annotates the pod, revalidates the DB row
  just
  before pod creation).
• /run rejects a worker whose token no longer equals the current TI token.
• Scope: KubernetesExecutor. Identity lives on the mutable/deletable TI row;
  it's an equality check, not a lifecycle.

Option B — durable launch-record table, state machine (PR #70931,

Draft/RFC)

• New durable task_instance_launch table (with migration) storing an
  immutable
  token independent of the mutable/deleted TaskInstance row, with an
  explicit
  active -> consumed | superseded lifecycle and guarded transitions.
• Executor-agnostic (Celery / K8s / Edge). The launch record is written
  atomically at SCHEDULED -> QUEUED.
• Supersession is first-class (stuck-queued reschedule, failed adoption,
  orphan
  reset, clear/next-try); successful adoption preserves the token.
• /run returns a typed 409 stale_executor_launch for a known terminal
  token; unknown tokens still 404. Gated behind Execution API version
  v2026_06_30 (Cadwyn) so older Task SDK clients keep the legacy 404.
• Task SDK maps 409 to a new TaskInstanceSupersededError; a superseded
  worker logs once and exits 0 rather than failing.

Open questions for the list

1. Which direction fits Airflow best — the minimal equality check on the
   existing field (A), or the durable, executor-agnostic launch record (B)?
   Is the extra table + migration in B justified, or is A's "no schema
   change,
   reuse external_executor_id" preferable and B a later follow-up?
2. Is a typed 409 stale_executor_launch on /run the contract you'd
   want, and
   is gating it behind a new Execution API version the accepted
   compatibility
   story here? Or is A's simpler reject sufficient?
3. Should this stay KubernetesExecutor-scoped (A) or land executor-agnostic
   from the first cut (B)?
4. Does either warrant an AIP, or is a PR + this thread sufficient given
   it's an
   additive change (schema/table in B) plus a versioned API behaviour?

Both PRs have Testing sections (unit + a live KubernetesExecutor
kind-cluster
validation on #70931) and #70931 carries a significant newsfragment. I'll
consolidate onto whichever approach the list favours and close the other.
Development was AI-assisted (disclosed in the PRs).

Extra:
task_instance_history is per-attempt, while the
proposed record is per-launch, so I don’t believe they are equivalent.

A single task attempt can be launched more than once—for example, after a
stuck-queued reschedule. Each launch has a different token, but
task_instance_history stores only one row per attempt and may be deleted
with the TI.

The launch table stores each token and atomically marks it consumed or
superseded. DAG/task/run/try fields identify the attempt, but only the
token identifies which launch remains valid.

The distinction is:

• task_instance_history: audit snapshot of a completed or replaced attempt;
• task_instance_launch: durable identity and atomic state for each executor
  launch.

Just to be clear: saying the current 404 necessarily
consumes a retry was too broad. The main issue is that it conflates an
unknown task with a known obsolete launch; the typed 409 makes that
distinction explicit.

For context, KubernetesExecutor is spawning duplicate, stale task pods that
later fail at /run with 404. These pods may trigger unnecessary node
provisioning, consume cluster capacity, and delay legitimate tasks from
starting. The launch record lets Airflow identify the obsolete launch
explicitly, while scheduler-side token validation can prevent stale pods
from being created in the first place.

P.s.: I used fable 5 for developing this.
Thanks,
Guilherme