API server 500s on PATCH ti/{id}/run for tasks directly downstream of a mapped task group whose expansion never occurred — task can never launch

[irfansiddiqui]

Apache Airflow version

3.0.6

What happened?

Tasks that are directly downstream of a mapped @task_group whose expansion never occurred (because the task producing the expansion input finished as failed/upstream_failed) can never launch. The API server raises an unhandled exception (HTTP 500, ERROR: Exception in ASGI application) on the task runner's startup call:

PATCH /execution/task-instances/{ti_id}/run

The worker-side SDK client (airflow.sdk.api.client) logs Server error {'message': 'Internal server error', ...}, retries via tenacity, and ultimately raises ReadTimeout / ServerResponseError inside Celery's execute_workload. The executor then reports the task as failed. The task instance ends up failed with:

• hostname empty
• start_date/duration null (run_start_date=None in the scheduler's "TaskInstance Finished" line)
• no task log stream ever created

Because the failure is reported while the TI is still queued, retry behavior is fragile. We additionally observed that when the DagRun's pinned serialized DAG version had been replaced by a redeploy, the scheduler's executor-event handling logged:

{scheduler_job_runner.py:952} ERROR - DAG '<dag_id>' for task instance
<TaskInstance: ... [queued]> not found in serialized_dag table
{taskinstance.py:2011} INFO - Marking task as FAILED. ...

and force-failed the TI bypassing its retry policy entirely (retries=2 was configured; the TI went straight to failed at try_number=1).

Behavioral fingerprint that isolated the cause (observed across ~6 runs on the same environment, minutes apart):

• every task directly downstream of the unexpanded mapped group fails to launch this way (in our DAG: a ONE_FAILED failure-callback leaf and two ALL_DONE cleanup tasks);
• every task not directly downstream of the mapped group launches normally in the same runs (including a task one extra hop downstream of the affected cleanup tasks);
• the same downstream tasks launch normally in runs where the expansion did occur;
• rewiring the failure-callback task so it no longer had the mapped group as a direct upstream made it launch and execute immediately, same environment, same failure scenario.

A side effect worth noting: the repeated doomed launch attempts also produced celery.concurrency.asynpool Timed out waiting for UP message from <ForkProcess...> noise, which initially misdirected us toward worker health.

What you think should happen instead?

The /run endpoint should be able to build the startup context for a TI whose direct upstream is a mapped task group with zero expanded instances (e.g. resolve upstream map indexes to an empty set), so the task can launch and let its trigger rule/callable decide what to do. Failing that, the scheduler should at least respect the TI's retry policy when handling executor-reported launch failures, and surface the server-side exception in a way that points at the affected task rather than a generic ASGI 500.

How to reproduce

Sketch (matches our production DAG's shape):

from airflow.decorators import dag, task, task_group
from airflow.utils.trigger_rule import TriggerRule

@dag(schedule=None)
def repro():
    @task
    def make_items() -> list:
        raise ValueError("expansion input fails")  # expansion never happens

    @task_group
    def per_item(item: dict):
        @task
        def work(item: dict): ...
        work(item)

    items = make_items()
    mapped = per_item.expand(item=items)

    # Any task scheduled despite the failure and directly downstream of
    # the mapped group exhibits the launch failure:
    @task(trigger_rule=TriggerRule.ONE_FAILED, retries=2)
    def on_failure(**context):
        ...

    flag = on_failure()
    mapped >> flag          # <-- direct mapped-group upstream: never launches
    items >> flag

repro()

Trigger the DAG; make_items fails; on_failure is scheduled (ONE_FAILED satisfied via upstream_failed) and queued, then fails with no hostname/logs; API-server log shows the ASGI exception on PATCH /execution/task-instances/{id}/run. Remove the mapped >> flag edge and the task launches and executes normally in the same scenario.

Operating System

Amazon Linux (AWS MWAA)

Versions of Apache Airflow Providers

As bundled with MWAA Airflow 3.0.6 (amazon provider; CeleryExecutor, Postgres metadata DB).

Deployment

Amazon (AWS) MWAA

Deployment details

MWAA environment class mw1.small, 2 workers, 2 schedulers. Observed 2026-08-02/03; consistent across many runs and two differently-wired DAGs (identical structure).

Anything else?

Occurs every time the expansion input fails; never when expansion succeeds. Happy to provide fuller (redacted) scheduler/worker/webserver log excerpts.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[irfansiddiqui]

Closing this issue, this is dumb claude workflow that opened this.