Scheduler deadlocks in pipe write during task dispatch (LocalExecutor) — runtime variant of #67881

[darren-cheung-sa]

Under which category would you file this issue?

Airflow Core

Apache Airflow version

3.3.0

What happened and how to reproduce it?

Issue Description

The scheduler permanently deadlocks during normal task dispatch (not shutdown), blocked in a pipe write() to the LocalExecutor's activity queue. All dispatch halts; the SchedulerJob heartbeat stops while the systemd service stays "active". ~10 minutes later, every running task's supervisor kills its task ("Server indicated the task shouldn't be running anymore"), burning retry attempts.

This appears to be the runtime/dispatch-path variant of the shutdown deadlock fixed in #67881 (merged 2026-07-02, not yet released — 3.3.0 still ships the pre-fix end() and empty terminate()).

Occurred 4 times in ~14 hours on one production host once dispatch volume rose (~150–250 concurrent tasks). Identical signature each time.

Evidence captured mid-deadlock:

• /proc/<scheduler-pid>/stack → blocked in pipe_write
• /proc/<scheduler-pid>/syscall → write(fd=9); fd 9 → pipe:[<inode>]
• Scanning /proc/*/fd: the same pipe inode held (read fd 7 + write fd 9) by the scheduler and every LocalExecutor worker process
• Host healthy: 100+ GB free RAM, idle CPU, no OOM, clean kernel log
• One occurrence happened with zero api-server errors/timeouts, ruling out API-side causes

Steps to reproduce:

Race condition — probabilistic and load-correlated, so not minimally deterministic, but reliably triggered in our environment within minutes-to-hours:

1. LocalExecutor, parallelism = 256, default max_tis_per_query (512), PostgreSQL backend
2. Generate dispatch bursts of 150–250 mostly lightweight, long-running tasks (ours poll remote Spark jobs), with tasks completing (returning results) concurrently with new dispatch
3. Scheduler eventually blocks forever in pipe_write; heartbeat stops

Notably there is no clean load threshold: the same host survived a 30-minute soak at 256 running tasks, then deadlocked at ~40 running during a later dispatch wave — consistent with a timing race on the shared queues, not resource exhaustion.

Suggested synthetic repro: a DAG with ~300 mapped time.sleep(300) tasks, cleared repeatedly to force dispatch bursts while earlier tasks are completing.

Mitigations tried (none sufficient):

Change	Result
[api] workers 4 → 12	fixed unrelated api saturation; deadlock recurred with 0 timeouts
default_pool 300 → 150	reduced frequency; recurred
[scheduler] max_tis_per_query 512 → 32	reduced frequency; recurred
Pausing the high-volume DAGs	stable (current workaround, at large throughput cost)

What you think should happen instead?

The scheduler should never block indefinitely on executor queue writes — a full result/activity queue should trigger draining (or a bounded/non-blocking put with retry), so a slow consumer can't freeze the entire scheduling loop.

Suspected mechanism — same as #67881 describes, but hit from the dispatch side:

1. Workers fill result_queue's OS pipe buffer (~64KB) and block on put()
2. Blocked workers stop draining activity_queue
3. Scheduler blocks in _process_workloads() → self.activity_queue.put(workload) (airflow/executors/local_executor.py)
4. Being single-threaded at that point, the scheduler never reaches _read_results(), so result_queue never drains → circular wait, permanent

#67881 adds result-queue draining during end()/join, but _process_workloads() still calls a blocking put() with no draining fallback, so the runtime deadlock remains possible on main as well.

Operating System

Amazon Linux 2023

Deployment

Virtualenv installation

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

n/a — core issue (apache-airflow-task-sdk 1.3.0 for reference)

Official Helm Chart version

Not Applicable

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

Standalone deployment: systemd units for scheduler (LocalExecutor), api-server, and dag-processor on a single EC2 host; PostgreSQL metadata DB.

Happy to provide full /proc captures, journald excerpts of the supervisor mass-terminations, and timing data from all 4 occurrences on request.

Mainly using Claude to diagnose & type this report out.

• Looked myself for existing GitHub issues, and the one I linked was the closest thing I could find to ours.
• Looked myself at our Airflow server, and found running tasks randomly killed (with the

[2026-07-27 13:41:09] INFO - spark app is running...check in again in 120
[2026-07-27 13:43:15] INFO - spark app is running...check in again in 120
[2026-07-27 13:45:20] INFO - spark app is running...check in again in 120
[2026-07-27 13:47:24] INFO - spark app is running...check in again in 120
[2026-07-27 13:49:29] INFO - spark app is running...check in again in 120
[2026-07-27 13:50:40] ERROR - Server indicated the task shouldn't be running anymore. Terminating process detail={"reason":"not_found","message":"Task Instance not found, it may have been moved to the Task Instance History table"`)

message). This is what Claude diagnoses as an effect of a scheduler deadlock

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[darren-cheung-sa]

@SakshamKapoor2911 wondering if you could confirm this issue was similar to yours / shed some light on it / suggest solution

I don't think your fix was included in 3.3.0, was it?

[SakshamKapoor2911]

Great diagnosis — this is indeed the dispatch-side gap that #67881 (shutdown end()) did not cover. The same drain-before-write pattern applies: calling _read_results() before each activity_queue.put() in _process_workloads ensures workers blocked on a full result pipe get unblocked and can resume consuming from the activity queue, preventing the circular wait.

Verified locally with existing tests (17 pass, 0 regressions) plus two new ordering tests. Reference branch with fix + tests:

https://github.com/SakshamKapoor2911/airflow/tree/fix-local-executor-dispatch-deadlock

Feel free to cherry-pick, adapt, or ignore — I will not open a competing PR since you have already put in the diagnostic work. Happy to review/support yours when ready.

---

Drafted-by: OpenCode (DeepSeek v4 Pro); reviewed by me before posting

[darren-cheung-sa]

Thanks @SakshamKapoor2911 , I will take a look at your changes and try to adapt them for our use case. If successful, I'll open a PR and tag you for review, much appreciated

Could you explain how you proceeded after that / how I should proceed after that? I assume your fix isn't in the published Airflow 3.3.0 version. Are you somehow running your tasks on a different version of Airflow (e.g. the one with your fix), and if so, could you tell me how I could do that?

[SakshamKapoor2911]

Correct — neither fix is in the published 3.3.0. #67881 was merged to main after the 3.3.0 cut and will ship in the next release.

Both can be hot-patched directly into your 3.3.0 installation by editing a single file: <venv>/lib/python3.x/site-packages/airflow/executors/local_executor.py.

Shutdown deadlock (#67881) — apply the merged diff, which drains result_queue during worker join in end() and implements terminate() to force-kill workers (currently a no-op).

Dispatch deadlock (#70526) — add one line in _process_workloads(), inside the for workload in workload_list: loop, before self.activity_queue.put(workload):

self._read_results()

This drains the result queue before each dispatch, unblocking workers so they can resume consuming the activity queue — breaking the circular wait during normal runtime.

I didn't deploy either in production myself — both were verified against unit tests. My branch with the dispatch fix + tests is here. Would you like me to open a PR? I think you should be able to drive it — you've already done the hard diagnostic work here — but happy to co-author or take it if you'd rather focus on getting your deployment stable first.

[darren-cheung-sa]

@SakshamKapoor2911 It would be great if you could take on the PR. I'm new/unfamiliar to this issue and to contributing to this repo (and as you said, also trying to make fixes to our pipelines). After that, I'll try to pull your fixes into my deployment (also new to that, but your steps seem easy enough)

Thanks for all the work & communication!

[SakshamKapoor2911]

I opened #70543 for discussion. It asks whether a continuous background drain thread would be a better long-term fix than the spot-drain approach both our PRs use.

The spot-fix from #70526 (drain before each put) is correct and works. The question in #70543 is whether we should prevent the underlying condition altogether so future contributors do not need to keep adding these defensive drains. No strong opinion either way, I wanted to surface it for maintainer guidance.

Would welcome your thoughts there since you have diagnosed the same root cause from the production side.