Deferred bash responses can be starved by runtime maintenance after task completion

[hheei]

Short description

A completed foreground Bash task can remain unanswered until the host transport times out because the standalone NDJSON loop runs runtime maintenance before flushing ready deferred responses.

What happened?

On macOS arm64 with AFT 0.49.0 under a Pi harness, two Bash requests each timed out at the host transport boundary after 130 seconds. The corresponding persisted AFT Bash task metadata showed that one task had already reached a terminal state after about 4 seconds. After the two Bash timeouts, bash_drain_completions also received no response and eventually caused a bridge restart.

This is not a slow shell command: the Bash child had already exited, but its deferred response was not written to NDJSON stdout.

Expected behavior: once a foreground Bash task is terminal, its response should be flushed promptly even if watcher/index/configure maintenance is busy.

Diagnostics

• Host: Pi-based integration
• AFT bridge package: 0.49.0
• AFT binary: 0.49.0
• Platform: macOS arm64
• Source revision inspected: 566bcde25cbb2c24f09fdc5e4840a58a65a97677

Related: #117 fixed passive status polls incorrectly killing a busy bridge. That fix is already an ancestor of the revision above. This report is a separate failure mode: a terminal Bash task has a ready deferred response, but the binary does not emit it.

Source analysis

In crates/aft/src/main.rs, the periodic/request path currently does:

drain_runtime_events(registry);
write_ready_pending(registry.current(), pending)

A synchronous stall in drain_runtime_events can therefore starve every ready deferred response, including a completed foreground Bash task and bash_drain_completions. The observed sequence is consistent with this ordering.

I do not have a stack trace from the original stalled process, so I cannot identify the exact maintenance sub-phase that blocked. The ordering itself is independently reproducible as a starvation condition.

Proposed fix

Flush ready pending responses before runtime maintenance, then flush again after maintenance so responses that become ready during the drain are also emitted:

let ctx = registry.current();
let mut written = write_ready_pending(ctx, pending)?;
drain_runtime_events(registry);
written += write_ready_pending(ctx, pending)?;
Ok(written)

The same ordering should be used by the test writer helper.

Regression coverage performed

I added a focused regression that registers a ready deferred response and a configure-warning maintenance event. It asserts the pending response is polled/written before the maintenance callback runs; the prior ordering fails that assertion.

Validation of the local patch:

• pending response suite: 8 passed
• Bash terminal/promote state tests: 4 passed
• release binary built successfully
• direct @cortexkit/aft-bridge smoke against the built binary completed a real Bash task and retrieved its output

I can provide the minimal patch and test if that would help.

[akumria]

Independent repro on the same version, with telemetry that points at the same ordering — plus a note that the proposed fix may only be half of it.

Hit this on AFT 0.49.0, macOS arm64 (same as the report) under an OpenCode harness. Different trigger, same starvation signature.

Trigger: git clone of a large monorepo (~213 MB) into a directory under a watched reference root. That pushed the working set to 144,209 files, and tier2 inspect maintenance then ran long enough to starve the NDJSON response loop. For ~15 min essentially every tool call (bash, read, edit, even configure/version) timed out at the transport boundary with bridge busy/under load — kept warm, retry.

Telemetry from aft-plugin.log during the stall:

perf tier2 phases category=dead_code      scan=9ms(144209 files)  db=7058ms
perf tier2 phases category=unused_exports scan=27565ms(3 files)   db=6985ms  rollup=77741ms
perf tier2 phases category=cycles         scan=5952ms(3 files)    db=5354ms(txn=5348)
perf tool=read  total=40614ms pre=40609ms bridge=6ms   post=0ms
perf tool=edit  total=12999ms pre=12915ms bridge=83ms  post=1ms

The read is the tell: 40.6 s queued (pre), 6 ms of actual work (bridge). The response path wasn't slow — it was blocked behind maintenance, exactly matching your drain_runtime_events → write_ready_pending ordering analysis in crates/aft/src/main.rs. A single unused_exports rollup=77741ms lines up with the multi-request stall windows.

Where my case may extend the report: your proposed patch —

let mut written = write_ready_pending(ctx, pending)?;
drain_runtime_events(registry);
written += write_ready_pending(ctx, pending)?;

fixes responses that are already ready before a drain. But when a single maintenance pass itself runs for ~77 s (large cold index), a request that arrives mid-drain still can't be serviced until the drain returns — the flush-before/after bracket doesn't help there, because the starving work is one synchronous call, not a scheduling-order artifact. So the ordering fix looks necessary but not sufficient for the large-repo cold-index case.

Might be worth pairing it with one of:

• bounding/chunking drain_runtime_events so it yields to the response loop periodically, or
• moving tier2 inspect rollups fully off the request/response thread (the perf tick already reports an interactive vs maintenance executor split, but interactive calls still queued here), or
• a per-tick maintenance time budget.

Happy to share the full log window or test against a patch. Repro is reliable: point AFT at a watched root, drop a 100k+ file tree into it, and issue tool calls during the first tier2 pass.