Report liveness while a provider call is in flight (Trace ABI v8)

[shanev]

A no-progress watchdog cannot distinguish "blocked on the network" from "wedged in the interpreter" — both are silence — so hosts were killing healthy runs whose subcall simply took a while.

Measured on a real 314k-message corpus: a run doing genuine work was SIGKILLed at 210s against a 180s no-progress budget, while every turn that ever succeeded finished in under 76s (p50 15.6s, p95 65.4s across 20 answered turns).

The new event

Transient heartbeat, carrying elapsed_ms and nothing else. Fires every 15s while any provider call is outstanding. No content, so it is not a retention or privacy question and can never reach a run record.

Why relay-side, not engine-side

This is the whole design and it is the reason the fix works at all.

Pyodide runs on the relay's thread. A timer here cannot fire while generated code spins in WASM — which is exactly when a watchdog should fire. It ticks only while the event loop is free, i.e. only while the process is genuinely healthy waiting on I/O.

So a heartbeat can never mask a real wedge. It only reports the healthy case, which is precisely the discrimination hosts were missing. A test pins it directly: ticks land during an await, and zero land while the thread is blocked.

Why raising the timeout was rejected

It trades one unmeasured number for another, still kills a batch that runs a minute longer, and leaves the watchdog unable to tell apart the two states it exists to distinguish.

Why reasoning_delta alone was not enough

It already streamed and was simply being ignored by hosts, which looked like a free fix. It is not sufficient:

_post_endpoint(self._batch_endpoint or "/responses/batch", body)

wantsStream requires the URL to end with /responses. Batch subcalls do not match, so they stream nothing and go completely silent between start and completion — and a batch is exactly the call most likely to run long. That gap is what made the ABI bump necessary rather than optional.

Verified against a live provider call

The second commit exists because the first implementation was wrong and only running it revealed that.

It wrapped fetch alone. A streamed response resolves fetch as soon as headers arrive and then spends its minutes reading the body, so the heartbeat stopped before the wait it exists to cover. A live query produced 4428 reasoning deltas and zero heartbeats. The unit test passed throughout — it proved the timer mechanism worked, not that it spanned the right window.

Now started before the request and stopped in a finally, covering headers, body, streaming, and every error path. Re-verified live:

HEARTBEATS: 7   version=8   class=transient
  15003 → 30004 → 45005 → 60006 → 75008 ms   (a single 75-second call)
  15002 → 30003 ms

That 75-second call previously emitted nothing for its entire duration. Two or three inside one execution is how a healthy run accumulates 180s of apparent silence.

Those frames were depth: 1 — subcalls, the case this exists for. The frame that run actually emitted is pinned verbatim as a test, including an envelope field (parent_run_id) the synthetic test lacked, and confirmed to pass isRlmEvent — the same forwarding filter that silently ate every checkpoint in the v7 emoji bug.

Testing

1261 passed, 3 skipped, ruff clean, deno test pyodide/ 65 passed, deno check clean.

Notes for review

• ABI 7 → 8: fixtures renamed trace_v7_* → trace_v8_*, and droste.testing exports change accordingly. Embedders must update; UPGRADING.md covers it in the release PR.
• The 15s interval is a first choice, not a measured one. It is well inside any plausible watchdog budget and cheap (one tiny transient event per 15s of waiting), but if a host wants tighter liveness it is a constant.
• Hosts that ignore the event are unaffected — it is additive and transient.

🤖 Generated with Claude Code