Repository navigation
Idea: opt-in runtime recovery middleware (auto-pause on quota exhaustion / tool-failure storms, checkpoint & resume) #6747
Replies: 4 comments
|
I prototyped this against my current local checkout:
All tests used deterministic offline models/tools, sockets disabled, and no production-source changes. ResultsQuota recovery: 5/5 runs caught a simulated quota exception in Tool-failure storm: 5/5 runs paused after three consecutive failures, with the failure counter committed to graph state before interruption. A freshly constructed agent/policy restored the checkpoint and resumed the same thread successfully. Completed failed calls were not replayed. Windowed-failure and streak-reset controls also passed. From these experiments, the current public APIs already provide the core mechanism:
I did not need private execution APIs or HITL approval semantics. The remaining difficulty looks mostly like policy definition and replay semantics: distinguishing hard quota from transient failures, defining failure windows/streaks, persisting counters before pausing, and deciding how provider recovery should interact with resume. So my current preference is an opt-in application-level helper first, built on the existing middleware APIs. I haven't yet found a missing runtime primitive that clearly requires a Deep Agents built-in. These tests were synchronous and used If this direction makes sense, I'd be happy to turn the findings into focused regression tests and a small public-API prototype/example before proposing a first-class abstraction. |
|
Thanks for the thorough validation — this is exactly the kind of evidence the idea needed, and the 10/10 pass rate on both scenarios is encouraging. A few thoughts in response:
Happy to review the regression tests and the public-API prototype example when you have them; a small runnable example would also give maintainers something concrete to evaluate alongside the discussion. |
|
One more signal worth having in the first set: a per-run spend ceiling, because it's the cleanest pause trigger of the lot. A provider 429 is ambiguous (did the attempt bill? is it per-minute or monthly?), while a budget refusal is deterministic and happens before any provider call, so pausing on it never wastes tokens and resuming is safe. If the run's model calls go through a gateway that enforces a per-run budget, the refusal reaches import openai
from langchain.agents.middleware import AgentMiddleware
from langgraph.types import interrupt
class PauseOnBudget(AgentMiddleware):
def wrap_model_call(self, request, handler):
try:
return handler(request)
except openai.APIStatusError as e:
if e.status_code != 402:
raise
interrupt({"signal": "budget_exhausted", "error": e.body})
return handler(request) # resumed, e.g. after the budget was raised
model = ChatOpenAI(
model="gpt-4o-mini",
base_url="http://127.0.0.1:8000/v1", # the gateway
api_key="unused",
max_tokens=500,
default_headers={"X-Inferrail-Attribute-Work-Id": thread_id, "X-Inferrail-Budget-Usd": "5.00"},
)
agent = create_deep_agent(model=model, middleware=[PauseOnBudget()], checkpointer=InMemorySaver())With a budget too small for the next call: 1 request, refused before the upstream, Disclosure: the gateway in the example is Inferrail, which I maintain (open source). The middleware itself only depends on a 402 arriving before the provider is called, so any budget-enforcing proxy that does that fits the same slot. For 宇宙的山峰 (@UniversePeak)'s two-hour case specifically, the budget turns "keeps going until someone looks" into a pause at a dollar amount you picked. |
|
I would start with an application-level helper, but give write tools a separate recovery decision: a tool exception does not establish that its remote side effect is absent. For that helper, I would record an
Before resuming an unconfirmed write, reconcile the intended destination/account, exact operation and any original receipt. If the service supports idempotency, reuse the original operation identifier only for the same intent/payload and within its documented contract. A local payload hash alone does not make a browser form idempotent. If neither reliable reconciliation nor an idempotency contract is available, I would keep that mutation unconfirmed rather than automatically create a second effect. AWS's retry/idempotency discussion explains the missing-response ambiguity and why the service contract matters. One additional prototype case: a mock executor changes remote state, then loses its response before the caller records success. Resume after that interruption, including from an older checkpoint. With a reliable lookup or the mock service's idempotency contract, the intended effect count should remain one; without either, the helper should leave the outcome unconfirmed and avoid another create. This would test the recovery decision separately from the existing quota and failure-counter cases. BrowseSprint's dated manual field note records a form reset with no receipt or matching email found during checking; it was retained as unconfirmed, not treated as proof of failure. I maintain BrowseSprint and its manual browser-workflow measurement guide. This is a proposed recovery test, not an executed Deep Agents integration or a measured speedup. Prepared with AI assistance and checked against the cited sources. |
Uh oh!
There was an error while loading. Please reload this page.
An unattended deepagents run that goes wrong mid-flight has only two outcomes today: it fails loudly, or it keeps going until a human happens to look. Nothing in the middleware stack watches for the common autonomous-run anomalies — provider quota exhaustion, a tool failing over and over, repeated sandbox/permission denials — and nothing pauses the run on its own. We've run into this ourselves: an unattended run kept executing for over two hours between the first alert and someone manually stopping it.
The building blocks are already there in the current middleware stack:
HumanInTheLoopMiddlewarecan interrupt a run, but resuming is always a manual human step.PatchToolCallsMiddlewarealready repairs dangling tool calls when a thread resumes, so a paused thread can be continued cleanly.Proposal: an opt-in recovery middleware. It observes the run and, when a signal trips, raises a typed interrupt that carries a checkpoint marker, so the run pauses with state persisted instead of dying or spinning. Signals to start with:
Resume semantics stay with the caller: explicit resume continues from the checkpoint; the anomaly context (signal type, counts, last error) travels with the interrupt so the resuming side can decide. Runs that don't opt in are completely unaffected.
The observable difference: today a run that hits a hard quota block either dies mid-task or, inside retry loops, churns for hours before anyone looks. With this, time from anomaly to paused is seconds, and resume continues from the checkpoint instead of restarting the task.
Tests I'd start with: a run hitting a quota block pauses with a persisted checkpoint; a resumed run continues from the checkpoint and finishes the task; a tool-failure storm pauses the thread instead of failing the whole workflow; and with recovery disabled the existing behavior is byte-for-byte unchanged.
One question before sketching a
RecoveryPolicyinterface: do you see this surface belonging in deepagents middleware, or would you rather treat it as an application-level pattern composed from the existing HITL interrupt plus a checkpointer?All reactions