Balance guard: refuse the next model request when the provider says is_available=false, and make an exhaustion stop resumable

[simorgh1]

Companion to the bug report in General: #7052 — HTTP 402 (insufficient provider balance) never reaches the terminal QUOTA failure code, which covers the classification. This post is the design question behind it.

Summary

A long run started, spent ~20.8M input tokens, and died at the finish line with no deliverable. By then my balance was −$0.24. The harness had no idea, at any point, how much money was left:

1. It never reads the balance. DeepSeek's GET /user/balance returns is_available — literally "Whether the user's balance is sufficient for API calls" — plus balance_infos (currency, total_balance, granted_balance, topped_up_balance). The harness calls it nowhere. Grepping the checkout for user/balance, balance_infos, or is_available returns nothing, and there is no maxCost / spendLimit / per-session budget setting in the configuration catalog.
2. The provider's own gate is not a substitute. Because requests were accepted below zero, a client-side cutoff at (or just above) zero would have stopped earlier and safer than the provider did. The cutoff has to be ours.
3. Nothing accounts for spend across a fan-out. packages/workflow/workflow/README.md lists "No journaling or resume" next to "No token-budget vocabulary — engines cap concurrency, items, and children, but neither the request nor the result accounts for model tokens across children." So during a workflow run no component sees the total spend.

The core ask is arithmetic, not forecasting

If the tank reads zero, the engine should not start. Concretely, before dispatching a request:

available_balance − (spend committed since the balance read, all consumers) − price(next request) ≤ floor  →  refuse

No prediction is involved:

• The provider hands over the go/no-go flag (is_available), so the harness does not even have to interpret the number.
• The price of the request about to be sent is already computable: packages/llm/token-meter prices the current surface per route (route-pricing.ts → priceSurface).
• The balance is stale between reads, so sample it per step (or every N steps), cache it, and surface the value with its timestamp.
• Balance is account-global while session spend is local, so the second term must aggregate nested children and concurrent sessions on the same key — otherwise a fan-out sails past the gate because no single session sees the total.
• A floor slightly above zero is advisable, since a request that has been dispatched is billed even when it is the one that empties the account.

The second half: make the stop resumable

A cutoff alone does not fix this failure. A task that begins with enough balance to get halfway will still stop halfway; what made it catastrophic is what happens next. packages/session/session-persistence/README.md states the resume contract plainly: "Synthetic closers are the only crash story… there is no partial-turn resume that continues an interrupted turn instead of closing it." Combined with the workflow package's lack of journaling, recovering means re-deriving the work and re-running every child — so the tokens already paid for are simply gone.

On a genuine exhaustion refusal, the run should enter a durable awaiting_balance state and resume from the log after a top-up, instead of closing the turn as a failure. For nested delegation, each child's structured result would need checkpointing so a resumed run replays completed items rather than re-burning them.

Proposed behavior

1. Refuse the next request when the account cannot cover it — on is_available === false, or when total_balance minus the price of the next request falls below the floor. Sample per step; show the value and its timestamp.
2. Apply the same check during a workflow fan-out, aggregating child spend or checking before each child dispatch. At minimum, stop dispatching new children at the threshold and let in-flight children return their results. The four durable records already written per run (run-start, member starts/endings, run-end) could carry the spend.
3. Make the refusal a state the user can leave — awaiting_balance plus resume from the durable log, rather than a closed turn.
4. Related: llm-retry supports an unlimited always mode (packages/llm/llm/src/retry-policy.ts). Against an exhausted balance that mode would retry forever, billing every attempt and never succeeding; a balance gate there would turn it into genuine "wait for top-up, then continue".

What I am not asking for

Not cost forecasting, and not a predicted total for a task. Step counts are model-decided and fan-out multiplies them, so an exact pre-flight estimate is not achievable, and I would not want one gating a run. A large-job estimate is at most a later, optional nicety. The request here is an arithmetic gate — can the account pay for the next request, yes or no — plus a stop that can be resumed.

Where this could live

packages/guard/ already hosts small loop-hygiene guards (repeat-tool-reminder, timeout-policy), and the agent/pre-step / agent/request-error waterfalls are the established seams (both llm-retry and compaction attach there). session-checkpoint-policy already enforces fail-closed boundaries before a model request, before top-level side-effecting tools, and before the next step — the same boundary a spend check would use, which is also where a refusal could be made resumable rather than terminal.

Environment

DSH 0.1.5-rc.2, DeepSeek provider; ~20.8M input tokens consumed before failure, ending balance −$0.24. Happy to supply session logs, or to run a repro against a drained test key if that helps.

[argszero]

You asked for the arithmetic, not forecasting, and for a stop the user can leave. Two of those three are answerable today, and one is honest-to-goodness not a plugin's to give. Here is what I measured, and what I shipped.

PreStepDecision already is the gate you describe. agent/pre-step runs after the inbox batch is claimed and before the request is built, and its decision type is { kind: 'reject' } | { kind: 'enter', messages } (packages/core/agent/src/runtime-types.ts:112). The loop maps a rejected step to turnEnds = { kind: 'blocked' } (packages/core/agent-loop/src/agent.ts:292), which is a non-error terminal state — TurnEndReasonMap.blocked, understood by the ACP codec, the subagent lifecycle and the web projections. So "if the tank reads zero, the engine does not start" is expressible: zero requests dispatched, zero billed, nothing half-executed.

The trap is that preStep() claims the batch before the waterfall runs (agent.ts claim → dispatch.waterfall('agent/pre-step', { messages: claimed, ... })). A naive reject therefore silently swallows the prompt the user just sent — the same failure class as #6964. The shape that does not is already in-tree: goal-round-driver restores what it refused (packages/goal/goal-round-driver/src/index.ts:127-135) using agent.inbox.prepend(...), and Inbox.prepend/append/replace/remove/clear are public contract (runtime-types.ts:48-93). Suppression by id matters: the inbox projection's fold rejects a duplicate message id.

Two other facts that decide the design:

• No hot loop. wakeRequested latches only for maintenance or aborted drivers (agent.ts:188-198); a live driver that is refused simply ends the turn. So a persistent "no" does not spin.
• Credentials are reachable through ctx.credentials.resolve(...), and without that service through the launch-environment snapshot — the same two-step resolution llm-deepseek itself uses.

What I shipped, so this is not a design sketch: @argszero/cordis-plugin-balance-guard@0.1.0 (source). It samples GET /user/balance on that seam and refuses the step when is_available === false, or when the reported total minus reserveAmount is at or below minBalance. Your three "no prediction involved" points are exactly the implementation: the provider's flag is the go/no-go, the balance is account-global (one cache for the whole process, not per session), and the value is sampled with its timestamp available in the log.

The measured behavior, from an end-to-end proof that mounts the packed tarball into the real harness (real Context/SessionStore/AgentRegistry/AgentLoop, and a model adapter that only counts dispatches), run on both ends of the declared range with identical results:

// phase A — provider reports is_available=false
{ "modelCalls": 0,
  "turnEndReasons": [{ "kind": "blocked" }],
  "eventTypes": ["agent/inbox/spliced", "turn/start", "agent/inbox/spliced",
                 "agent/inbox/spliced", "turn/end"],
  "durablyRecordedUserMessage": false, "stepsStarted": 0,
  "inboxTextAfterA": "and now spend the rest of the balance on this" }

// phase B — after a top-up, the surviving prompt runs
{ "modelCalls": 1, "firstRequestCarriedTheSurvivor": true, "inboxAfterB": 0 }

modelCalls: 0 is the claim that matters for your −$0.24: the refusal dispatched nothing, so it billed nothing. And phase B is the part that addresses your second half as far as a plugin can: the message that would have been eaten by the refusal is the one the model receives after the top-up. The stop loses no work — it just does not continue the turn.

Where I am deliberately weaker than your proposal, stated plainly:

1. No awaiting_balance state. "Resume from the durable log after a top-up instead of closing the turn" is a session-persistence property — synthetic closers are the harness's only crash story, and no plugin can add a resume primitive. What the plugin guarantees is that the stop is free and the input survives; the resumable half is core's.
2. No price(next request) term. Pricing the request about to be sent means pricing the latest durable routed envelope through ctx.tokenMeter — real, but it couples the guard to routing, so v0.1.0 takes reserveAmount as the manual stand-in.
3. No veto of llm-retry retries. With retry mode always, the retry happens inside a step that already passed the gate, and agent/request-error cannot override another listener's { kind: 'retry' }. Your point that a balance gate belongs there too is correct and is not expressible from a plugin.
4. Fan-out: nothing in the workflow engine accounts for child spend. The plugin's sample is account-global, so the gate is evaluated for child agents too rather than being blind to them — but "stop dispatching new children and let in-flight ones return" is workflow-engine work, not a guard's.

Everything else — bounded by sampleTimeoutMs and the step's own abort signal, cached per account with a TTL, and fail-open: unreachable endpoint, non-2xx, malformed body, missing credential or a throwing credential plane all allow the request. A guard that can stop work because a monitoring call failed would be worse than no guard, so that is the one property I tested from every angle.

Your companion report #7052 is real too, and its fix is smaller than it looks — I replied there with the reason (the sibling Messages path in the same package has had the 402 rule all along).