rpc: answer calls pipelined on an already-failed answer with that exception - #8
Closed
nullstyle wants to merge 1 commit into
Closed
rpc: answer calls pipelined on an already-failed answer with that exception#8nullstyle wants to merge 1 commit into
nullstyle wants to merge 1 commit into
Conversation
nullstyle
force-pushed
the
claude/quirky-jones-d7ee04
branch
from
July 31, 2026 05:29
020cf35 to
286d77f
Compare
…eption A Call targeting the promisedAnswer of an answer that already returned an exception queued in pending_promises forever: resolved_answers records results Returns only, so a failed answer missed there identically to a still-pending one, and the queued-call drain in sendReturnException runs before a pipelined call sent in the same burst can arrive. Found on the wire by the cross-impl L3 lane (pipelined-provide scenarios): the C++ recipient pipelines its Call on the Accept question, the host refuses the Accept fail-closed, and the Call never got a Return — the recipient hung on the exactly-one-Return-per-call guarantee. Record exception Returns in a failed_answers map (owned reason + type, kept until Finish — the resolved_answers lifecycle), written in sendReturnExceptionNoDrain, the funnel every exception Return passes through, so drained pipeline children self-record and transitive late arrivals are covered. planPromisedTarget consults the record exactly where it would otherwise queue, and the call is answered immediately with a COPY of the recorded exception, preserving the retryability signal — the broken-pipeline behavior of the C++ reference. failed_answers also joins the inbound answer-id namespace so a stale record can never be replayed against a reused id, and the record is skipped for loopback and finished-early answers (no Finish would ever clear it). Ablation-proven tests pin both shapes: the refused cross-peer Accept with a pipelined Call (the e2e wire trace, in the vatc suite) and the plain two-party failed answer (from_peer suite), each asserting reason AND type survive the copy and that Finish drains the record. The e2e C++ driver now observes the refusal THROUGH a pipelined call instead of working around the gap with whenResolved(); the full lane passes against the vendored C++ reference (18/18), full suite 914/914, Stable API surface unchanged (experimental snapshot +4 lines). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nullstyle
force-pushed
the
claude/quirky-jones-d7ee04
branch
from
July 31, 2026 05:56
286d77f to
cade98e
Compare
nullstyle
added a commit
that referenced
this pull request
Jul 31, 2026
…remove Its original staging used a receiverHosted target, which f2d89ee now SERVES, so the test referenced a constant that no longer exists and did not compile. The rule under test is the broken-pipeline one -- every Call gets exactly one Return, and a call pipelined on a failed answer gets a copy of that exception -- not any particular refusal, so it moves to the refusal that survives: a site-2 `.promised` target whose import dies before the Accept (site 2 takes no Provide-time pin by design). Also applies the lift's probe-lifetime lesson: the staging probes live in the test frame, because the staged question is still outstanding when the vat is torn down and is cancelled THROUGH that ctx pointer. Ablation: disabling the failed_answers lookup reddens this test with `expected .exception, found null` -- the parked-forever symptom -- and #8's own unit test alongside it. The teeth survive the rebase. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
nullstyle
added a commit
that referenced
this pull request
Jul 31, 2026
…eption (rebased on L17) (#10) * rpc: answer calls pipelined on an already-failed answer with that exception A Call targeting the promisedAnswer of an answer that already returned an exception queued in pending_promises forever: resolved_answers records results Returns only, so a failed answer missed there identically to a still-pending one, and the queued-call drain in sendReturnException runs before a pipelined call sent in the same burst can arrive. Found on the wire by the cross-impl L3 lane (pipelined-provide scenarios): the C++ recipient pipelines its Call on the Accept question, the host refuses the Accept fail-closed, and the Call never got a Return — the recipient hung on the exactly-one-Return-per-call guarantee. Record exception Returns in a failed_answers map (owned reason + type, kept until Finish — the resolved_answers lifecycle), written in sendReturnExceptionNoDrain, the funnel every exception Return passes through, so drained pipeline children self-record and transitive late arrivals are covered. planPromisedTarget consults the record exactly where it would otherwise queue, and the call is answered immediately with a COPY of the recorded exception, preserving the retryability signal — the broken-pipeline behavior of the C++ reference. failed_answers also joins the inbound answer-id namespace so a stale record can never be replayed against a reused id, and the record is skipped for loopback and finished-early answers (no Finish would ever clear it). Ablation-proven tests pin both shapes: the refused cross-peer Accept with a pipelined Call (the e2e wire trace, in the vatc suite) and the plain two-party failed answer (from_peer suite), each asserting reason AND type survive the copy and that Finish drains the record. The e2e C++ driver now observes the refusal THROUGH a pipelined call instead of working around the gap with whenResolved(); the full lane passes against the vendored C++ reference (18/18), full suite 914/914, Stable API surface unchanged (experimental snapshot +4 lines). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: repoint the broken-pipeline test at a refusal the lift did not remove Its original staging used a receiverHosted target, which f2d89ee now SERVES, so the test referenced a constant that no longer exists and did not compile. The rule under test is the broken-pipeline one -- every Call gets exactly one Return, and a call pipelined on a failed answer gets a copy of that exception -- not any particular refusal, so it moves to the refusal that survives: a site-2 `.promised` target whose import dies before the Accept (site 2 takes no Provide-time pin by design). Also applies the lift's probe-lifetime lesson: the staging probes live in the test frame, because the staged question is still outstanding when the vat is torn down and is cancelled THROUGH that ctx pointer. Ablation: disabling the failed_answers lookup reddens this test with `expected .exception, found null` -- the parked-forever symptom -- and #8's own unit test alongside it. The teeth survive the rebase. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * e2e: the pipelined probe is a real call, so B's cap is invoked twice Merge artifact, caught by CI's e2e lane. On main the probe was `whenResolved()`, which settles on the Accept's Return WITHOUT invoking the capability, so one real call followed and the driver asserted `localNum.calls == 1`. The rebase correctly kept the stronger probe from the broken-pipeline work -- a genuinely pipelined `getNumberRequest()`, which is what exercises the rule -- but L17's count assertion rode along unchanged. Two invocations is the right expectation now, and both are load-bearing: the first proves a call pipelined on the Accept question is answered, the second that the settled capability still routes to B's own cap rather than host-Carol's. Lane verified locally: 18/18. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A Call targeting the
promisedAnswerof an answer that already returned an exception was never answered — it queued inpending_promisesforever. Found on the wire by the cross-impl L3 lane (just e2e-l3-vatc, pipelined-provide scenarios): the C++ recipient pipelines its Call on the Accept question, the host refuses the Accept fail-closed (CrossPeerReceiverHostedTargetUnsupported), and the pipelined Call never receives a Return. The recipient hung on the spec's exactly-one-Return-per-call guarantee (only the harness's 15s deadline unwedged the e2e).Why it happened
resolved_answersrecords results Returns only (kept until Finish so late pipelined calls can resolve against them). Exception Returns were recorded nowhere, so at the planner a failed answer was indistinguishable from a still-pending one →.queue_promised_call. The queued-call drain insendReturnExceptiononly reaches calls queued before the exception Return; a call pipelined in the same burst always arrives after a synchronously-failed answer. Not Accept-specific — any answer failed synchronously during dispatch had the same window.Fix
failed_answersmap onPeer(owned reason + exception type, kept until Finish — the exactresolved_answerslifecycle). Written insendReturnExceptionNoDrain, the funnel every exception Return passes through, so drained pipeline children self-record and a call pipelined on a failed pipelined call is covered transitively.planPromisedTargetconsults the record exactly where it would otherwise queue; the new.fail_broken_answerplan arm answers the call immediately with a copy of the recorded exception, preserving the retryability type — the broken-pipeline behavior of the C++ reference.failed_answersjoinsinboundAnswerQuestionIdInUseso a reused id can never replay a stale record, and the map is bounded bymax_active_inbound_questionsrather than a newPeerLimitsfield, sincePeerLimitsis Stable-frozen. Recording is best-effort under OOM/budget pressure — a skip degrades to the old behavior for that answer, never a crash.Reviewer notes
51abe13(the revert of the receiverHosted lift). This branch briefly tracked the lift while it was on main; with it reverted, the fail-closed refusal this bug is observed through is back, and the branch is once again the original change — no lift-specific adaptation remains.check-apigreen;docs/api-snapshot-experimental.txt+4 lines: the Peer field andstate.FailedAnswer).whenResolved()with a comment marking this exact gap; it now probes through a pipelined call — that wait hung forever pre-fix, so the lane itself is the cross-impl regression proof.lookupFailedAnswerstubbed to return null, both new tests fail exactly as the pre-fix bug predicts (no Return for the pipelined call; call parked inpending_promises).Commands run
zig build test --summary all— 143/143 steps, 1166/1166 testszig build check,zig build check-api,just fmt-check— greenjust e2e-l3-vatc— 18/18 TAP against the vendored C++ reference🤖 Generated with Claude Code