Repository navigation
Description
Title: Embedded runner: freshly streamed thinking signatures intermittently invalid on replay (Anthropic); recovery wrapper never fires because error text is genericized
Summary
On 2026.6.5, the embedded agent runner (Slack plugin) intermittently persists Anthropic thinking blocks whose signatures fail validation when history is replayed — permanently poisoning the session: every subsequent message fails with invalid_request_error: messages.N.content.0: Invalid signatureinthinking block, surfaced to the user as LLM request failed: provider rejected the request schema or tool payload.
Additionally, wrapAnthropicStreamWithRecovery — which exists precisely for this and would auto-heal by stripping thinking blocks and retrying — never fires on this path: zero [session-recovery] log lines across all incidents. The error reaching the recovery predicate appears to be the genericized "LLM request failed: provider rejected the request schema or tool payload." text (raw provider error attached separately as rawError= in logs), which THINKING_BLOCK_ERROR_PATTERN cannot match.
Environment
- OpenClaw 2026.6.5 (ghcr.io/openclaw/openclaw:latest pulled 2026-06-10), embedded runner via Slack plugin
- Provider: anthropic, model claude-sonnet-4-6, thinkingLevel high
- First occurrence ever ~17h after upgrading the image; 3 separate poisonings of the same session within ~2h on 2026-06-11; at peak, every fresh thinking block in the affected thread was bad (one reply works → next message bricks)
Evidence isolating the layer
- Replay/persistence exonerated: extracted each stored thinkingSignature from the session jsonl and replayed each block individually against api.anthropic.com (same key/model): blocks generated earlier that day validate; blocks generated from 13:36 UTC onward fail. Stored signature and thinking text are byte-identical to the post-assembly trajectory log → nothing mutated after assembly.
- Anthropic exonerated: direct curl→API generate+replay with the same key/model, both non-streaming and streaming (manual thinking_delta/signature_delta concatenation), validated 4/4 while OpenClaw was concurrently producing invalid pairs.
- → corruption happens between the wire and the assembled message inside the runner's stream assembly. Note the build already contains the signature_delta append fix from fix(agents): concatenate signature_delta chunks in transport stream #87582 (block.thinkingSignature = (block.thinkingSignature || "") + delta.signature), so this is something else — possibly related to 2026.6.5's deferred stream-start / message_start changes.
Two requested fixes
- Find/fix the stream-assembly corruption (related: Bug: anthropic-transport-stream overwrites signature_delta instead of appending, causing permanent session corruption #87574, Compaction/pruning corrupts thinking blocks → Anthropic API rejection #30157, Anthropic thinking block 'signature' field lost during session persistence — causes API rejection #43691).
- Make shouldRecoverAnthropicThinkingError see the raw provider error (e.g. include rawError in the matched text, or match before genericizing) so sessions self-heal as designed even when assembly produces a bad block.
Workaround we applied
Force dropReasoningFromHistory for anthropic in the embedded runner (build-time patch) — past thinking is never replayed, so a bad stored signature is harmless; thinking generation unaffected. Happy to provide session jsonl excerpts (sans content) / probe scripts.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity
clawsweeper commentedon Jun 11, 2026
Thanks for the context here. I did a careful shell check against current
main, and this is already implemented.Current main prevents partial streamed Anthropic signatures from entering history and recovers a pre-output replay rejection by retrying without thinking blocks, so the reported permanent session-brick behavior is addressed.
Possible regression — probable (reproduction; failure trace). No predecessor PR is attributed.
So I’m closing this as already implemented rather than keeping a duplicate issue open.
Review details
Best possible solution:
Keep the staged-signature and durable retry design; file any newly observed malformed-signature mechanism with a raw stream-to-transcript trace, since #108443 appears to concern a distinct persistence path.
Do we have a high-confidence way to reproduce the issue?
No: current main’s stream and recovery tests cover the reported failure boundaries, and source now prevents or repairs the permanent replay failure.
Is this the best way to solve the issue?
Yes: staging opaque signature deltas at the transport boundary plus a single durable pre-output recovery retry fixes the producer and the operator-visible failure without a new setting.
Security review:
Security review: This is issue triage, not a proposed patch review.
AGENTS.md: found and applied where relevant.
What I checked:
src/agents/embedded-agent-runner/run/attempt-stream.ts:118, c1217886119c)packages/ai/src/transports/anthropic-transport-stream.ts:1602, c1217886119c)src/agents/embedded-agent-runner/replay-history.ts:826, c1217886119c)src/agents/anthropic-transport-stream.ts, 3bcb90b2d08f)qa/maturity-scores.yaml, c1217886119c)Likely related people:
packages/ai/src/transports/anthropic-transport-stream.ts)src/agents/embedded-agent-runner/thinking.ts)src/agents/embedded-agent-runner/thinking.ts)Codex review notes: model internal, reasoning high; reviewed against c1217886119c; fix evidence: commit 3bcb90b2d08f, main fix timestamp 2026-07-10T20:54:37-05:00.
davidcittadini commentedon Jun 14, 2026
Independent live repro on 2026.6.6 — Telegram embedded runner, Opus 4.8 (not Slack, not Sonnet) — and confirms the genericized-error → recovery-wrapper-never-fires diagnosis
We hit this exact failure in production today and it reproduces on a newer build than the original report, on a different runner and model, so the bug is in the embedded-runner core, not the Slack plugin or a Sonnet-specific path.
Environment
anthropic, modelclaude-opus-4-8,thinkingLevel: high(original report:claude-sonnet-4-6).Trigger (matches "intermittently persists on replay")
The poisoning was immediately preceded by a session-takeover/abort during a stalled, queued run — not compaction and not
/new:A ~26-minute stalled
model_callwith a second prompt queued behind active work (activeAhead=1) → the prompt lock was released and the session file mutated concurrently →EmbeddedAttemptSessionTakeoverError. The post-takeover replay then persisted/sent thinking blocks whose signatures don't validate.Result — permanent poisoning, identical on every retry
Confirms the core hypothesis of this issue: recovery wrapper never fires
session-recoverylog lines across the entire day's gateway logs (grep -c session-recovery= 0 in bothgateway.logandgateway.err.log).wrapAnthropicStreamWithRecoverydid not engage.LLM request failed: provider rejected the request schema or tool payload.— the realinvalid_request_error … Invalid signature in thinking blockis only present as a separaterawError=field. SoTHINKING_BLOCK_ERROR_PATTERNhas nothing to match. This is the same genericization called out in Filter noisy invalid_request_error subtypes (e.g. thinking-signature) out of user-facing assistant text #87475, and here it directly defeats the auto-heal, leaving the session permanently bricked.Session-file forensics — the persisted signatures were empty, not stale
Inspecting the on-disk
.jsonl: 37thinkingblocks, every one with a zero-lengthsignature(not a stale/mismatched value). The reassembled request merged the assistant turn so the offending block landed atmessages.1.content.87. This points at the takeover/replay path persisting freshly-streamed thinking blocks without their signature — distinct from #90108 (compaction re-injecting stale signatures) and #90483 (deleteon shared object refs), though the user-visible symptom is identical. (Possibly the same underlyingsignature_deltaoverwrite-vs-append bug as #87574/#87582.)Manual remediation that worked (= the wrapper's intended action, done by hand)
Stripping all
thinkingblocks from the session.jsonl(preserving text + tool_use + tool_result) immediately unblocked the session on the next turn. This confirms the fix shape:stripInvalidThinkingSignatures/dropThinkingBlocksbefore persisting (and never persist a thinking block with a blank signature).rawError(invalid_request_error+signature/thinking), not the genericized user-facing text — otherwisewrapAnthropicStreamWithRecoverycan never trigger on the embedded-runner path (ref Filter noisy invalid_request_error subtypes (e.g. thinking-signature) out of user-facing assistant text #87475).Happy to provide the full sanitized
.jsonl, additional log windows, or the before/after block counts if useful.davidcittadini commentedon Jun 14, 2026
Cross-link / upstream root cause: this signature poisoning is a downstream effect of the
EmbeddedAttemptSessionTakeoverErrorprompt-lock-release race — same incident, samesessionId=d27c050e…. The aborted/taken-over turn is what persisted the thinking blocks without valid signatures here. Full repro steps, frequency (186 events, 2–18×/day across 3 agents), and paired-lane log evidence are documented at #86966. Fixing the takeover race upstream (serialize same-session inbound instead of aborting) would prevent this class of poisoning; stripping invalid thinking signatures on the abort/replay path is the belt-and-suspenders catch.davidcittadini commentedon Jun 14, 2026
Triage escalation — this issue is OPEN but unlabeled/unassigned, and the automated review never completed
Flagging because this looks at risk of being silently dropped:
invalid_request_error: Invalid \signature` in `thinking` blockuntil someone manually edits the session.jsonl`. That is total loss of a live session, not a transient hiccup.retryable codex transport failure (exit 1)— so it was never labeled or work-laned. It needs a human pass..6.Suggested labels:
impact:session-state(+impact:message-loss), and P0/P1 to match #86966/#88838 in the same cluster.Cluster map (so the relationship is unambiguous)
EmbeddedAttemptSessionTakeoverErrorprompt-lock-release race. The aborted/taken-over turn is what persists the signature-less thinking block. ([Bug]: EmbeddedAttemptSessionTakeoverError: paired lanes on same session race after prompt-lock release #86966 was closed as superseded by Track core session/transcript SQLite migration via accessor seam #88838.)needs-product-decision.The single highest-leverage fix (cheap, decoupled, self-healing)
The runner already ships
wrapAnthropicStreamWithRecovery/shouldRecoverAnthropicThinkingError, which exist precisely to strip bad thinking blocks and retry — but on this path it never fires (zero[session-recovery]log lines). Reason: the error reaching the recovery predicate is the genericized"LLM request failed: provider rejected the request schema or tool payload.", while the raw provider error (Invalid \signature` in `thinking` block, attached separately asrawError=) is whatTHINKING_BLOCK_ERROR_PATTERN` needs to match.Two decoupled mitigations, neither blocked on #88838:
shouldRecoverAnthropicThinkingErrorsee the raw provider error (matchrawError/ match before genericizing). This alone would let every currently-poisoned session self-heal as designed — the biggest bang for the buck.stripInvalidThinkingSignatureson the abort/replay path so an unavoidable takeover/abort doesn't poison the session in the first place (belt-and-suspenders, ties back to the [Bug]: EmbeddedAttemptSessionTakeoverError: paired lanes on same session race after prompt-lock release #86966 race).Happy to provide sanitized session
.jsonlexcerpts, the per-agent daily histogram, and paired-lane samples across the three affected agents. This is causing daily, user-visible, unrecoverable-without-manual-surgery failures — would appreciate a prioritization decision.fix(embedded-runner): detect thinking block errors in genericized cau…
fix(agents): recover genericized Anthropic thinking errors (#92916)
steipete commentedon Jun 14, 2026
The recovery half of this report landed in #92916 via fbc3fa3.
OpenClaw now detects thinking-signature errors carried through Anthropic SDK error bodies, failover
rawError, nested causes, direct messages, and terminal stream errors. The traversal is cycle-safe, does not inspect assistant text or tool arguments, and retries at most once without thinking blocks.Proof included a live Sonnet 4.6 replay with a real signed thinking block truncated to a non-empty invalid signature: Anthropic rejected the first request, recovery retried exactly once, the transcript-repair callback fired, and the final response was
RECOVERED_OK. Official OpenAI and Anthropic tool-call live tests also passed 9/9, and full PR CI was green.Keeping this issue open: the intermittent path that first creates or persists invalid/empty signatures is not identified or fixed by #92916. The remaining useful evidence is a raw stream/transcript transition showing where a freshly valid signature becomes invalid before replay.
CarlCapital commentedon Jun 14, 2026
Original reporter (#92201) here — thanks @steipete, the recovery-side fix in #92916 matches our diagnosis exactly.
One data point for the still-open create-side: our original incident was the non-empty-but-invalid variant, not the zero-length one @davidcittadini saw.
thinkingSignaturewas full-length (~2.2k chars), not blank.Caveat: our incident-window gateway logs have since rotated, so I can't confirm whether a session-takeover/abort race was involved in our case specifically.
Happy to share a sanitized jsonl (signature lengths + per-block validate pass/fail, content stripped) and the per-block API replay probe if useful.
davidcittadini commentedon Jun 14, 2026
Create-side evidence: the poison is a single thinking block whose signature is truncated/malformed at capture — confirmed against the live Anthropic API, and offline-detectable
Correcting our own earlier framing first (we implied "empty signatures"): that was wrong, and so is "all blocks affected." After probing the persisted blocks directly and replaying every one against the live Anthropic API, the real picture is narrow and clean.
Setup
One bricked production session (OpenClaw 2026.6.6,
claude-opus-4-8, adaptive thinking, Telegram embedded runner), 103 messages, 37 persistedthinkingblocks. Each block on disk is{"type":"thinking","thinking":"", "thinkingSignature":"<…>"}.Live API probe — 36 valid, exactly 1 invalid
Replaying each stored block individually against
POST /v1/messages(same model, thinking enabled):Invalid signature in thinking block(control)The single rejected block is
Invalid \signature` in `thinking` block` — the exact error that bricked the session. The other 36 replay fine today, so they were never the problem.The bad block is structurally truncated (offline-detectable)
The one rejected signature is malformed base64: it has no trailing padding and fails strict base64 decode, whereas all 36 valid siblings end with a proper trailer + padding (
…MGAE=,…AQ==) and decode cleanly. A plain base64-validity check over the 37 signatures flags exactly one block — the same one the API rejects (1:1). So at least this variant is catchable without a provider round-trip, e.g. as a persist-time or pre-send guard.Origin: invalid at capture, during a normal turn — not the takeover
The bad block was written at 06:53:09 during an ordinary streamed turn (msg 49 of 103). 16 later thinking blocks (through 07:03) are all valid. The session kept working for ~13 min because OpenClaw strips thinking from history on normal sends; it only died at 07:07 when an
EmbeddedAttemptSessionTakeoverError/ prompt-lock-release abort forced a full, un-stripped replay that finally sent the latent bad block. So the takeover (#86966) is the exposure trigger, not the create cause — this aligns with @CarlCapital's "already invalid at capture, not mutated at persist," and points the create-side at signature truncation/mis-assembly during streaming (cf. #87574/#87582), not an empty/zero-length path.Offer
Sanitized, content-stripped: the per-block table (signature length, base64-valid yes/no, API verdict) and the probe script. Happy to re-run against any specific replay shape you'd like to test (e.g. whether validation is position/context-sensitive — in our tests it was not; blocks validated standalone).
fix(agents): recover genericized Anthropic thinking errors (openclaw#…
fix(runtime): catch up openclaw upstream to d1b33a6 (openclaw#1733) (#99
fix(agents): recover genericized Anthropic thinking errors (openclaw#…
26 remaining items
vincentkoc commentedon Aug 11, 2026
Closing as fixed on current
main.The create-side root cause was fixed by #104043, merged as 3bcb90b. The Anthropic transport now keeps streamed
signature_deltabytes private and publishes them only after the matchingcontent_block_stop, so interrupted streams cannot persist partial signatures into replayable history.The recovery side is also covered by #92916, #95430, and #92286: nested/raw provider errors reach the matcher, Promise-resolved streams enter recovery, and a successful retry durably repairs the active transcript branch.
Exact-head CI and Real behavior proof passed. The live fault-proxy reproduction showed the old path persisting a truncated signature and failing the next request with HTTP 400; with the fix, the partial signature was omitted and subsequent same-session turns returned HTTP 200.
Release note: npm
latest(2026.7.1-2) does not contain the producer fix because that release line diverged earlier. npmbeta(2026.8.1-beta.1) and currentmaindo contain it. Stable users should use beta or wait for the next stable containing the fix.The open U+2026 persistence report #108443 and Bedrock-specific report #109881 remain separate investigations.
Thanks @CarlCapital for the original report and detailed evidence, and @joshavant, @mmyzwl, and @lzyyzznl for the producer and recovery repairs.