Skip to content

Embedded runner: freshly streamed thinking signatures intermittently invalid on replay (Anthropic); recovery wrapper never fires because error text is genericized #92201

Closed

Description

@CarlCapital

Title: Embedded runner: freshly streamed thinking signatures intermittently invalid on replay (Anthropic); recovery wrapper never fires because error text is genericized

Summary

On 2026.6.5, the embedded agent runner (Slack plugin) intermittently persists Anthropic thinking blocks whose signatures fail validation when history is replayed — permanently poisoning the session: every subsequent message fails with invalid_request_error: messages.N.content.0: Invalid signatureinthinking block, surfaced to the user as LLM request failed: provider rejected the request schema or tool payload.

Additionally, wrapAnthropicStreamWithRecovery — which exists precisely for this and would auto-heal by stripping thinking blocks and retrying — never fires on this path: zero [session-recovery] log lines across all incidents. The error reaching the recovery predicate appears to be the genericized "LLM request failed: provider rejected the request schema or tool payload." text (raw provider error attached separately as rawError= in logs), which THINKING_BLOCK_ERROR_PATTERN cannot match.

Environment

  • OpenClaw 2026.6.5 (ghcr.io/openclaw/openclaw:latest pulled 2026-06-10), embedded runner via Slack plugin
  • Provider: anthropic, model claude-sonnet-4-6, thinkingLevel high
  • First occurrence ever ~17h after upgrading the image; 3 separate poisonings of the same session within ~2h on 2026-06-11; at peak, every fresh thinking block in the affected thread was bad (one reply works → next message bricks)

Evidence isolating the layer

  1. Replay/persistence exonerated: extracted each stored thinkingSignature from the session jsonl and replayed each block individually against api.anthropic.com (same key/model): blocks generated earlier that day validate; blocks generated from 13:36 UTC onward fail. Stored signature and thinking text are byte-identical to the post-assembly trajectory log → nothing mutated after assembly.
  2. Anthropic exonerated: direct curl→API generate+replay with the same key/model, both non-streaming and streaming (manual thinking_delta/signature_delta concatenation), validated 4/4 while OpenClaw was concurrently producing invalid pairs.
  3. → corruption happens between the wire and the assembled message inside the runner's stream assembly. Note the build already contains the signature_delta append fix from fix(agents): concatenate signature_delta chunks in transport stream #87582 (block.thinkingSignature = (block.thinkingSignature || "") + delta.signature), so this is something else — possibly related to 2026.6.5's deferred stream-start / message_start changes.

Two requested fixes

  1. Find/fix the stream-assembly corruption (related: Bug: anthropic-transport-stream overwrites signature_delta instead of appending, causing permanent session corruption #87574, Compaction/pruning corrupts thinking blocks → Anthropic API rejection #30157, Anthropic thinking block 'signature' field lost during session persistence — causes API rejection #43691).
  2. Make shouldRecoverAnthropicThinkingError see the raw provider error (e.g. include rawError in the matched text, or match before genericizing) so sessions self-heal as designed even when assembly produces a bad block.

Workaround we applied

Force dropReasoningFromHistory for anthropic in the embedded runner (build-time patch) — past thinking is never replayed, so a bad stored signature is harmless; thinking generation unaffected. Happy to provide session jsonl excerpts (sans content) / probe scripts.

Activity

  1. clawsweeper commented on Jun 11, 2026

    @clawsweeper
    Contributor

    Thanks for the context here. I did a careful shell check against current main, and this is already implemented.

    Current main prevents partial streamed Anthropic signatures from entering history and recovers a pre-output replay rejection by retrying without thinking blocks, so the reported permanent session-brick behavior is addressed.

    Possible regression — probable (reproduction; failure trace). No predecessor PR is attributed.

    So I’m closing this as already implemented rather than keeping a duplicate issue open.

    Review details

    Best possible solution:

    Keep the staged-signature and durable retry design; file any newly observed malformed-signature mechanism with a raw stream-to-transcript trace, since #108443 appears to concern a distinct persistence path.

    Do we have a high-confidence way to reproduce the issue?

    No: current main’s stream and recovery tests cover the reported failure boundaries, and source now prevents or repairs the permanent replay failure.

    Is this the best way to solve the issue?

    Yes: staging opaque signature deltas at the transport boundary plus a single durable pre-output recovery retry fixes the producer and the operator-visible failure without a new setting.

    Security review:

    Security review: This is issue triage, not a proposed patch review.

    AGENTS.md: found and applied where relevant.

    What I checked:

    Likely related people:

    • Josh Avant: Authored the current prevention commit that stages signature deltas until content-block completion. (role: introduced stream-signature staging; confidence: high; commits: 3bcb90b2d08f; files: packages/ai/src/transports/anthropic-transport-stream.ts)
    • mmyzwl: Authored the merged recovery work that recognizes wrapped Anthropic signature errors. (role: recovery error-carrier contributor; confidence: high; commits: fbc3fa387637; files: src/agents/embedded-agent-runner/thinking.ts)
    • lzyyzznl: Authored the merged async-stream recovery work that pumps resolved streams through the retry logic. (role: async-stream recovery contributor; confidence: high; commits: 8c54704b77f2; files: src/agents/embedded-agent-runner/thinking.ts)

    Codex review notes: model internal, reasoning high; reviewed against c1217886119c; fix evidence: commit 3bcb90b2d08f, main fix timestamp 2026-07-10T20:54:37-05:00.

  2. davidcittadini commented on Jun 14, 2026

    @davidcittadini

    Independent live repro on 2026.6.6 — Telegram embedded runner, Opus 4.8 (not Slack, not Sonnet) — and confirms the genericized-error → recovery-wrapper-never-fires diagnosis

    We hit this exact failure in production today and it reproduces on a newer build than the original report, on a different runner and model, so the bug is in the embedded-runner core, not the Slack plugin or a Sonnet-specific path.

    Environment

    • OpenClaw 2026.6.6 (8c802aa) — so this is not fixed in .6 (report was 2026.6.5).
    • Telegram embedded runner (original report: Slack plugin).
    • Provider anthropic, model claude-opus-4-8, thinkingLevel: high (original report: claude-sonnet-4-6).
    • Long-lived direct session that had accumulated ~100 turns / ~37 thinking blocks.

    Trigger (matches "intermittently persists on replay")

    The poisoning was immediately preceded by a session-takeover/abort during a stalled, queued run — not compaction and not /new:

    07:06:40 [provider-transport-fetch] [model-fetch] error provider=anthropic api=anthropic-messages
             model=claude-opus-4-8 name=AbortError message=This operation was aborted
    07:07:40 [agent/embedded] embedded attempt cleanup detected session takeover after prompt failure;
             preserving prompt error: promptError=This operation was aborted
             cleanupError=session file changed while embedded prompt lock was released: .../sessions/<sid>.jsonl
    07:07:40 [diagnostic] lane task error: lane=session:agent:agent-B:telegram:direct:<chat-id>
             durationMs=1560084 error="EmbeddedAttemptSessionTakeoverError: This operation was aborted"
    07:07:40 [diagnostic] lane wait exceeded: waitedMs=28230 queueAhead=0 activeAhead=1 activeNow=0 queueBehind=0
    

    A ~26-minute stalled model_call with a second prompt queued behind active work (activeAhead=1) → the prompt lock was released and the session file mutated concurrently → EmbeddedAttemptSessionTakeoverError. The post-takeover replay then persisted/sent thinking blocks whose signatures don't validate.

    Result — permanent poisoning, identical on every retry

    07:07:41 [telegram] embedded run agent end: isError=true model=claude-opus-4-8 provider=anthropic
      error=LLM request failed: provider rejected the request schema or tool payload.
      rawError={"type":"error","error":{"type":"invalid_request_error",
      "message":"messages.1.content.87: Invalid `signature` in `thinking` block"},"request_id":"sha256:7d9e52f3373f"}
    07:07:51 ... request_id sha256:30146a321d92   (identical)
    07:08:15 ... request_id sha256:a2d89791783a   (identical)
    

    Confirms the core hypothesis of this issue: recovery wrapper never fires

    • Zero session-recovery log lines across the entire day's gateway logs (grep -c session-recovery = 0 in both gateway.log and gateway.err.log). wrapAnthropicStreamWithRecovery did not engage.
    • The error reaching the predicate is the genericized LLM request failed: provider rejected the request schema or tool payload. — the real invalid_request_error … Invalid signature in thinking block is only present as a separate rawError= field. So THINKING_BLOCK_ERROR_PATTERN has nothing to match. This is the same genericization called out in Filter noisy invalid_request_error subtypes (e.g. thinking-signature) out of user-facing assistant text #87475, and here it directly defeats the auto-heal, leaving the session permanently bricked.

    Session-file forensics — the persisted signatures were empty, not stale

    Inspecting the on-disk .jsonl: 37 thinking blocks, every one with a zero-length signature (not a stale/mismatched value). The reassembled request merged the assistant turn so the offending block landed at messages.1.content.87. This points at the takeover/replay path persisting freshly-streamed thinking blocks without their signature — distinct from #90108 (compaction re-injecting stale signatures) and #90483 (delete on shared object refs), though the user-visible symptom is identical. (Possibly the same underlying signature_delta overwrite-vs-append bug as #87574/#87582.)

    Manual remediation that worked (= the wrapper's intended action, done by hand)

    Stripping all thinking blocks from the session .jsonl (preserving text + tool_use + tool_result) immediately unblocked the session on the next turn. This confirms the fix shape:

    1. On the takeover/abort replay path, run stripInvalidThinkingSignatures / dropThinkingBlocks before persisting (and never persist a thinking block with a blank signature).
    2. Make the recovery predicate match on the structured rawError (invalid_request_error + signature/thinking), not the genericized user-facing text — otherwise wrapAnthropicStreamWithRecovery can never trigger on the embedded-runner path (ref Filter noisy invalid_request_error subtypes (e.g. thinking-signature) out of user-facing assistant text #87475).

    Happy to provide the full sanitized .jsonl, additional log windows, or the before/after block counts if useful.

  3. davidcittadini commented on Jun 14, 2026

    @davidcittadini

    Cross-link / upstream root cause: this signature poisoning is a downstream effect of the EmbeddedAttemptSessionTakeoverError prompt-lock-release race — same incident, same sessionId=d27c050e…. The aborted/taken-over turn is what persisted the thinking blocks without valid signatures here. Full repro steps, frequency (186 events, 2–18×/day across 3 agents), and paired-lane log evidence are documented at #86966. Fixing the takeover race upstream (serialize same-session inbound instead of aborting) would prevent this class of poisoning; stripping invalid thinking signatures on the abort/replay path is the belt-and-suspenders catch.

  4. davidcittadini commented on Jun 14, 2026

    @davidcittadini

    Triage escalation — this issue is OPEN but unlabeled/unassigned, and the automated review never completed

    Flagging because this looks at risk of being silently dropped:

    • This issue has no labels, no assignee, no milestone, yet it is a permanent session-poisoning bug — every subsequent turn on the affected session hard-fails with invalid_request_error: Invalid \signature` in `thinking` blockuntil someone manually edits the session.jsonl`. That is total loss of a live session, not a transient hiccup.
    • The only automated pass here (ClawSweeper Codex review, 2026-06-11) failed before triaging — retryable codex transport failure (exit 1) — so it was never labeled or work-laned. It needs a human pass.
    • Severity in the field: reproduced across 3 separate agents on one host, 2–18×/day since 2026-05-29, and confirmed on the latest build 2026.6.6 (8c802aa) with Telegram + claude-opus-4-8 — i.e. not Slack- or Sonnet-specific (original report was Slack/Sonnet on 2026.6.5). It is core embedded-runner, still live in .6.

    Suggested labels: impact:session-state (+ impact:message-loss), and P0/P1 to match #86966/#88838 in the same cluster.

    Cluster map (so the relationship is unambiguous)

    The single highest-leverage fix (cheap, decoupled, self-healing)

    The runner already ships wrapAnthropicStreamWithRecovery / shouldRecoverAnthropicThinkingError, which exist precisely to strip bad thinking blocks and retry — but on this path it never fires (zero [session-recovery] log lines). Reason: the error reaching the recovery predicate is the genericized "LLM request failed: provider rejected the request schema or tool payload.", while the raw provider error (Invalid \signature` in `thinking` block, attached separately as rawError=) is what THINKING_BLOCK_ERROR_PATTERN` needs to match.

    Two decoupled mitigations, neither blocked on #88838:

    1. Make shouldRecoverAnthropicThinkingError see the raw provider error (match rawError / match before genericizing). This alone would let every currently-poisoned session self-heal as designed — the biggest bang for the buck.
    2. Run the persisted transcript through stripInvalidThinkingSignatures on the abort/replay path so an unavoidable takeover/abort doesn't poison the session in the first place (belt-and-suspenders, ties back to the [Bug]: EmbeddedAttemptSessionTakeoverError: paired lanes on same session race after prompt-lock release #86966 race).

    Happy to provide sanitized session .jsonl excerpts, the per-agent daily histogram, and paired-lane samples across the three affected agents. This is causing daily, user-visible, unrecoverable-without-manual-surgery failures — would appreciate a prioritization decision.

  5. added 2 commits that reference this issue on Jun 14, 2026
    7ce5900
    fbc3fa3
  6. steipete commented on Jun 14, 2026

    @steipete
    Contributor

    The recovery half of this report landed in #92916 via fbc3fa3.

    OpenClaw now detects thinking-signature errors carried through Anthropic SDK error bodies, failover rawError, nested causes, direct messages, and terminal stream errors. The traversal is cycle-safe, does not inspect assistant text or tool arguments, and retries at most once without thinking blocks.

    Proof included a live Sonnet 4.6 replay with a real signed thinking block truncated to a non-empty invalid signature: Anthropic rejected the first request, recovery retried exactly once, the transcript-repair callback fired, and the final response was RECOVERED_OK. Official OpenAI and Anthropic tool-call live tests also passed 9/9, and full PR CI was green.

    Keeping this issue open: the intermittent path that first creates or persists invalid/empty signatures is not identified or fixed by #92916. The remaining useful evidence is a raw stream/transcript transition showing where a freshly valid signature becomes invalid before replay.

  7. CarlCapital commented on Jun 14, 2026

    @CarlCapital
    Author

    Original reporter (#92201) here — thanks @steipete, the recovery-side fix in #92916 matches our diagnosis exactly.

    One data point for the still-open create-side: our original incident was the non-empty-but-invalid variant, not the zero-length one @davidcittadini saw.

    Caveat: our incident-window gateway logs have since rotated, so I can't confirm whether a session-takeover/abort race was involved in our case specifically.

    Happy to share a sanitized jsonl (signature lengths + per-block validate pass/fail, content stripped) and the per-block API replay probe if useful.

  8. davidcittadini commented on Jun 14, 2026

    @davidcittadini

    Create-side evidence: the poison is a single thinking block whose signature is truncated/malformed at capture — confirmed against the live Anthropic API, and offline-detectable

    Correcting our own earlier framing first (we implied "empty signatures"): that was wrong, and so is "all blocks affected." After probing the persisted blocks directly and replaying every one against the live Anthropic API, the real picture is narrow and clean.

    Setup

    One bricked production session (OpenClaw 2026.6.6, claude-opus-4-8, adaptive thinking, Telegram embedded runner), 103 messages, 37 persisted thinking blocks. Each block on disk is {"type":"thinking","thinking":"", "thinkingSignature":"<…>"}.

    Note: the empty thinking plaintext is normal, not corruption — a freshly generated claude-opus-4-8 adaptive-thinking block is also empty-plaintext-plus-signature (reasoning is carried in the signed blob). So "empty content" is a red herring.

    Live API probe — 36 valid, exactly 1 invalid

    Replaying each stored block individually against POST /v1/messages (same model, thinking enabled):

    case result
    fresh real block, replayed intact ACCEPT (control)
    fresh block, signature truncated by 200 chars REJECT Invalid signature in thinking block (control)
    37 stored blocks from the incident 36 ACCEPT, 1 REJECT

    The single rejected block is Invalid \signature` in `thinking` block` — the exact error that bricked the session. The other 36 replay fine today, so they were never the problem.

    The bad block is structurally truncated (offline-detectable)

    The one rejected signature is malformed base64: it has no trailing padding and fails strict base64 decode, whereas all 36 valid siblings end with a proper trailer + padding (…MGAE=, …AQ==) and decode cleanly. A plain base64-validity check over the 37 signatures flags exactly one block — the same one the API rejects (1:1). So at least this variant is catchable without a provider round-trip, e.g. as a persist-time or pre-send guard.

    Origin: invalid at capture, during a normal turn — not the takeover

    The bad block was written at 06:53:09 during an ordinary streamed turn (msg 49 of 103). 16 later thinking blocks (through 07:03) are all valid. The session kept working for ~13 min because OpenClaw strips thinking from history on normal sends; it only died at 07:07 when an EmbeddedAttemptSessionTakeoverError / prompt-lock-release abort forced a full, un-stripped replay that finally sent the latent bad block. So the takeover (#86966) is the exposure trigger, not the create cause — this aligns with @CarlCapital's "already invalid at capture, not mutated at persist," and points the create-side at signature truncation/mis-assembly during streaming (cf. #87574/#87582), not an empty/zero-length path.

    Offer

    Sanitized, content-stripped: the per-block table (signature length, base64-valid yes/no, API verdict) and the probe script. Happy to re-run against any specific replay shape you'd like to test (e.g. whether validation is position/context-sensitive — in our tests it was not; blocks validated standalone).

  9. 26 remaining items

  10. added
    issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.
    clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.
    and removed
    clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.
    issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.
    on Aug 1, 2026
  11. added
    issue-rating: 🦪 silver shellfishThin issue quality; more reproduction proof or environment detail is needed.
    clawsweeper:not-repro-on-mainClawSweeper found high-confidence evidence that this issue no longer reproduces on main.
    and removed
    clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.
    clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.
    clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.
    issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.
    on Aug 9, 2026
  12. self-assigned this
    on Aug 11, 2026
  13. vincentkoc commented on Aug 11, 2026

    @vincentkoc
    Member

    Closing as fixed on current main.

    The create-side root cause was fixed by #104043, merged as 3bcb90b. The Anthropic transport now keeps streamed signature_delta bytes private and publishes them only after the matching content_block_stop, so interrupted streams cannot persist partial signatures into replayable history.

    The recovery side is also covered by #92916, #95430, and #92286: nested/raw provider errors reach the matcher, Promise-resolved streams enter recovery, and a successful retry durably repairs the active transcript branch.

    Exact-head CI and Real behavior proof passed. The live fault-proxy reproduction showed the old path persisting a truncated signature and failing the next request with HTTP 400; with the fix, the partial signature was omitted and subsequent same-session turns returned HTTP 200.

    Release note: npm latest (2026.7.1-2) does not contain the producer fix because that release line diverged earlier. npm beta (2026.8.1-beta.1) and current main do contain it. Stable users should use beta or wait for the next stable containing the fix.

    The open U+2026 persistence report #108443 and Bedrock-specific report #109881 remain separate investigations.

    Thanks @CarlCapital for the original report and detailed evidence, and @joshavant, @mmyzwl, and @lzyyzznl for the producer and recovery repairs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:not-repro-on-mainClawSweeper found high-confidence evidence that this issue no longer reproduces on main.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦪 silver shellfishThin issue quality; more reproduction proof or environment detail is needed.

Type

No type

Fields

Priority

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions