fix(message): guard ProviderData against empty RawMessage entries

[andybons]

Summary

Forensics round 2 on the recurring MarshalJSON … unexpected end of JSON input incident. The actual offending field is message.ProviderData (map[string]json.RawMessage) on Reasoning parts — one indirection away from #42's ToolCall.Arguments guard, marshaled unguarded on every persist/journal/hook encode and read by direct map indexing in all three transcoders. A present-but-zero-length provider-data entry reproduces the incident error exactly (red-verified by reverting the fix); goal-supervised sessions hit it repeatedly because they run with reasoning enabled and replay provider data same-family.

Guard applied at the type level mirroring #42's pattern, with the full worker-turn path (transcode + persist) covered by the incident-shaped test, not just the encoder.

Why #42 missed it: its repro matched the error string, not the site — the lesson (forensic fixes must re-run the failing workload shape) goes to AGENTS.md in the next docs pass.

Box forensics session ses_01kx3se8wqexatnesyxmgj5b0q, box-pushed.

🤖 Generated with Claude Code

[claude]

Claude finished @andybons's task in 4m 10s —— View job

---

PR Review: guard ProviderData against empty RawMessage entries

• Read AGENTS.md and gather context
• Review the ProviderData guard implementation
• Review transcoder changes (anthropic, openai, openaicompat)
• Review test coverage
• Post review feedback

---

Verdict: LGTM. This correctly root-causes the recurring MarshalJSON … unexpected end of JSON input incident one indirection past #42, and fixes it at the right layer (the message type) so every producer is protected, not just the shipped adapters. I built (go build ./...) and ran the affected packages (message, provider/...) green.

What it does

Three coordinated guards on message.ProviderData (map[string]json.RawMessage), mirroring #42's ToolCall.safeArguments/MarshalJSON pattern:

• Get(family) — single read choke point; a present-but-zero-length entry reports absent. All three transcoders switch from direct map indexing to Get.
• MarshalJSON — drops zero-length entries from any direct encode (session log, server journal, plugin hook payload), so no encoder can trip the json.RawMessage footgun.
• Normalize (called in Session.append) — deletes zero-length entries in place at ingest, so an in-memory message remarshals byte-identically to what LoadSession reloads.

Verification I did on the trickier claims

• omitempty round-trip equality holds. After Normalize, a map that held only an empty entry is len 0 → omitempty drops the field; the reloaded side has no field → nil map → also dropped. A map with a real entry + an empty one keeps the real entry on both sides. The historyJSON(loaded) == historyJSON(s) assertions are sound.
• The outside-the-lock Normalize in Session.append is not a race. For the assistant path (engine.go:445-446), s.append(*asst) fully completes Normalize (mutating the shared *Reasoning map) before s.emit(... Message: asst ...) hands the same pointer to subscribers — sequential on one goroutine, so there's a happens-before. No subscriber ever reads the map concurrently with the delete.
• The openai transcoder fix closes a real latent bug, not just the crash: previously append(json.RawMessage(nil), raw...) normalized an empty entry to nil, which marshals as a spurious null item in the request input list. The test (transcode_test.go) documents and guards exactly this.

Notes (all minor / optional — nothing blocking)

• openaicompat has no dedicated regression test (anthropic + openai each got one). This is fine here: that branch continues and drops Reasoning regardless of Get's result (transcode.go:213-222), and its own stream never populates ProviderData, so the Get swap is a behavioral no-op kept for consistency. Worth a one-line note in the PR that it's intentionally untested rather than an oversight.
• MarshalJSON allocates a fresh out map on every encode even in the common all-non-empty case. Negligible (cold persist path, tiny map), but a len==0 / no-empty-entries fast path would avoid it if you want the persist path allocation-free.
• Doc quality is excellent — the ProviderData and Normalize comments capture why fix(message): normalize empty json.RawMessage tool-call arguments #42 missed this and the omitempty-before-MarshalJSON subtlety precisely. This is the kind of comment that prevents round 3.

Consistent with AGENTS.md: TDD (incident-shaped tests at message, transcoder, persist, and goal-loop levels), red-verification called out, no raw sleeps, golden round-trip comparisons. Good work.
• fix/rawmessage-systematic