[BUG] API Error 400 "no low surrogate in string" when Bash output contains invalid Unicode

[coygeek]

Preflight Checklist

• I have searched existing issues and this hasn't been reported yet
• This is a single bug report (please file separate reports for different bugs)
• I am using the latest version of Claude Code

What's Wrong?

When running commands that produce output with invalid Unicode characters (specifically unpaired surrogate pairs), Claude Code crashes with an API error. The Bash tool successfully captures the output, but when the next API request is serialized to JSON, the invalid Unicode causes JSON encoding to fail.

The error occurs because JSON requires valid Unicode, and unpaired surrogates (U+D800-U+DBFF without corresponding U+DC00-U+DFFF) are not valid.

This commonly happens with:

• Mutation testing tools (mutmut, Stryker) that produce progress bars/terminal graphics
• Tools that emit partial terminal escape sequences
• Commands that capture binary data or corrupted text

What Should Happen?

Claude Code should sanitize Bash output to ensure valid Unicode before:

1. Adding the output to the conversation context, OR
2. Serializing the API request payload

Invalid Unicode characters should be replaced with the Unicode replacement character (U+FFFD) or removed entirely. This is standard practice for handling untrusted text input.

Error Messages/Logs

API Error: 400 {"type":"error","error":{"type":"invalid_request_error","message":"The request body is not valid JSON: no low surrogate in string: line 1 column 280696 (char 280695)"},"request_id":"req_011CWo9ijhAZvXz786YjeMai"}


The error position (280696) indicates the invalid character is somewhere in the full request body, not necessarily at the end of the Bash output.

Steps to Reproduce

1. Run a command that produces output with terminal control sequences or binary data:

   source .venv/bin/activate && mutmut run --max-children 4 2>&1 | tail -30

2. The Bash command completes successfully, producing output (in my case: 30,029 characters / ~14,608 tokens)

3. On the next API call, Claude Code crashes with the "no low surrogate in string" JSON error

Why mutmut triggers this:

• mutmut uses terminal escape sequences for progress indicators
• Some sequences may be partially captured or corrupted
• The | tail -30 captures output that may include incomplete escape sequences

Minimal reproduction (if you can create invalid Unicode):

# This would need a command that outputs actual invalid Unicode
# The key is any output containing bytes like 0xED 0xA0 0x80 (unpaired high surrogate)

Claude Model

None

Is this a regression?

I don't know

Last Working Version

No response

Claude Code Version

2.0.76

Platform

Anthropic API

Operating System

macOS

Terminal/Shell

Terminal.app (macOS)

Additional Information

Root Cause Analysis

The issue is that Bash tool output is not sanitized for valid Unicode before being included in the API request. While the Bash tool has a 30KB size limit (BASH_MAX_OUTPUT_LENGTH), there's no character validity check.

Suggested Fix

Add Unicode sanitization in one of these locations:

1. Bash tool result processing (before adding to context):

   function sanitizeUnicode(text) {
     // Replace unpaired surrogates with replacement character
     return text.replace(/[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/g, '\uFFFD');
   }

2. API request serialization (global safety net):

   • Sanitize the entire request body before JSON.stringify()
   • This would catch issues from any source, not just Bash

Workaround

Users can work around this by using hooks to sanitize Bash output (PostToolUse on Bash), but this should be handled by Claude Code itself.

Related Context

• The Bash tool already has size limiting (30KB)
• MCP tools have token limiting (25K tokens)
• Task/TaskOutput has no built-in limiting (addressed by user hooks)
• None of these tools sanitize for valid Unicode

[github-actions]

Found 3 possible duplicate issues:

1. [BUG] Session broken -> API Error: 400 {"type":"error","error":{"type":"invalid_request_error","message":"The request body is not valid JSON: no low surrogate in string: line 1 column 410559 (char 410558)"}} #1709
2. [BUG] Invalid Unicode in Edit tool results causes 400 'invalid JSON' API errors #13157
3. Claude Code crashes on tool output containing invalid Unicode surrogates #15027

This issue will be automatically closed as a duplicate in 3 days.

• If your issue is a duplicate, please close it and 👍 the existing issue instead
• To prevent auto-closure, add a comment or 👎 this comment

🤖 Generated with Claude Code

[coygeek]

Follow-up: Hook-based workaround is not possible

I attempted to implement a workaround using Claude Code hooks, but discovered an architectural limitation that makes this impossible to fix at the hook level.

What I Tried

Added a sanitize_unicode() function to my PostToolUse hook (output_validator.py) that:

• Detects unpaired surrogates via text.encode("utf-8")
• Replaces invalid characters with errors="replace"

Why It Doesn't Work

PostToolUse hooks cannot modify tool_result.

Hook Type	Available Output Fields	Can Modify Output?
PreToolUse	updatedInput, permissionDecision	Yes (command input)
PostToolUse	additionalContext, systemMessage	No (read-only)

The hook schema explicitly defines:

• PreToolUse: Has updatedInput to modify the command before execution
• PostToolUse: Only has additionalContext to add commentary—no updatedResult equivalent

My sanitization runs, but the original corrupted tool_result is still serialized and sent to the API. The hook's sanitized copy is discarded.

Confirmed Root Cause Location

The sanitization must happen in Claude Code core, specifically:

1. After the Bash tool captures stdout/stderr
2. Before the tool_result is added to the conversation context

There is no hook point between these two steps that allows modification.

Recommendation

Add Unicode sanitization in the Bash tool's output processing, similar to how the tool already has size limiting (30KB BASH_MAX_OUTPUT_LENGTH):

// Pseudocode for where the fix should go
function processBashOutput(output) {
  const truncated = truncateToMaxLength(output);  // Existing
  const sanitized = sanitizeUnicode(truncated);   // NEW
  return sanitized;
}

This would catch invalid Unicode from any source (mutmut, binary data, corrupted files) before it can break JSON serialization.

[coygeek]

GitHub Issue #16294 Follow-up: Workaround Implemented

Issue: Invalid Unicode in Bash Output Causes API Errors

Status: Workaround implemented until Anthropic fixes the core issue

---

The Problem

Running mutmut (mutation testing tool) via Claude Code crashes with:

API Error: 400 {"type":"error","error":{"type":"invalid_request_error","message":"The request body is not valid JSON: no low surrogate in string..."}}

Root cause: mutmut's terminal output contains invalid Unicode characters (unpaired UTF-16 surrogates from ANSI escape sequences and progress bars). When Claude Code serializes the bash output to JSON for the API, these invalid surrogates cause JSON encoding to fail.

Why hooks don't help: PostToolUse hooks can only add additionalContext—they cannot modify the original tool_result. The corrupted output is still serialized and sent to the API.

---

The Workaround

Created scripts/mutmut_safe.sh — a wrapper script that intercepts mutmut output before Claude sees it.

How It Works

1. Output Redirection: Instead of letting mutmut write directly to stdout (which Claude captures), the wrapper redirects all output to a temp file (var/logs/mutmut-output.log)

2. Sanitization Pipeline: After mutmut completes, the output is piped through sanitization filters:

   • Strip ANSI escape sequences (\x1b[...m patterns)
   • Remove all non-printable/control characters
   • Limit output size to prevent token bloat

3. Safe Return: Only the sanitized output is displayed to stdout, which Claude then captures. The corrupted bytes never reach Claude's API serialization.

Files

File	Purpose
scripts/mutmut_safe.sh	Wrapper script
var/logs/mutmut-output.log	Raw output (may contain invalid chars)
var/logs/mutmut-safe.log	Sanitized output (safe for Claude)

Usage

# Safe commands (via Claude Code)
make mutation              # Run mutation tests
make mutation-results      # Show results
make mutation-show ID=123  # Show specific mutant

# Direct mutmut (separate terminal only, NOT via Claude)
make mutation-unsafe

---

Why This Works

The key insight is that Claude Code's crash happens during JSON serialization of tool output, not during command execution. By ensuring only valid UTF-8 reaches stdout, we prevent the serialization failure entirely.

The wrapper acts as a "Unicode firewall" between mutmut and Claude Code.

---

Proper Fix (for Anthropic)

The correct solution is for Claude Code to sanitize bash output before JSON serialization, similar to how it already handles output size limits (30KB BASH_MAX_OUTPUT_LENGTH). A simple regex replacement of unpaired surrogates with the Unicode replacement character (U+FFFD) would prevent this class of errors entirely.

Until that fix ships, this wrapper provides a reliable workaround for mutation testing workflows.

[konard]

Additional data point and workaround

We encountered this same error while using Claude Code programmatically via @link-assistant/hive-mind (an AI issue solver that orchestrates Claude Code sessions).

Our case

• Claude Code version: Latest (via claude CLI)
• Model: claude-opus-4-5-20251101
• Context: Claude was analyzing Rust source code and log files from a previous solve session
• Error: "no low surrogate in string: line 1 column 160309 (char 160308)"
• Session: 21 turns, ~$1.03 spent before the crash
• Full log: solution-draft-log-pr-1769796128836.txt

Root cause analysis

The issue is that JSON.stringify() in JavaScript does NOT reject lone surrogates — it produces technically invalid JSON that JavaScript's own JSON.parse() accepts (lenient parser), but Anthropic's server-side parser (likely Rust/Python) correctly rejects per RFC 8259 Section 8.1.

Common sources of lone surrogates:

1. Binary file content interpreted as text (tool reads of compiled output, .af files, etc.)
2. Log files with mixed encodings
3. Terminal escape sequences from mutation testing tools, progress bars, etc.

Suggested fix

Add a surrogate sanitization step in the tool result processing pipeline, before the result is added to the conversation history. Here's a minimal implementation:

const sanitizeSurrogates = str => {
  if (typeof str !== 'string') return str;
  return str.replace(
    /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/g,
    '\uFFFD'
  );
};

This regex:

• Matches high surrogates (U+D800-U+DBFF) NOT followed by a low surrogate
• Matches low surrogates (U+DC00-U+DFFF) NOT preceded by a high surrogate
• Replaces them with U+FFFD (Unicode replacement character)
• Preserves valid surrogate pairs (used for emoji and supplementary characters)

Our workaround

We implemented this sanitization in our wrapper (hive-mind) at two points:

1. Prompts sent to the Claude CLI are sanitized before being passed
2. safeJsonStringify() in our interactive mode sanitizes string values during JSON serialization

This doesn't fix the core issue (tool output corruption within Claude Code's internal conversation history), but it prevents the prompts we control from triggering the error.

Detailed case study: https://github.com/link-assistant/hive-mind/blob/main/docs/case-studies/issue-1204/README.md
Related issue: link-assistant/hive-mind#1204

[edgeq]

Can confirm this is still an issue - especially when using Playwright MCP to get console logs and snapshots. Might try to use this sanitize function. Or just strip emojis out of all our console logs. Who knew that emojis are Human-readable but Not Claude-friendly. 🤷🏻‍♂️