[Bug] Malformed tool-call JSON via NVIDIA NIM (GLM-5) causes parser failures in OpenCode

[chindris-mihai-alexandru]

Summary

When using z-ai/glm5 through NVIDIA NIM in OpenCode, tool calls intermittently contain malformed JSON arguments (missing/truncated braces), causing repeated parse failures and retry loops.

Example error

Invalid input for tool tavily_tavily_search: JSON parsing failed: Text: {\"query\":\"Geoffrey Huntley LinkedIn Ralph Wiggum most viral reposts\".
Error message: JSON Parse error: Expected '}'

Also observed for multiple tools in the same session (tavily_tavily_search, brave-search_brave_web_search, webfetch).

Environment

• Client: OpenCode 1.2.6
• Model: z-ai/glm5
• Provider: NVIDIA NIM (https://integrate.api.nvidia.com/v1)
• OS: macOS Sequoia 15.7.4

Reproduction

1. Configure OpenCode with z-ai/glm5 via NVIDIA NIM.
2. Ask for a web research task that triggers multiple tool calls.
3. Observe occasional malformed tool JSON payloads and repeated retries.

Expected

Tool-call arguments are valid strict JSON.

Actual

Occasional malformed JSON payloads (e.g., missing closing }), causing parser failures and agent loops.

Notes

• It is unclear whether this is model output formatting, streaming/chunking interaction, or client-side parsing/recovery.
• Related OpenCode issue: [BUG] GLM-5 via NVIDIA NIM emits malformed MCP tool JSON (Expected '}') in OpenCode v1.2.6 anomalyco/opencode#13900
• Potentially related upstream report in vLLM: [Bug]: GLM-5-FP8 malformed tool calls vllm-project/vllm#34449

If there is a preferred GLM-5 parser/config for strict tool-calling compatibility via OpenAI-compatible endpoints, please share it.

[chindris-mihai-alexandru]

Update: corresponding report filed on NVIDIA Developer Forums (NIM) with the same symptom and repro context:
https://forums.developer.nvidia.com/t/nim-glm-5-malformed-tool-call-json-missing-via-openai-compatible-endpoint-opencode/360809

Cross-linking for maintainers tracking GLM-5 + NIM tool-calling behavior.

[chindris-mihai-alexandru]

Added minimal reproducible prompt + exact malformed payloads gist: https://gist.github.com/chindris-mihai-alexandru/f3c114915f18618406030e6a2d8fa59b

[QwertyJack]

Hi there! Could you please share the output SSE stream?

[regismesquita]

@chindris-mihai-alexandru Are you running it with MTP? I remember having a lot of tool calling issues when using MTP.

[MoshiLL]

I encountered the same issue when using the vLLM GLM-5 patched Docker image in Claude Code. The error in Claude Code was "invalid tool parameters". After removing the two MTP-related command arguments --speculative-config.method mtp and --speculative-config.num_speculative_tokens 1, the error disappeared.

I suspect this is a compatibility bug between the vLLM inference engine and the GLM-5 model when the MTP (Multi-Token Prediction) speculative sampling feature is enabled. vLLM's --speculative-config.method mtp feature is designed to accelerate generation, but it malfunctions when processing structured outputs from the GLM-5 model (particularly the JSON format for Tool Calls). MTP causes the generated JSON tokens to become malformed or duplicated. Downstream programs (whether your client or Agent framework) cannot parse these corrupted JSON payloads, resulting in "Invalid tool parameters" errors or parsing failures.

I'm not sure if the latest version of vLLM has fixed this issue. However, based on my experience, the latest vLLM v0.17 version cannot properly deploy GLM-5 due to transformers version constraints (limited to below 5).

[romancone]

I got the same error message when working with the Litellm proxy over Nvidia NIM, so I found this issue.
I swapped the proxy to free-claude-code and this error has gone

I believe this is not a model issue, but a processing tool call on the client side

[zhulianhua]

I got similar error: Error message: JSON Parse error: Expected '}']

• vllm 0.19.0 run with --speculative-config.method mtp and --speculative-config.num_speculative_tokens 3
• GLM-5-FP8 (on-premise deployed)
• OpenCode 1.4.3

Then I switched to GLM 5.1 FP8 still got the json parsing issue.

Then I removed the vllm arguments --speculative-config.method mtp and --speculative-config.num_speculative_tokens 3, and the issue disappeared.

[pridesong]

Replying to the original question about a preferred parser/config — I spent some time tracing this across the three serving paths (NIM, Z.AI API, vLLM local), and the malformed tool-call JSON does not seem to be one root cause. There are two independent defect chains, and they need different fixes.

Chain 1: vLLM MTP spec-decode (the duplicated/malformed-token pattern)

This one is already tracked upstream at vllm-project/vllm#34449. The ablation table in that thread is decisive: vLLM 0.15.0 + MTP + tool calling works, nightly + MTP breaks it, and it reproduces with GLM-4.7 as well — so it's a vLLM regression, not a GLM model issue. Looking at the code, the commit point in rejection_sampler.py only compares token IDs; there is no structural-token validation anywhere in the accept path, and an accepted draft plus the bonus token are committed atomically with no check in between. Once committed, the malformed sequence is already in the output stream and no downstream parser can repair it.

The practical takeaway: for tool-call workloads, just turn MTP off. The throughput numbers make this painless — on B200 FP8 the MTP-vs-off crossover is around 33.3 tok/s/user. Interactive tool-calling is well below that, so MTP is a net loss there anyway (about -28% at low concurrency). "Disable MTP" is not a compromise for this workload; it's the correct config, and it also happens to fix the JSON.

Chain 2: SGLang streaming serialization (the truncation/frame-embedding/unescaped pattern)

This one is in the open source. In python/sglang/srt/function_call/glm47_moe_detector.py, _process_xml_to_json_streaming passes object/array argument content through verbatim:


json_output += content # raw passthrough, no escaping


The string branch right next to it goes through json.dumps(content)[1:-1] and is fine. The object/array branch is not. A raw 0x0A inside a string value cuts the SSE frame at the next line break (truncation, or an embedded data: { when the rest re-surfaces), and a raw " closes the string early. That matches the truncated } samples in this issue and the "object":"chat.c / data: { pattern reported in #66. It also lines up with sgl-project/sglang#15721, where the same streaming path crashes on malformed input.

Worth noting: vLLM's own serialization chain (serving layer, Python parser, Rust parser) escapes correctly in 0.19.0 — so "vLLM is safe" does not imply "SGLang is safe." The two official serving paths behave differently here.

What I'd suggest, per path


• vLLM local: default --no-enable-mtp for tool-call traffic (or bisect 0.15.0→nightly and fix the spec-decode regression properly).
• SGLang: escape the object/array streaming branch the same way the string branch is handled — stateful escaping that only touches bytes inside string values and leaves structure tokens alone. I have a small patch and test cases ready if useful.
• Either path: a final json.loads on the completed frame, failing loudly instead of silently truncating, catches whatever leaks through. Client-side recovery (the OpenCode PR that tolerates malformed frames) is fine as a stopgap, but it swallows the evidence that upstream needs to fix the real defect — the more it absorbs, the harder this is to root-cause.
  
  This is all reproducible from public sources with zero GPUs. Happy to share the patch or the full analysis.