[Bug] DSML-like tool markup returned as assistant text instead of native tool calls interrupts long-running tasks (0.1.7-rc.2)

[J1nCw]

Summary

During a long-running task, tool-protocol markup appeared in ordinary assistant text instead of structured tool_calls. The intended tool operation was therefore not executed, and repeated occurrences interrupted task progress.

This report concerns tool-call handling and recovery. We have not established whether the initial malformed response originated in model generation, the upstream serving/parser layer, or an interaction with the client request. We are not claiming that context truncation or Harness compaction caused it.

Environment

• Harness: @deepseek-ai/dsh 0.1.7-rc.2, Linux Web deployment.
• OS: Ubuntu Linux; Node.js 24.21.0.
• Model identifier: deepseek-flash through a third-party, OpenAI-compatible chat-completions endpoint. Its backend implementation and actual context capacity have not been independently verified.
• Configured context window: 1,000,000 tokens; this is a client configuration value, not a verified provider capacity.
• Local custom plugins are present. A limited retry guard was used during later investigation, and a separate recovery prototype was tested only in a diagnostic session and log copies. This is not a clean stock-install reproduction; plugin influence on the original request/history has not been completely excluded.
• Not reproduced or validated against 0.2.0-rc.2 or the official DeepSeek API in this investigation.

Observations from the affected session

• The first detected occurrence followed hundreds of successful native tool calls, with approximately 749k input tokens reported by the API, including cached input.
• A scan of the preceding recorded message bodies, reasoning, and tool arguments found no earlier matching protocol markers. This is a result for this session, not a universal guarantee.
• A shell file-check error occurred immediately before the first malformed response. Temporal proximity alone does not establish a trigger.
• The first recorded compaction/prune event occurred after the first malformed response, so that later compaction event cannot explain the initial occurrence.
• Intended writes represented only by malformed assistant text had no corresponding native tool call/result. We found no evidence of a successful filesystem write subsequently disappearing.
• Native tool calls also succeeded later in the same session. This was intermittent, rather than a demonstrated permanent inability to execute any tool.

Diagnostic comparisons

These are limited observations, not a deterministic minimal reproduction or a success-rate benchmark. Returned tools in isolated probes were not executed.

Comparison	Result	Limitation
Replay of a later failing context through the same endpoint, streaming	Protocol markers appeared in raw API content without a native tool call	One reproduction; upstream fault boundary unresolved
Same later context, non-streaming	Markers still appeared; output limit reached	Disabling streaming was not a verified fix
Remove malformed assistant bodies from that later context	Two consecutive probes returned native write calls	Small sample; does not explain the first occurrence
Replay of the context immediately before the first occurrence, retaining system instructions and 26 tool definitions	HTTP 200; a native bash call with parseable JSON arguments and finish_reason=tool_calls; no malformed content markers	One later replay, with a 2,400-token output cap; not an identical server-state reproduction
Quarantine malformed messages in a copy using the built-in surface replacement mechanism	A subsequent probe returned a native write call with parseable arguments	Prototype recovery in a copy, not a deployed fix for the affected session

An isolated reproduction loaded the credential provider, LLM runtime, and Pi adapter, without the custom retry or Web-rendering plugins. Thus those plugins were not required to reproduce the later failure, but effects already present in the history and request definitions remain possible.

Expected behavior and recovery concern

Tool operations should arrive as structured calls and execute through the normal tool dispatch path. If the upstream response instead contains tool-protocol text, the task should expose a clear failure and support bounded recovery. Plain assistant text must not be silently interpreted and executed as a tool command.

The history-cleaning comparisons suggest that retaining malformed protocol text can contribute to recurrence after the first failure. They do not establish the cause of the initial malformed response.

Questions for maintainers/community

1. Is this a known compatibility issue with DeepSeek-style tool markup on third-party OpenAI-compatible endpoints?
2. Is there an existing supported way to detect this condition and quarantine malformed assistant bodies from subsequent model context while preserving the audit log?
3. What minimal, redacted response fields or synthetic reproducer would help distinguish provider parsing from Harness adapter/recovery behavior?

Related discussion

Discussion #2158 reports a similar visible symptom: tool markup in ordinary content. This report adds observations from 0.1.7-rc.2 on a third-party chat-completions route and limited history-isolation comparisons. The model/route differs, and a shared root cause has not been established.

Privacy and evidence limitations

This report is a manually summarized, redacted account of local diagnostic records. No raw prompts, task documents, tool arguments/results containing private data, full session logs, screenshots, credentials, endpoint URLs, institutional identifiers, hostnames, IP addresses, filesystem paths, or session IDs are attached. Exact private-task reproduction steps and raw response bodies are intentionally omitted. A deterministic synthetic reproducer remains unavailable.

中文概述：长任务中出现工具协议标记进入普通助手正文、未形成原生工具调用的间歇故障。原始 API 正文中也能看到异常，因此尚不能归因于 Harness 页面显示；首次问题的模型/服务端解析根因仍未确认。隔离后续异常历史的少量对照可恢复原生调用，但不构成普遍修复保证。希望社区协助确认兼容性与安全恢复机制。

[XuZhangxing]

Thanks for the detailed investigation. Your observation that the malformed DSML-like markup is already present in the raw API response is particularly interesting.

I previously reported a related issue in the DeepSeek-V3 repository:

deepseek-ai/DeepSeek-V3#1668

In that issue, I initially observed internal ds_safety, <thinking>, and DSML tool-call markup being exposed in the DeepSeek Web client when Web Search was disabled. I later extended the investigation to the API level.

In a larger API test, I made 400 successful /chat/completions requests with stream=true, without providing a structured tools parameter while describing tool capabilities in the system prompt:

• deepseek-flash: 187/200 requests (94%) contained at least one internal marker
• deepseek-v4-pro: 159/200 requests (80%) contained at least one internal marker

The leakage formats differed by model. Flash mainly produced DSML and ds_safety markers, while V4-Pro more often produced XML-style tool-call markup.

This does not prove that DeepSeek-V3 #1668 and this Harness #8509 discussion have the same root cause. However, there seems to be a similar boundary problem: internal tool/protocol representations sometimes appear to enter the ordinary content channel instead of remaining in the structured tool-call path.

Your observation that the malformed markup is already present in the raw API response is therefore particularly relevant to this comparison.

I would separate the investigation into two layers:

1. Upstream/model/API response: why an internal tool representation is emitted through ordinary content instead of a structured tool call.
2. Harness recovery: how Harness should detect such malformed responses and prevent them from contaminating subsequent context or repeatedly interrupting a long-running task.

For reference, the DeepSeek-V3 #1668 discussion contains the later API-level test results and cross-model comparison.

[J1nCw]

Follow-up (2026-10-03): partial mitigation with an unofficial response guard

We now have evidence of partial recovery and prevention using a local, unofficial guard. This is not an upstream fix, and the initial model/provider parsing cause remains unresolved. The operator reports an improvement; the observations below are the narrower, verifiable evidence.

Harness is still 0.1.7-rc.2. The current dynamic guard package is 1.0.1. An earlier session-scoped V3 guard remains a separate implementation; the historical live recovery below belongs to V3 and must not be attributed to 1.0.1.

Mechanism

1. Explicit, one-time history recovery: isolate malformed assistant bodies from future model context using built-in surface replacement, while preserving the original event log. This was applied to the affected session after the original report's diagnostic work. It is not automatically performed when the dynamic guard switch is enabled.
2. Check before history/dispatch: buffer a response, detect selected patterns of tool markup in ordinary text, and reject a detected leak as TOOL_CALL_TEXT_LEAK before it becomes a successful assistant message or dispatches tools. Never translate arbitrary prose into executable calls. Valid structured calls still use the normal permission and scheduling path.
3. Bounded recovery: allow at most two retries for this failure per turn; count failures from persisted attempts so plugin reload does not reset the budget. Persistent failure ends explicitly in an error.
4. Private evidence: retain rejected response chunks locally with restricted permissions. The 1.0.1 guard still rejects a response if diagnostic-file writing fails. Only aggregate observations are published here.

The detector deliberately excludes fenced/inline code and block quotes; it also skips messages containing native tool-call blocks. These heuristics have false-positive/false-negative limits. Buffering is capped at 16 MiB and can delay display of long replies. Protection is scoped to a configured provider and selected sessions; child sessions need separate enablement.

Detection and validation record

Dates below use UTC+08. Historical observations were rechecked on 2026-10-03; no fresh production model task was started for this update.

Evidence	Observation	Interpretation / limit
Original-session recovery, 2026-10-01	23 malformed historical replies isolated; 435 unaffected messages retained; 3,889 pre-migration events preserved	Local recovery record; does not explain the initial failure
Live consecutive-tool test after recovery, 2026-10-01	54 native calls: 26 write, 26 read, 2 bash; 54 results; 0 tool errors; 0 failed model attempts; 0 guard interventions; 2 completed turns	Same affected session; 25 test files independently checked for strict UTF-8, exact content and parsed JSON equality. No retries were needed in this test window
Later V3 runtime interception, 2026-10-01	Audit records one pre-history rejection and one retry. The corresponding turn contains one guard-failed attempt, followed by 38 native calls with 38 matching successful results, and ends completed	This is an existing runtime/session-log observation, distinct from the synthetic tests below. Result matching used call IDs internally; IDs are not published. One additional result event outside that set was excluded from the matched-call count
Current 1.0.1 deployment, checked 2026-10-03	Deployed package version and four core implementation files matched the locally reviewed hotfix; runtime audit contains 1.0.1 load records	Loading is not proof of protection in every session. The inspected 1.0.1 audit records contain no rejection/retry/exhaustion events, so there is no measured live interception success rate for this version
Host, client-state, and stream regression suites, rerun 2026-10-03	All three suites exited successfully	Covers state persistence/failure handling, client toggle synchronization, response detection/buffering and retry-budget cases. This is isolated testing, not a real browser-click acceptance test

The following three integrations were also rerun successfully on the server's Node.js 24.21.0 using the installed Harness AgentLoop, isolated sessions, a mocked model adapter, and an in-memory tool. They did not call the external model API or execute production tasks:

{
  "validation_date": "2026-10-03",
  "harness_version": "0.1.7-rc.2",
  "guard_package_version": "1.0.1",
  "test_type": "isolated installed AgentLoop; mock model; in-memory tool",
  "cases": [
    {
      "scenario": "one malformed response followed by recovery",
      "model_requests": 3,
      "rejected_responses": 1,
      "tool_executions": 1,
      "malformed_text_in_derived_assistant_history": false,
      "turn_end": "completed",
      "result": "PASS"
    },
    {
      "scenario": "persistent malformed responses",
      "model_requests": 3,
      "rejected_responses": 3,
      "tool_executions": 0,
      "malformed_text_in_derived_assistant_history": false,
      "turn_end": "error",
      "result": "PASS"
    },
    {
      "scenario": "plugin remount during persistent failure",
      "model_requests": 3,
      "rejected_responses": 3,
      "tool_executions": 0,
      "malformed_text_in_derived_assistant_history": false,
      "turn_end": "error",
      "result": "PASS"
    }
  ]
}

In the recovery simulation, the three requests are a rejected response, a valid tool call, and a final response after the tool result. They are not three failed attempts. These tests also verified that rejected-response diagnostic files were retained inside the isolated test environment.

Remaining limits and privacy

This supports partial mitigation, not a claim of zero future failures, a quantified production success rate, or general long-task reliability. There is no randomized before/after comparison, and the 54-call test follows both history recovery and guard installation, so it cannot isolate the guard's causal contribution. Existing contaminated histories are not automatically repaired by enabling the switch. The guard does not fix the upstream cause.

This comment contains manually reviewed summaries and synthetic test counts only. No raw rejected chunks, full logs, private prompts, task contents, tool arguments, credentials, endpoint URLs, institutional identifiers, hostnames, IP addresses, filesystem paths, session/call IDs, or private-file hashes are uploaded. The raw detection records remain private.

中文小结：护航机制已有“部分缓解”的证据，包括原会话恢复后的连续工具验证，以及一次拦截重试后同轮继续完成的历史记录。当前 1.0.1 的隔离回归通过，但尚无其真实拦截成功率；不是上游根治，也不能据此保证所有长任务可靠。