[Bug]: Missing Stop Tokens in responses_utils.py Causes Content Leakage in Tool Use Scenarios

[zzw09773]

System Info

Environment

• CPU architecture: aarch64
• CPU/Host memory size: ~450GB (Estimated for GH200 systems)
• GPU properties:
• GPU name: NVIDIA GH200 480GB
• GPU memory size: 97871 MiB
• Libraries:
  TensorRT-LLM branch or tag: release:1.2.0rc7
• Versions:
  PyTorch: 2.9.0a0+145a3a7bda.nv25.10
  CUDA: 13.0
• Container used: nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc7
  NVIDIA driver version: 580.65.06
  OS: Linux (aarch64)

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Steps to Reproduce
Minimal Reproduction Script
Run this script against a deployed trtllm-serve instance hosting a Harmony model (e.g., gpt-oss-20b).

import openai
# Configuration
API_KEY = "EMPTY"  # Functionality doesn't require actual auth for local repro
BASE_URL = "http://localhost:8000/v1"
MODEL_NAME = "openai/gpt-oss-20b"
client = openai.OpenAI(api_key=API_KEY, base_url=BASE_URL)
try:
    response = client.chat.completions.create(
        model=MODEL_NAME,
        messages=[{"role": "user", "content": "Use search function"}],
        tools=[{
            "type": "function",
            "function": {
                "name": "search",
                "description": "Search function",
                "parameters": {
                    "type": "object",
                    "properties": {"q": {"type": "string"}}
                }
            }
        }],
        temperature=1.0, # High temperature increases likelihood of hitting the bug
        max_tokens=100
    )
    content = response.choices[0].message.content
    print(f"Response Content: {content}")
    
    if "<|channel|>" in str(content) or "<|message|>" in str(content):
        print("FAIL: Leak detected! Internal tokens found in output.")
    else:
        print("PASS: Output looks clean (or bug not triggered this time).")
except Exception as e:
    print(f"Error: {e}")

Expected behavior

The model should inherently generate appropriate EOS tokens (e.g., <|call|>, <|return|>) even in tool-use scenarios and high temperatures. stop_token_ids should be passed correctly to the generation engine so that:

The output stream is terminated correctly at the end of a tool call or message.
No internal raw tokens (like <|channel|>) are exposed to the parser or the user.

actual behavior

Due to stop_token_ids being conditionally set to [] (empty list) in certain request paths (use_harmony=False), the model ignores stop conditions. This leads to:

Uncontrolled Generation: The model continues generating past the intended end of the message.
Handling Failures: The Harmony Parser fails to interpret the malformed/extended sequence and falls back to raw text decoding.
Content Leakage: The user receives the raw internal representation, including <|channel|> tags and potential hallucinations/spam.

additional notes

We identified the potential root cause in tensorrt_llm/serve/responses_utils.py.

The code conditionally sets stop_token_ids:

sampling_params = request.to_sampling_params(
    default_sampling_params={
        "stop_token_ids":
        get_harmony_adapter().get_stop_tokens() if use_harmony else []
    })

When use_harmony is unexpectedly False (which occurs intermittently in certain request flows), stop_token_ids defaults to []. This leaves the model without explicit stop conditions.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[LinPoly]

Thanks for reporting this issue. The client script is using client.chat.completions.create but IIRC responses_utils.py should be only about responses API which should be query by client.responses.create, so I suspect that it is not the root cause. And use_harmony is determined by model_type rather than requests. cc @JunyiXu-nv for comments.

[LinPoly]

@zzw09773 Could you please also share the server launching command and possibly the config file you are using? Would be easier to reproduce that way.

[JunyiXu-nv]

Yes responses_utils.py is not used by chat completions API flow.

Uncontrolled Generation: The model continues generating past the intended end of the message.

This sounds more like an accuracy issue. Does the model just cannot stop generating util hit the max_tokens limit? Could you post the error log from server side? @zzw09773

CC @dongfengy for a similar can't stop reasoning issue. Would setting temperature to 1.0 for gpt-oss lead to this issue?

[zzw09773]

@LinPoly @JunyiXu-nv Thanks for the quick response and clarification regarding responses_utils.py.

Here is the requested information to help reproduce/debug:

Server Launching Command

Running via docker-compose:

  gpt-oss-20b:
    image: nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc7
    container_name: gpt-oss-20b
    runtime: nvidia
    restart: unless-stopped
    shm_size: 16g
    ipc: host
    working_dir: /workspace
    ulimits:
      stack: 67108864
      memlock: -1
    environment:
      TZ: Asia/Taipei
      NVIDIA_VISIBLE_DEVICES: "0"
      TRTLLM_ENABLE_PDL: "1"
      HF_HOME: /workspace/model
      TIKTOKEN_RS_CACHE_DIR: /workspace/vocab
      TIKTOKEN_CACHE_DIR: /workspace/vocab
    ports:
      - "7000:8000"
    expose:
      - "8000"
    volumes:
      - ./models/GPT-OSS-20B-V2:/workspace:rw
    networks:
      - private_network
    command: >
      trtllm-serve openai/gpt-oss-20b
        --host 0.0.0.0
        --port 8000
        --backend pytorch
        --max_num_tokens 65536
        --max_batch_size 720
        --kv_cache_free_gpu_memory_fraction 0.9
        --tp_size 1
        --ep_size 1
        --trust_remote_code

{
  "bos_token_id": 199998,
  "do_sample": true,
  "eos_token_id": [
    200002,
    199999,
    200012
  ],
  "pad_token_id": 199999,
  "temperature": 0.5,
  "top_p": 0.9,
  "transformers_version": "4.55.0.dev0"
}

Error Logs

When the issue occurs (mostly at higher temperatures), the server logs show:

[TRT-LLM] [W] Failed to parse harmony messages from tokens: Unknown role: assistant<|channel|>commentary
[TRT-LLM] [W] Failed to parse harmony output: Harmony parsing failed: Unknown role: assistant<|channel|>commentary. Raw output: <|channel|>analysis<|message|>...
[TRT-LLM] [W] ⚠️ Harmony parsing fell back to raw text decoding

We initially suspected responses_utils.py because the issue appeared after upgrading to 1.2.x, but your explanation makes sense that it's not used in the chat.completions flow. The core issue seems to be that when the model generates "unparseable" output (which happens more often at high temp), the Harmony Adapter falls back to raw text decoding (fallback_content = raw_text), which exposes the internal <|channel|> tokens to the user. We have temporarily worked around this by patching harmony_adapter.py to return a generic error message instead of the raw text during fallback.

Actual behavior

Due to stop_token_ids being conditionally set to [] (empty list) in certain request paths (use_harmony=False), the model ignores stop conditions. This leads to:

Uncontrolled Generation: The model continues generating past the intended end of the message.
Handling Failures: The Harmony Parser fails to interpret the malformed/extended sequence and falls back to raw text decoding.
Content Leakage: The user receives the raw internal representation, including <|channel|> tags and potential hallucinations/spam.

Observed Leak Example:

[dongfengy]

Not sure if related but I see some strange words in the output, like “开元棋牌", which is a bug, fix (workaround): #10713

[dongfengy]

Also I see the 1.2.0rc7 was used, which didn't include an important accuracy fix: https://github.com/NVIDIA/TensorRT-LLM/pull/10619/files

We should try a later release with both fixes merged

[LinPoly]

Cannot reproduce on our latest release/1.2 branch with 20 runs, 1 out of 20 times it fails to parse the tool info cuz the model didn't generate correct format, but I think it is acceptable because of randomness, could you please try our release branch? @zzw09773

[zzw09773]

Thanks for the clarification.

I understand there are two potentially separate factors here:

1. The unexpected token output issue (e.g. strange strings like “開元棋牌”), which seems to be addressed by the workaround in [https://nvbugs/5791242][fix] workaround for flashinfer.sampling.sampling_from_logits #10713.
2. The accuracy-related fix that was not included in 1.2.0rc7 (PR [https://nvbugs/5772396][fix] WAR: Disable TinyGEMM PDL due to accuracy issues #10619).

Based on your suggestion, I will try reproducing this on a newer release / release-1.2 branch where both fixes are merged and will report back once I finish testing on the newer release.
If you have a recommended tag or specific commit hash that includes both fixes, please let me know — that would help us align faster.

Thanks again for the detailed follow-up.

[dongfengy]

This is the 1.2 release branch: https://github.com/NVIDIA/TensorRT-LLM/commits/release/1.2/
You can built it yourself, or if you want pre-built containers (https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/release/tags) :
I think 1.2.0rc8 is too old (there is another allreduce fix not included). You can either wait for official release 1.2.0 or wait for next main branch weekly release (1.3.0rc1) which should also include all the fix. Let us know if you have any issue verifying the fixes. Thanks!