[None][fix] Hide trailing EOS from generated text

[Wanli-Jiang]

Summary

When a request stopped on the natural EOS (FinishReason.END_ID), TRT-LLM previously relied solely on skip_special_tokens=True to keep the EOS string out of output.text. Whenever a caller set skip_special_tokens=False (e.g. some tool / structured-output parsers that need raw special tokens), or whenever the tokenizer did not flag the EOS id as a special token, the EOS string leaked into the OpenAI response text. vLLM does not exhibit this because it always slices the trailing stop token from the detokenized text. Observed on Nemotron-Super-V3 (EOS = <|im_end|>, id 11).

Before vs After

Repro (programmatic; the example file is untouched):

from tensorrt_llm import LLM, SamplingParams

llm = LLM(model="<path-to-Nemotron-Super-V3>",
          tensor_parallel_size=4, trust_remote_code=True)

prompt = llm.tokenizer.apply_chat_template(
    [{"role": "user", "content": "The capital of France is"}],
    tokenize=False, add_generation_prompt=True,
)
out = llm.generate(prompt, SamplingParams(
    max_tokens=128,
    skip_special_tokens=False,   # forces the EOS-leak code path
)).outputs[0]
print("text tail:", repr(out.text[-80:]))
print("last token_ids:", out.token_ids[-8:])

Before fix — <|im_end|> leaks into output.text:

[0] text tail: "...started to introduce yourself but didn't finish the sentence. What is your name?<|im_end|>"
    last token_ids: [19286, 1046, 5675, 1395, 2143, 2564, 1063, 11]

[1] text tail: '...I should provide the answer directly.</think>The capital of France is **Paris**.<|im_end|>'
    last token_ids: [8961, 1307, 5498, 1395, 1603, 42572, 12617, 11]

After fix — output.text is clean, output.token_ids is unchanged:

[0] text tail: "...started to introduce yourself but didn't finish the sentence. What is your name?"
    last token_ids: [19286, 1046, 5675, 1395, 2143, 2564, 1063, 11]   # id 11 still here

[1] text tail: '...I should provide the answer directly.</think>The capital of France is **Paris**.'
    last token_ids: [8961, 1307, 5498, 1395, 1603, 42572, 12617, 11]  # id 11 still here

Invariants preserved:

• output.token_ids[-1] == sampling_params.end_id whenever the request ended on natural EOS — useful for distinguishing "ended on EOS" (stop_reason is None + tail id matches end_id) from "ended on a configured stop_token_id" (stop_reason set). Matches vLLM.
• len(output.token_ids) == len(output.logprobs) for callers that requested logprobs.

Default-path control (the unchanged case where the EOS happens to be a tokenizer-marked special token AND skip_special_tokens=True):

text tail: "...What is your name?"            # same as before fix
last token_ids: [..., 1063, 11]                # same as before fix

— i.e. no regression on the previously-working default path; the fix only changes behavior on the path that was leaking.

Files changed

tensorrt_llm/executor/result.py only (+19 / −3 lines):

• GenerationResultBase._handle_sequence END_ID branch: comment only (no behavior change here).

• DetokenizedGenerationResultBase._handle_response: before each tokenizer.decode / decode_incrementally call, exclude the trailing end_id from the slice when:

  • finish_reason == 'stop'
  • stop_reason is None
  • not include_stop_str_in_output
  • the local list (token_ids or token_ids_diff) ends with end_id

  output.token_ids is left intact.

The pre-existing STOP_WORDS branch already strips its matched ids from output.token_ids upstream, so the new slice is a no-op for that path (its stop_reason is non-None, which gates the check off).

Test plan

• Re-ran Nemotron-Super-V3 (TP=4, B300, NVFP4) with both skip_special_tokens=True (default) and skip_special_tokens=False. Before fix: EOS leaks into text under False. After fix: no leak under either setting; output.token_ids ends with end_id in both cases.
• Length-capped requests (no EOS reached) are unchanged in both text and token_ids.
• Existing test_generate_with_stop_words (tests/unittest/llmapi/test_llm.py:835) covers the end_id=stop_id path with similar(..., threshold=0.8). The fix only tightens the match (text no longer contains the trailing end_id glyph).

Summary by CodeRabbit

Bug Fixes

• Improved end-of-sequence token handling in text generation to ensure EOS tokens are properly hidden from decoded output text while maintaining internal accuracy.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR modifies EOS token handling in TensorRT-LLM's generation result processing. It documents that END_ID tokens remain in the token_ids collection while being hidden from output text by the detokenizer, then refactors detokenization logic to conditionally strip trailing EOS tokens before decoding.

Changes

EOS Token Handling in Generation Results

Layer / File(s)	Summary
END_ID handling documentation in sequence finalization tensorrt_llm/executor/result.py	GenerationResultBase._handle_sequence clarifies that END_ID tokens remain in token_ids but are hidden from text by the detokenizer, with stop_reason remaining None for natural EOS.
EOS trimming logic in detokenization tensorrt_llm/executor/result.py	DetokenizedGenerationResultBase._handle_response introduces end_id and include_stop variables, defines trim_eos gating, and conditionally strips trailing end_id from both incremental and non-incremental decode paths before producing output text.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is clear and directly related to the main change: hiding trailing EOS tokens from generated text output.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description comprehensively explains the issue, solution, before/after behavior, test coverage, and invariants preserved.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Wanli-Jiang]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #49351 [ run ] triggered by Bot. Commit: 51b9c06 Link to invocation

[Superjomn]

Please add some unittests for it, or it tends to be broken in the following iterations.

[tensorrt-cicd]

PR_Github #49351 [ run ] completed with state SUCCESS. Commit: 51b9c06
/LLM/main/L0_MergeRequest_PR pipeline #39007 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[Wanli-Jiang]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #49567 [ run ] triggered by Bot. Commit: 51b9c06 Link to invocation

[tensorrt-cicd]

PR_Github #49567 [ run ] completed with state SUCCESS. Commit: 51b9c06
/LLM/main/L0_MergeRequest_PR pipeline #39194 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation