Skip to content

fix(ds4): enforce generation boundaries - #11821

Merged
mudler merged 1 commit into
mudler:masterfrom
FiloSpaTeam:fix/ds4-generation-boundaries
Sep 2, 2026
Merged

fix(ds4): enforce generation boundaries#11821
mudler merged 1 commit into
mudler:masterfrom
FiloSpaTeam:fix/ds4-generation-boundaries

Conversation

@FiloSpaTeam

Copy link
Copy Markdown
Contributor

Clamp requested generation to the usable context after prompt sync while preserving the legacy 256-token fallback for omitted limits.

Constrain each speculative MTP cycle to the remaining request budget so accepted tokens cannot advance beyond the visible output limit.

Assisted-by: Codex:gpt-5.6-sol

Description

This PR fixes DS4 generation boundary handling in both streaming and non-streaming requests.

Explicit generation limits are clamped after prompt synchronization using the remaining context capacity. Omitted or non-positive limits retain the legacy 256-token fallback.

Speculative MTP decoding is also constrained by the remaining request budget, preventing accepted tokens from advancing beyond the visible output limit.

Notes for Reviewers

  • The existing 256-token fallback remains unchanged.
  • An explicit max_tokens: 32768 is treated as an upper bound and may be reduced to fit the available model context.
  • Streaming and non-streaming paths use the same limit calculation.
  • Request cancellation is intentionally excluded and will be submitted separately.
  • All 17 standalone DS4 C++ test files pass.
  • A full CUDA/GB10 Docker end-to-end build was not available in the development workspace.

Signed commits

  • Yes, I signed my commits.
  • Documentation updated (docs/content/) for user-facing changes, or not applicable

@FiloSpaTeam
FiloSpaTeam force-pushed the fix/ds4-generation-boundaries branch 3 times, most recently from 877efad to de45df4 Compare September 2, 2026 07:30
Clamp requested generation to the usable context after prompt sync while preserving the legacy 256-token fallback for omitted limits.

Constrain each speculative MTP cycle to the remaining request budget so accepted tokens cannot advance beyond the visible output limit.

Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
@FiloSpaTeam
FiloSpaTeam force-pushed the fix/ds4-generation-boundaries branch from de45df4 to e610696 Compare September 2, 2026 09:19
@mudler
mudler merged commit 30e53f8 into mudler:master Sep 2, 2026
1 check passed
@FiloSpaTeam
FiloSpaTeam deleted the fix/ds4-generation-boundaries branch September 2, 2026 10:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants