Per Sequence Draft Limits

Per-sequence and zero draft limits

Parallel server slots can have different remaining token budgets or request-local draft limits. This feature makes the speculative coordinator honor each active sequence's effective maximum and treats zero as a valid request to skip drafting for that sequence.

Behavior

The coordinator resolves the configured global maximum, the drafter's trained/effective capacity, and the per-sequence request limit. A sequence stops drafting when its own positive limit is reached. A zero limit bypasses draft generation without disabling speculation for unrelated sequences in the same server process.

This prevents a parallel batch from over-generating draft rows for one slot merely because another slot allows a larger depth. It also keeps grouped verification row counts and output capacity consistent with the work actually requested.

There is no new global CLI flag added by this feature; it corrects how existing speculative limits are applied. Normal request fields and speculative controls remain documented by the upstream server.

Implementation

• Limit resolution and draft loop: common/speculative.cpp
• Per-sequence parameter structure: common/speculative.h
• Server integration: tools/server/server-context.cpp

Implementation commits: bfc28713b and 6de9e8c9e.

Feature index