Retained MTP State

Retained MTP state

One-layer integrated MTP normally computes a draft token and later needs the corresponding hidden/recurrent boundary again when accepted target tokens are committed. This feature retains the one-step draft state so accepted work can be finalized without replaying the same MTP step from scratch.

Eligibility and lifecycle

The runtime retains state only for an independently owned, one-layer MTP context whose positions, tokens, row structure, and memory boundary match exactly. Shared-memory models and incompatible shapes keep the established path.

After target verification, accepted suffix updates from multiple active sequences are gathered into one bounded finalization batch. Rejected or invalid retained states are removed, and deferred state is cleared when a server slot is released. Any mismatch declines retention rather than applying hidden state to the wrong sequence.

The feature reduces redundant MTP work; it does not improve acceptance probability and does not change the draft token. Its benefit is most visible when depth-one acceptance is high and MTP compute is a meaningful fraction of the loop.

Note

The retained-state coordinator is generic C++ runtime code. When used with a cached MoE MTP context, its graph execution is CUDA grouped work.

Implementation

• Retained-state contract and batched finalization: common/speculative.cpp, common/speculative.h
• Server ownership and release: tools/server/server-context.cpp
• Prompt checkpoint namespace: common/common.h

Implementation commits: 6451562be, 47d1eaff4, and 55aeee80d.

Feature index