Independent Draft Ubatch

Independent draft ubatch

A separately loaded draft model can use a physical microbatch size different from the target context. This controls draft prompt/workspace geometry independently from the target's -ub value.

Control and MTP restriction

Use --spec-draft-ubatch-size N, --ubatch-size-draft N, or -ubd N. Zero inherits the target. A smaller draft ubatch can reduce draft workspace and make a separate drafter fit, but can lower its prompt throughput.

Integrated MTP deliberately does not accept a different physical ubatch: it must inherit or exactly equal the target ubatch. Long-output validation showed that changing recurrent prompt synchronization geometry can change later output even when short screens pass. Use phase-aware workspace to reduce MTP generation workspace without changing physical MTP geometry.

This feature does not choose a drafting algorithm and does not alter draft depth. Use the upstream speculative guide for ordinary drafting configuration.

Branch implementations

Branch	Implementation
beellama/main	common/speculative.cpp, common/arg.cpp
llama/main	common/speculative.cpp, common/arg.cpp
llama/dev	common/speculative.cpp, common/arg.cpp
moe-cache	common/speculative.cpp, common/arg.cpp

Design history: PR 27, independent draft ubatch.

Feature index