Jump Forward Decoding Disabled?

[tshreyas-aws]

Hello, is there a specific reason jump forward decoding was disabled, other than maintenance issues (as described here: [PR #4032] (#4032))?

[yhay81]

There is no documented model-quality or regex-correctness failure that caused the removal. The public record supports two reasons: the old implementation was tightly coupled to the scheduler/cache/detokenizer path, and the intended replacement direction was speculative decoding.

The removal PR says explicitly, “Remove jump forward to simplify the code maintenance” (#4032). When a contributor asked the same question in that PR, the response was that jump forward might later be implemented using speculative decoding (maintainer/contributor reply). I could not find a stronger public claim such as “it was slower” or “it produced incorrect regex output.”

The deleted code explains why maintenance was unusually expensive. The old path did much more than skip a few sampling steps. For every deterministic grammar span it had to:

• remove the request from the running decode batch and later requeue it;
• append provisional token IDs, incrementally decode them, concatenate the deterministic text, then tokenize the whole prompt/output again to handle token-boundary fusion;
• update the grammar state and the detokenizer version;
• reconcile output IDs and logprobs after retokenization;
• insert the pre-jump prefix into the radix cache, reapply multimodal padding, and run another prefill for the jumped request.

It was also forcibly disabled whenever overlap scheduling was enabled. Those details are visible in the #4032 scheduler and ScheduleBatch deletion. In other words, the optimization cut across batching, streaming/detokenization, logprobs, prefix caching, multimodal handling, and scheduling. Each new execution mode had to preserve all of those invariants.

This also explains a confusing detail in the current tree: grammar backends still contain methods such as try_jump_forward, and the Outlines file still contains the map class. They are not evidence that the old runtime optimization is active. The scheduler call (batch.check_for_jump_forward(...)) and request retokenize/requeue path were removed, and Outlines constructs its grammar with jump_forward_map = None. Merely changing that one line, as suggested in older issue #8954, cannot restore the feature because the consumer side no longer exists.

The trade-off is therefore:

• lost benefit: deterministic characters in a regex/JSON grammar are no longer injected as one jump by the old scheduler path; constrained decoding normally advances through the regular generation machinery;
• gained benefit: one unified scheduler/cache/logprob/streaming path, including overlap scheduling, without a special request-retokenization cycle;
• replacement direction: use the general speculative-decoding machinery to accept multiple tokens while keeping verification and cache state in the normal execution model, rather than surgically rewriting a request around deterministic FSM spans.

Recent work such as grammar + speculative decoding support and overlapping grammar work with speculative verification follows that direction, but it is not the same as restoring the old compressed-FSM jump-forward algorithm. So the honest answer is: maintenance/integration complexity is the only confirmed removal reason; compatibility with overlap scheduling and the amount of cross-cutting state in the deleted implementation show what that phrase meant, and speculative decoding was the stated future alternative.