EvSpark: ~3× lossless speculative decoding for Evo2 7B with hybrid-state rollback

[dhnihaoya]

Hi Evo2 team and community — we recently released EvSpark, a lossless speculative decoding implementation for Evo2 7B / StripedHyena2.

TL;DR: EvSpark speeds up single-stream Evo2 7B generation by 3.16× end-to-end in our preprint benchmark, while preserving the target model exactly: greedy decoding is token-for-token identical to native Evo2, and sampling preserves the target distribution through rejection-sampling verification.

The part that turned out to be most interesting was not the drafter itself, but making speculative decoding work correctly and efficiently with Evo2's hybrid inference state.

The systems problem

For a standard Transformer, after verifying a speculative block, rejected tokens can essentially be rolled back by truncating the KV cache.

For StripedHyena2, the inference state is heterogeneous:

• attention KV caches are append-only;
• Hyena FIR states are fixed-length sliding windows;
• Hyena IIR states are recurrent states whose updates depend on the entire accepted prefix.

A naive snapshot-and-replay implementation works, but requires another target-model forward pass after every partially accepted draft, which removes much of the benefit of speculation.

What we implemented

EvSpark adds a block-verification forward pass + per-position state-slicing protocol.

During verification, we retain the intermediate inference state corresponding to each position in the speculative block. Once the verifier decides how many tokens to accept, the target state is sliced directly to the accepted boundary across the Hyena FIR/IIR and attention KV components.

This avoids replaying the accepted prefix entirely.

With an identity drafter, the speculative infrastructure adds only 1.05–1.16× overhead relative to native decoding in our measurements, including verification and rollback.

On top of that, we trained a small γ-parallel distilled drafter using frozen Evo2 hidden states. At γ=12, the configuration reported in the preprint reaches 3.16× end-to-end speedup on Evo2 7B.

A few other observations:

• the target Evo2 weights are never modified;
• greedy decoding showed 0 non-tie divergences in our evaluation;
• sampling uses exact rejection-sampling verification rather than an approximate acceptance rule;
• acceleration remains useful at long context lengths — we tested out to 262k context;
• the drafter itself is quite cheap: a competitive checkpoint can be trained in roughly one RTX 4090 GPU-hour.

Links

Code + checkpoints + reproduction:
https://github.com/dhnihaoya/EvSpark

Preprint:
https://www.biorxiv.org/content/10.64898/2026.09.02.749017v1

The repository includes a runnable demo, pretrained drafters, the speculative decoding implementation, and the evaluation/reproduction pipeline.

We'd be very interested in feedback from the Evo2 / Vortex team, particularly on the state-slicing approach. We currently implement this on top of the existing Evo2/Vortex inference stack, but it seems like the underlying mechanism could potentially be useful more generally for block decoding or other inference-time methods on StripedHyena models.

Happy to discuss implementation details or share any additional profiling / ablations that would be useful.

[jstjohn]

FYI we recently added sequence packing/batched generation to decode in bionemo's implementation. Might be interesting to see how this would work with that as well. https://github.com/NVIDIA-BioNeMo/bionemo-recipes/tree/main/recipes/evo2_megatron

[jstjohn]

It was in this PR: NVIDIA-BioNeMo/bionemo-recipes#1735

[dhnihaoya]

Thanks for the pointer — I went through the recipe's inference path (ragged/packed prefill + batched recurrent decode, block-scope CUDA graph, TE FP8). Two honest notes on how this would combine with speculative decoding:

Regime overlap. Our speedup comes from the memory-bound B=1 step: on a 4090, roughly 80% of the ~20 ms native step is the 13.2 GB weight read. Packing/batching amortizes exactly that same read, so the two techniques compete for the same headroom rather than composing additively. The Limitations section of the preprint extrapolates the leftover gain on that 4090 / bf16 / Vortex path (~1.5–2× at B=8, break-even around B≥32); those B numbers are not meant to transfer. The crossover tracks device arithmetic intensity (and shrinks further under FP8 / fused recurrent decode), so I would expect it to move earlier on compute-rich Hopper/Blackwell and later on bandwidth-heavy SKUs. The split that should still hold is: speculation for latency-bound single-stream / small-batch generation, packing+batching for throughput. Worth measuring rather than assuming, but I wouldn't expect the two gains to multiply.

What would have to transfer is the state-slicing contract, not the drafter. Two concrete prerequisites on the dynamic backend:

1. Block verification needs the per-position intermediate Hyena FIR/IIR state kept for all γ+1 candidate positions, so the state can be sliced to the accepted boundary. In fused recurrent decode kernels (b2b_causal_conv1d and friends) that intermediate state is typically not exposed — so this would need either a new path or a fallback to the rectangular compatibility path.
2. Accepted lengths are dynamic and differ per sequence within a batch. That conflicts with block-scope CUDA graph capture and needs ragged rollback bookkeeping (every sequence truncates to a different boundary). This is the genuinely hard part, and also the more interesting one.
   Happy to share the slicing protocol, the drafter checkpoints, and the evaluation/losslessness harness with anyone who wants to prototype this on the dynamic backend. On our side we will measure the small-batch end of the picture on the Vortex stack (Ada); that should at least pin down where the crossover sits, independent of the packed/fused path.

Thanks again for the FYI.