Repository navigation
MNN Supports Eagle3 #4355
wangzhaode
announced in
Technical Blog
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Inference
Eagle3 trains a draft model approximately equal to one layer of the target model plus a
lm_headof about 32K parameters. During generation, the draft model produces multiple draft tokens. The target model then "verifies" these drafts in one go, rather than generating them token by token as in traditional autoregressive decoding. This reduces the number of calls to the target model, thereby achieving inference acceleration.The entire inference pipeline can be broken down into the following key steps:
Step 1: Initialization and Context Processing
Step 2: Eagle3 Generates Draft Tokens
Step 3: Target Model Batch Verification
(1 + K), Qwen2 computes the probability distributions for all(1 + K)positions at once.Step 4: Accept or Reject and Start the Next Cycle
This pipeline effectively replaces multiple target model inferences with "multiple small model inferences + one target model verification," resulting in a total memory access volume that is smaller than pure target model inference.
For example, assuming the target model parameters are 2B and the draft model size is 0.1B, if we generate 3 drafts per round and accept 1 on average, the memory access volumes are as follows:
The memory access volume of speculative decoding is 57.5% of standard decoding, effectively reducing bandwidth requirements and improving the model's Decode speed.
Eagle3's advantage lies in achieving a high acceptance rate even with a relatively small number of parameters, which is why we chose to use Eagle3.
Eagle3 Inference in MNN
To efficiently implement Eagle3's speculative decoding algorithm in the MNN C++ inference engine, we designed and implemented a complete end-to-end pipeline. We abstracted the complex logic of draft token generation and management into an independent, reusable
TokenTreeclass, which greatly simplifies the main flow's complexity and enhances code maintainability.Core Design: The
TokenTreeClassIn speculative decoding, the draft model needs to generate a "tree" containing multiple candidate sequences. Managing the growth, pruning, and information extraction of this tree is the most complex part of the entire pipeline. To address this, we created the
TokenTreeclass, which encapsulates all the following core functionalities:TokenTreeNodestruct to represent each token in the tree, clearly storing the token ID, log probability, depth, and parent-child relationships.init(): Receives the first-round Top-K prediction results and initializes the first layer of tree nodes.grow(): In each iteration, receives the Top-K prediction results for all current leaf nodes, generates new candidate nodes, and performs sorting and pruning based on cumulative probability, retaining only the best K candidates as new leaf nodes.TokenTreesynchronously and efficiently computes and updates the attention masks needed for the next iteration, ensuring each token can only attend to its ancestor nodes on the path.finalize(): Once the tree grows to the preset depth, this method organizes all candidate draft tokens and generates all inputs required for the target model's batch verification, including:draftTokens: Flattened draft token sequence.positionIds: Correct position IDs for each draft token in the sequence.attentionMask: Attention mask conforming to the tree structure, for use by the target model.retrieveIndices: An index list for quickly backtracking and verifying the best path from the target model's output.EagleGeneration: A Clear Main FlowBy delegating the complex tree structure management logic to
TokenTree, the responsibilities of the main flow classEagleGenerationbecome very clear and linear:TokenTreeinstance and call itsgrow()method to expand the draft.TokenTree'sfinalize()method to obtain, in one go, all packed input tensors for the target model's verification.Training
Eagle3 Training Characteristics
Eagle3's excellent performance actually stems from the design of its training process. Before Eagle3, draft models like EAGLE had a common problem:
Eagle3 thoroughly solves the above problems through the following two closely integrated improvements:
1. Eliminate Feature Prediction Constraints, Fuse Multi-Level Features
First, Eagle3 completely abandons the old paradigm of predicting intermediate features, directly targeting final token distribution prediction. More importantly, it no longer relies solely on the target model's top-layer features, but innovatively fuses feature information from low, middle, and high levels. Through a simple fully connected layer (FC) for information compression, it provides the draft model with semantic input that is far richer and more comprehensive than before.
2. Introduce "Test-Time Training" (TTT) to Solve Error Accumulation
This is the most core innovation in Eagle3's training methodology. The idea is to completely simulate the multi-step generation process during inference at training time:
This "self-generated, self-consumed" closed-loop training approach forces the model to learn how to handle and correct minor biases that may exist in its own predictions. It fundamentally solves the training-inference inconsistency problem, making the trained draft model exceptionally robust.
In summary, through the "test-time training" mechanism, our trained Chinese Eagle3 model not only learns the target model's outputs but, more importantly, learns how to perform stable, high-quality continuous generation on its own output sequences. This enables the model to produce longer draft sequences with higher acceptance rates during actual inference, laying a solid foundation for achieving industry-leading on-device inference acceleration ratios.
Training Framework Selection: SpecForge
When initiating Eagle3 training tasks, choosing an efficient, flexible, and easily extensible training framework is critical.
Although the official Eagle repository provides the original training implementation code, which gives us valuable reference for understanding its core algorithm, we found it has some limitations in practical engineering applications. For example, its natively supported model types are limited. When we needed to apply it to newer models like Qwen2, it often required significant effort for code adaptation and modification, which somewhat slowed down the R&D and iteration pace.
To address these challenges, we ultimately chose SpecForge, a framework carefully crafted by the SGLang team. SpecForge is a training library designed specifically for speculative decoding draft models, and it perfectly aligns with our project needs:
Broad Model Support: SpecForge has out-of-the-box features, seamlessly supporting a wide range of mainstream open-source LLM architectures, including the Llama series, Qwen series, and GPT-OSS. This means we can focus entirely on model training and data optimization rather than spending time on tedious low-level code adaptation.
Excellent Usability: The framework provides highly encapsulated and user-friendly APIs. It elegantly abstracts core components such as the "test-time training (TTT)" logic and dynamic attention mask management. Developers can start a complete Eagle3 training pipeline with simple configuration, significantly lowering the barrier to entry. Additionally, it supports training on AMD GPUs, which made it easy for me to get started in an AMD environment.
Active Community and Maintenance: As part of the SGLang ecosystem, SpecForge benefits from active community support and continuous updates, ensuring the framework's stability and cutting-edge capabilities.
Training Data Construction: EagleChat
In the early stages of Eagle3 training for the Qwen3 series based on the SpecForge framework, we primarily used the community's common
ShareGPTandUltraChatdatasets. However, during actual testing, we found that this general data ratio had notable shortcomings in specific scenarios, mainly reflected in:To solve the above problems, we constructed a high-quality Chinese-English bilingual dialogue dataset -- EagleChat -- and conducted extensive ablation studies on it.
Dataset Composition: Mixing and Balancing
EagleChat aims to be a high-quality corpus that significantly enhances the comprehensive dialogue capabilities of LLMs (especially draft models like EAGLE). We adopted a hybrid strategy, introducing
smoltalk-chineseinto the existingShareGPTandUltraChatdatasets, which not onlycompletes the Chinese capability but also strengthened chain-of-thought (CoT) reasoning ability.The final constructed dataset contains over 1 million high-quality dialogue entries, distributed as follows:
All data has been uniformly formatted, cleaned, and randomly shuffled to ensure training stability.
Effect Verification: Let the Data Speak
To verify the effectiveness of EagleChat, we used
Qwen3-4B-Instructas the base model and trained three versions of the Eagle3 draft model using EagleChat, ShareGPT, and UltraChat respectively. We then compared their average acceptance length (acc_length) across multiple authoritative benchmarks.The core experimental results are shown in the following table:
(Draft=N means the target model verifies N tokens at a time)
Result Analysis:
smoltalk-chinesein solving the "Chinese aphasia" problem.Visual Comparison (Draft Tokens = 4 vs 8):
The experimental data shows that a high-quality, well-balanced EagleChat dataset is the key to training high-performance, Chinese-supporting Eagle3 models.
Engineering Adaptation: Improving SpecForge
During training, we discovered some issues in SpecForge and fixed and improved them, contributing our modifications back to the SpecForge community.
num_warpssetting for Triton kernel when training in AMD environments PR #259Testing
Deploying the trained models to real on-device hardware was the final step of our entire effort and a criticalstep for validating our results. During testing, we found that there is a gap between the theoretical optimal configuration and practical application scenarios, especially on compute-constrained devices, where tuning the number of drafts is crucial.
Trade-offs in Verification Overhead
On servers or devices with powerful GPUs, the overhead of verifying multiple draft tokens at once is relatively small. However, on on-device hardware, especially when running on CPUs, we found that verification itself is not without overhead.
When the draft tree is set too wide or too deep (i.e., generating a large number of draft tokens at once), the computation required for the target model's batch verification increases accordingly. If the acceptance rate of these drafts is not high enough, then the overhead of such "large-batch verification" may even offset the benefits brought by speculative decoding, causing the overall acceleration ratio to decrease rather than increase.
Optimal Draft Count Configuration
After multiple rounds of experiments, we found that in compute-constrained scenarios like on-device CPUs, more draft tokens does not necessarily mean better results. A smaller number of draft tokens means lower verification overhead and higher average acceptance rates, whichinstead leads to more stable acceleration.
Our recommended configuration is to generate 3 or 7 draft tokens per round. Together with the first context token, the target model only needs to verify 4 or 8 tokens per pass. This number achieves the best balance between draft generation overhead, batch verification overhead, and high acceptance rate on current mainstream on-device hardware.
Measured Acceleration Results
After adopting the above optimization configuration, we achieved significant acceleration effects on on-device hardware.
Scenario Performance Differences:
Quantified Acceleration Results:
This result demonstrates that, with careful training and parameter tuning, the Eagle3 draft model can deliver stable and substantial performance improvements on on-device hardware.
Done! And "Out-of-the-Box" Models
That concludes the entire process of MNN supporting Eagle3.
We started by refactoring the C++ code, paired the model with a Chinese-English mixed
EagleChatdataset, contributed bug fixes back to the SpecForge community, and finally ran some quick performance tests on on-device hardware -- essentially stepping into every pitfall along the way.We've put the final trained "finished products" -- the Eagle3 models adapted for Qwen3 -- below for everyone to download and try out, or to open issues and discuss with us.
Hugging Face Portal:
https://huggingface.co/collections/taobao-mnn/eagle3
ModelScope Portal:
https://modelscope.cn/collections/Eagle3-408e79eec8b441
That's it. Happy experimenting!
All reactions