Speculators v0.8.0

Speculators v0.8.0

This release builds on the v0.7.0 DSpark/DFlash foundation with a unified speculators CLI, first-class Mooncake / hidden-state connectors (now published to PyPI), a fused Triton loss kernel for lower-memory training, experimental DFlash2 support, and broad hardening across the data-generation, preprocessing, and evaluation pipelines.

Key Features

• Unified speculators CLI consolidating response regeneration, preprocessing, and online/offline training entry points.
• Mooncake hidden-state extraction backend and standalone hs-connectors package, now installable from PyPI.
• Memory-efficient training through a single fused Triton online-softmax loss kernel.
• Experimental DFlash2 training and checkpoint support.
• Laguna warm-start support with DFlash weight remapping.
• Reproducibility artifacts for evaluation runs, including eval_command.txt and vLLM launch metadata.
• More robust data pipelines, including NaN skipping, pre-tokenized dataset support, hidden-state validation, and recovery from generation failures.

Unified Speculators CLI

A single speculators CLI now provides a consolidated interface for training and related workflows.

• Consolidated response regen, preprocessing, and training workflows.
• Migrated all training example scripts to the unified CLI.
• Migrated documentation to the unified CLI.
• Unstyled CLI output for more stable comparisons and snapshotting.

DFlash2 Training

• DFlash defaults now use D-PACE, 5 layers, and CE loss.
• Added experimental DFlash2 training and checkpoint support. Note: you will need this commit in order to run the DFlash2 models in vLLM: vllm-project/vllm#53797
• Added Laguna warm-start support:
  • Fused QKV weights are remapped for Laguna warm-start.
  • Laguna-style nested rope_parameters are flattened for compatibility.
• DSpark drafts now default to five layers.
• Reverted the confidence-head detach change.
• Verifier-owned weights are omitted from saved DFlash/DSpark checkpoints to reduce checkpoint size.
• DSpark's Markov lookup embedding is now initialized and optimized as a proper embedding.

Multi-Node Training, Mooncake & hs-connectors

• Added a new Mooncake hidden-state extraction backend.
• Added the standalone hs-connectors package with:
  • Standard licensing and README.
  • Complete package metadata.
  • mooncake-transfer-engine dependency.
  • PyPI release support.
  • Nightly and release build support.
• Speculators now installs the matching hs-connectors package based on build type.
• Mooncake samples are validated with a checksummed manifest.
• Segment, buffer, and writer-thread sizing are configurable.
• Updated KV-cache extraction to follow the upstream restructure and use extract_from_kv_cache.
• Added Mooncake / hs_connectors documentation.

Loss Functions

• Added a fused Triton online-softmax loss kernel used by all losses to reduce memory usage.
• Improved fused CE backward performance by stopping target-logit pinning.
• On Ascend NPU, fused-loss BLOCK_SIZE is capped at 4096.

Data Generation, Preprocessing & Regen

• Support pre-tokenized datasets without requiring a chat template.
• Hardened hidden-state payload validation.
• Recover from hidden-state generation failures during training.
• preprocess.py now uses the vLLM render endpoint.
• Removed --no-enable-chunked-prefill.
• Unknown conversation roles are mapped to assistant during preprocessing.
• Regen now supports:
  • Local prompt files.
  • Sweep and sampling-parameter support.
  • Timing metrics for offline data generation.
• Log the number of broken records during data generation.
• Log the resolved training configuration to the metric backend.

Evaluation, Benchmark & Provenance

• Added evaluation provenance artifacts:
  • eval_command.txt
  • vLLM launch metadata
• Benchmark now bounds the measured window and reports effective throughput.
• Updated evaluation requirements.

Bug Fixes

• MTP: Default to SDPA attention instead of eager attention.
• Regen: Create parent directories for --outfile before opening.
• Locking: Use O_RDWR for flock compatibility on NFS.
• Data: Skip NaN hidden states instead of crashing training.
• Loading: Resolve LLM final norm for multimodal models.
• Training: Make cache cleanup accelerator-agnostic.
• Reject duplicate or negative target_layer_ids.
• Fail clearly when using tiny verifiers.
• Map unknown conversation roles to assistant during preprocessing.
• Suppress httpx logs during vLLM rendering.
• Trust remote code where required for model loading.
• Set the default maximum anchor to 512.

Dependencies

• Updated transformers to >=4.56.1,<5.17.0, and subsequently to 5.0.0.
• Updated datasets requirement to >=4.0.0,<=5.0.1.

CI & Testing

• Migrated regression testing to nightly,
• Enabled per-commit smoke testing.
• Added pre-commit hooks mirroring make quality.

Documentation

• Added DSpark algorithm documentation.
• Added loss-function documentation.
• Added P-EAGLE/DSpark training-argument pages.
• Added a multi-node Kimi-K3 DSpark training example.

Full Changelog

• Add DSpark algorithm page by @guan404ming in #849
• Add timing metrics to offline data generation pipeline by @orestis-z in #804
• fix(regen): create parent dirs for --outfile before opening by @jialefu in #875
• Add loss functions page by @guan404ming in #858
• fix(mtp): default to SDPA attention instead of eager by @rahul-tuli in #881
• docs(tutorial): merge the four training tutorials into one by @WindChimeRan in #840
• Add 429 (Too many requests) to accepted return codes for link checks by @fynnsu in #891
• Add Mooncake hidden states extraction backend by @fynnsu in #836
• Document P-EAGLE and DSpark training arguments by @guan404ming in #857
• revert Detach dspark outputs before feeding them into the confidence head by @shanjiaz in #908
• fix(preprocessing): map unknown conversation roles to assistant by @orestis-z in #907
• fix(tests): fix weekly regression CI failures by @rahul-tuli in #909
• build(deps): update datasets requirement from <=5.0.0,>=4.0.0 to >=4.0.0,<=5.0.1 by @dependabot[bot] in #917
• [Data] Derive off-policy loss masks from vLLM render boundaries by @WindChimeRan in #794
• docs: fix supported models table structure by @cupkk in #925
• Revert "[Data] Derive off-policy loss masks from vLLM render boundaries" by @shanjiaz in #943
• ci(quality): run Quality workflow on release-* branches by @dsikka in #949
• fix(lock): use O_RDWR for flock compatibility on NFS by @fynnsu in #948
• removed no-enable-chunked-prefill by @shanjiaz in #976
• feat(train): default dflash to D-PACE, 5 layers, CE loss (RFC #979) by @shubhra in #980
• feat(dflash): remap fused QKV weights for Laguna warm-start by @orestis-z in #922
• feat: add Inkling weight aliases and training script by @orestis-z in #989
• update last release version by @dhuangnm in #990
• [Memory] Fuse every loss through one Triton online-softmax kernel by @WindChimeRan in #951
• fix(data): skip NaN hidden states instead of crashing training by @orestis-z in #994
• set default max anchor to 512 by @shanjiaz in #995
• fix(dflash): flatten Laguna-style nested rope_parameters by @orestis-z in #921
• update preprocess.py to use vllm render endpoint by @shanjiaz in #975
• cleanup(train): remove the deprecated --legacy-data path by @WindChimeRan in #914
• Update what's new and models list by @dsikka in #998
• [hs_connectors] Add a standard license and readme to enable initial pypi release by @dsikka in #997
• feat(eval): add eval_command.txt provenance artifact (RFC #880) by @orestis-z in #957
• [hs_connectors] Enhance pyproject.toml with description and keywords by @dsikka in #999
• test(mtp): xfail flaky weekly MTP online regression test by @rahul-tuli in #1003
• feat(launch_vllm): add vLLM provenance artifacts (RFC #880) by @orestis-z in #958
• build(deps): update transformers requirement from <5.15.0,>=4.56.1 to >=4.56.1,<5.16.0 by @dependabot[bot] in #1002
• trust remote code by @fynnsu in #963
• feat(data): accept pre-tokenized datasets without a chat template by @fynnsu in #964
• feat(train): log the resolved training config to the metric backend by @fynnsu in #965
• feat(data-generation): harden hidden-state payload validation by @fynnsu in #966
• [hs_connectors] Add mooncake-transfer-engine dependency by @dsikka in #1001
• feat(hs-connectors): validate Mooncake samples with a checksummed manifest by @fynnsu in #967
• feat(hs-connectors): expose Mooncake segment, buffer, and writer-thread sizing by @fynnsu in #968
• perf(losses): stop pinning target logits through the fused ce backward by @WindChimeRan in #1000
• revert: remove speculators import from launch_vllm.py by @rahul-tuli in #1008
• fix(pr-review): flag speculators imports in launch_vllm.py by @rahul-tuli in #1009
• [Testing] Migrate regression testing to nightly by @dsikka in #1004
• Support nightly/release build of hs-connectors and allow speculators to install corresponding package based on the build type by @dhuangnm in #1005
• fix(train): default DSpark drafts to five layers by @WindChimeRan in #1033
• fix(loading): resolve LLM final norm for multimodal models by @orestis-z in #1038
• Omit verifier-owned weights from saved DFlash/DSpark checkpoints by @qianlihuang in #962
• feat(losses): cap fused-loss BLOCK_SIZE at 4096 on Ascend NPU by @PHOEBEMOON0802 in #993
• fix(train): make cache cleanup accelerator-agnostic by @zihanlin-ai in #1025
• stablize eagle3 regression test by @shanjiaz in #1045
• Update eval requirements.txt by @orestis-z in #1042
• fix: reject duplicate or negative target_layer_ids, fail clearly on tiny verifiers by @zihanlin-ai in #1029
• Update transformers version to 5.0.0 by @dsikka in #1047
• Suppress httpx logs on vllm render by @fynnsu in #1040
• Add experimental DFlash2 training and checkpoint support by @mgoin in #1006
• feat(regen): support local prompt files by @WindChimeRan in #991
• ci: add pre-commit hooks mirroring make quality by @orestis-z in #1044
• [Testing] Enable per commit smoke testing by @dsikka in #1016
• [feat]: Log number of broken records by @pavelgein in #1035
• feat(train): recover from hidden-state generation failures by @fynnsu in #969
• No error to build from non-release tagged commit by @dhuangnm in #1039
• feat(benchmark): bound the measured window and report effective throughput by @fynnsu in #970
• fix(dspark): initialize and optimize the Markov lookup embedding as an embedding by @fynnsu in #971
• Regen improvements: sweep + sampling params by @shanjiaz in #1037
• docs(examples): add multi-node Kimi-K3 DSpark training example by @fynnsu in #972
• Add mooncake/hs_connectors docs by @fynnsu in #1046
• feat(cli): unified CLI surface by @rahul-tuli in #1024
• Update kv cache extraction following restructure by @fynnsu in #1068
• decrease gpu memory utilization for hidden sates vllm instance by @shanjiaz in #1069
• Add job checks for Unit, Integration, and Smoke tests by @fynnsu in #1070
• feat(examples): migrate train example scripts + docs to unified speculators CLI by @rahul-tuli in #1067
• Unstyle cli command output for more stable comparisons by @fynnsu in #1074
• Update hs_connector to use upstream extract_from_kv_cache by @fynnsu in #1072
• Move smoke tests into separate workflow by @fynnsu in #1073
• Update transformers version constraint in pyproject.toml by @dsikka in #1075
• perf(data): auto-size render endpoint concurrency for higher throughput by @WindChimeRan in #1013

New Contributors

• @jialefu made their first contribution in #875
• @cupkk made their first contribution in #925
• @qianlihuang made their first contribution in #962
• @PHOEBEMOON0802 made their first contribution in #993
• @zihanlin-ai made their first contribution in #1025
• @pavelgein made their first contribution in #1035

Full Changelog: v0.7.0...v0.8.0