Skip to content

Speculators v0.9.0

Latest
Latest

Choose a tag to compare

View all tags
@dhuangnm dhuangnm released this 19 Oct 13:53
· 13 commits to main since this release
v0.9.0
1c800f2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
spec_v09

This release builds on the v0.8.0 foundation with key performance optimizations across data loading and Mooncake connectors, enhanced evaluation and profiling utilities, automated evaluation dashboard integration, support for GLM 5.3 hidden-state extraction and DFlash 2 strict candidate selection, and the deprecation of Python 3.10 support (PR #1242).

Key Features

  • Mooncake transfer manifest returned directly through kv_transfer_params for simplified tensor packing and improved transfer performance.
  • Integration of torch.profiler into the evaluation benchmark suite to provide detailed GPU kernel breakdowns.
  • InferenceX-style throughput vs. interactivity benchmark utility to evaluate GPU performance across concurrency levels.
  • Dataloader throughput improvements via simplified finiteness checks and removal of noise transforms.
  • Automated evaluation dashboard integration replacing static documentation tables.
  • Deprecation of Python 3.10 support across the codebase and build infrastructure (PR #1242).

Mooncake & Connectors

  • Return transfer manifest through kv_transfer_params directly instead of storing inside byte payload, simplifying tensor packing logic and improving queue transfer performance (PR #1192).
  • Pack transfer tensors into a single mooncake buffer (PR #1166)
  • GLM 5.3 hidden states extraction backend support merged for feature extraction pipelines.

Evaluation, Benchmarking & Dashboard

  • Torch profiler integration: Added torch.profiler support to the evaluation benchmark suite for detailed GPU kernel breakdowns and latency metrics (PRs #1122-1124).
  • Fused loss kernel optimizations yielding an up to 2x forward training pass speed up for small (~1B) models (PR #1111).
  • Throughput vs. Interactivity benchmark: Introduced an InferenceX-style benchmark utility (scripts/evaluate/throughput_interactivity.py) with example scripts for DSpark and MTP configurations (PR #1204).
  • Live evaluation dashboard: Completed automated evaluations and linked documentation directly to the live Speculators evaluation dashboard.
  • MRCR benchmark added to (scripts/evaluate/evaluate.py) (PR #1167).

Data Generation & Pipeline Performance

  • Dataloader performance optimizations: Simplified finiteness check logic and removed noise transforms to accelerate data loading speed (PR #1211).
  • FP8 hidden-state storage: Added an FP8-quantized hidden-state backend, reducing storage requirements by 50% while maintaining speculator quality (PR #1028).
  • Deprecate and remove hidden states noise injection, following experiments showing it doesn’t impact final model quality and slows training loop.
  • Token frequency calculation is optional, turning it off for full vocab size training would reduce preprocessing time
  • Added reasoning effort and temperature distribution support.
  • Deprecated hybrid training mode

Python 3.10 Deprecation

  • Deprecated Python 3.10 support across the codebase and build infrastructure in alignment with release toolchain requirements (PR #1242).

Documentation

  • Replaced static supported models table in documentation with direct link to the automated Speculators evaluation dashboard (PR #1201).

Full Changelog

  • feat(launch_vllm): re-add vLLM provenance via generated sibling module by @orestis-z in #1043
  • Support truncating on the vLLM server during rendering by @fynnsu in #1079
  • enable vLLM render endpoints by @shanjiaz in #1080
  • Fix test monkeypatch by @fynnsu in #1086
  • Make vocab mapping usage more explicit by @fynnsu in #1081
  • fix(dflash): unify per-block accepted-length metrics by @WindChimeRan in #1032
  • feat(launch_vllm): add eval subcommand for spec-decode serving by @orestis-z in #959
  • feat(train): write speculators.patch alongside train_command.txt by @orestis-z in #960
  • Expose all draft models from the package root by @rahul-tuli in #1085
  • fix(eval): sum per-position acceptance counters across engines by @LOGO127 in #1096
  • feat(hs-connectors): add FP8-quantized hidden-states backend by @shubhra in #1028
  • fix(data): skip chat template check when render endpoint is set by @stormchasingg in #1103
  • Link to multi-node training and python api reference by @fynnsu in #1101
  • fix(train): use max_steps as scheduler total when set by @Leslie360 in #1104
  • Patch Qwen3.5 3D M-Rope handling by @fynnsu in #1108
  • fix(models): resolve verifier_norm class by verifier family by @minziyu in #1090
  • fix(utils): isolate inherited class registries by @edgeai1 in #1083
  • add gradient checkpointing for dflash/dspark by @walterchenchn in #1100
  • Fix eager-path draft-token mask extension inverting additive mask semantics by @khazic in #897
  • perf(losses): autotune the fused-loss tile instead of a fixed 131072 block by @orestis-z in #1111
  • Add spec-dec demo with identical text and proper metrics by @eldarkurtic in #1121
  • Add Buildkite nightly pipeline for unit and integration tests by @rashmigottipati in #1087
  • Add dco failure label and comment by @fynnsu in #1128
  • perf(train): batch metric reduction into one all_reduce by @ys2025-AI in #1119
  • fix(train): restore signal handlers on every exit from the decorator by @dchaudhari7177 in #1097
  • Add launch vllm enable scale out flag by @shanjiaz in #1117
  • fix(train): use float32 for error_records metrics to support NPU hccl by @zhujiem in #1116
  • Fix dco failure discovery by @fynnsu in #1134
  • docs: drop stale assistant_tokens_mask references by @WindChimeRan in #1146
  • Update learning rate defaults by @fynnsu in #1148
  • fix(eval): select train split for local JSON datasets by @RainieLLM in #1150
  • fix(evaluate): support nested gen_kwargs in guidellm --backend by @khushali9 in #1149
  • fix(hs_connectors): support non-CUDA accelerators in the Mooncake bac… by @ys2025-AI in #1133
  • skip token frequencey calculation by @shanjiaz in #1153
  • test: pin Muon learning rate in sanity check by @rahul-tuli in #1154
  • Add Hugging Face JSONL dataset URI support by @fynnsu in #1147
  • test(cli): stabilize help rendering width by @rahul-tuli in #1155
  • Added reasoning effort and temperature distribution support for regen by @shanjiaz in #1152
  • Treat layer_types and sliding_window as optional in DFlash by @Hyungkeun-Park in #1141
  • Warn when a cached token frequency distribution is silently reused by @khazic in #903
  • fix(loading): resolve Mamba-backbone verifier embedding key by @ra-srivid in #1158
  • test(hs-connectors): warm up engine before FP8 roundtrip e2e request by @shubhra in #1160
  • Spawn mooncake clients on dataloader's associated rank by @fynnsu in #1168
  • feat(train): use global values for heterogeneous verifier configs by @fynnsu in #1163
  • fix(loading): resolve Mamba-backbone verifier final-norm key by @ra-srivid in #1165
  • Check DFlash2 layers by capability, not by exact type by @Hyungkeun-Park in #1140
  • feat(hs_connectors): add --mooncake-device transport selection by @qianlihuang in #1174
  • fix: keep DSpark confidence loss on fixed decay by @RainieLLM in #1172
  • Remove hybrid mode by @shanjiaz in #1162
  • fix(train): pass the training seed to the batch sampler by @qianlihuang in #1173
  • perf(hs-connectors): zero-copy Mooncake transfer with checked status codes by @xiaguan in #1166
  • feat(evaluate): add MRCR long-context benchmark support by @reed-meyerson in #1167
  • Remove hybrid references from docs by @shanjiaz in #1179
  • feat(benchmark): add trainer timing marks, memory tracking, and statistical tests by @orestis-z in #1122
  • build(deps-dev): update pytest-mock requirement from ~=3.15.1 to ~=3.16.0 by @dependabot[bot] in #1182
  • build(deps-dev): update pymdown-extensions requirement from ~=11.0.0 to ~=12.1.0 by @dependabot[bot] in #1183
  • build(deps-dev): update setuptools requirement from >=61.0 to >=84.0.0 by @dependabot[bot] in #1184
  • build(deps-dev): update build requirement from >=1.5.0 to >=1.6.1 by @dependabot[bot] in #1186
  • build(deps-dev): update mooncake-transfer-engine-cuda13 requirement from >=0.3.12 to >=0.3.13.post1 by @dependabot[bot] in #1185
  • build(deps-dev): update pygments requirement from <2.20.0 to <2.22.0 by @dependabot[bot] in #1187
  • build(deps-dev): update mkdocstrings-python requirement from ~=2.0.5 to ~=2.0.9 by @dependabot[bot] in #1188
  • build(deps-dev): update pre-commit requirement from ~=4.0 to ~=4.6 by @dependabot[bot] in #1189
  • build(deps-dev): update mkdocs-material requirement from ~=9.7.6 to ~=9.7.7 by @dependabot[bot] in #1190
  • build(deps-dev): update pytest-rerunfailures requirement from ~=16.4 to ~=16.7 by @dependabot[bot] in #1191
  • [Refactor][Data] Reject raw presets in prepare-data [1/n] by @WindChimeRan in #1170
  • Fix speculators build error found in downstream source build by @dhuangnm in #1193
  • Add XPress: a causal refiner over the DFlash block drafter by @ZKBig in #1061
  • [Mooncake] Return transfer manifest through kv_transfer_params by @fynnsu in #1192
  • Dataloader perf improvements (finiteness checks & noise transforms) by @fynnsu in #1211
  • feat(benchmark): add torch.profiler support and GPU kernel breakdown by @orestis-z in #1123
  • Add a throughput vs. interactivity benchmark (InferenceX-style) with a DSpark and MTP example by @eldarkurtic in #1204
  • [Bugfix] Make offline data generation robust to truncated files by @DarkLight1337 in #1219
  • Remove deprecated script compatibility shims by @rahul-tuli in #1228
  • feat(benchmark): add HTML and Markdown report generation by @orestis-z in #1124
  • docs: Link dashboard and remove supported models table by @rahul-tuli in #1201
  • Update AGENTS.md to avoid AI bloat by @orestis-z in #1235
  • Deprecate Python 3.10 support by @rahul-tuli in #1242
  • [ReadMe] Update repo readme plots by @dsikka in #1243
  • Update dependency upper bounds by @dhuangnm in #1236

New Contributors

Full Changelog: v0.8.0...v0.9.0

Contributors

shubhra, rashmigottipati, and 28 other contributors
Assets 6
Loading