Skip to content

v26.10

Latest

Choose a tag to compare

@shijieliu shijieliu released this 17 Nov 01:47
· 1 commit to main since this release
2995c41

This release adds a hierarchical input and output dist to DynamicEmb. Against TorchRec's dists, it shortens the training step of DynamicEmb's EmbeddingCollection benchmark by about 50%–80%. DynamicEmb also gains table-row-wise sharding, the FTRL optimizer, FP8 table storage, probabilistic admission, per-key scores, a background erase by score, and configurable exportable inference embedding collections that can live in GPU, host or Redis-backed storage and take incremental updates. HSTU inference adds layer-wise KV-cache loading with FlexKV and a Transformer backbone with FP8 DeepGEMM and FlashAttention 4. SID-GR inference gets faster default decode kernels contributed by Netflix. Two new examples join the repository, AI-Search-Plugin and Feature-Extractor-Plugin, and the container moves to NGC PyTorch 26.07. The release also includes 20 DynamicEmb fixes covering memory safety, lost keys, hangs and checkpoint loading.

🚀 Highlights

DynamicEmb: hierarchical sequence dist

With DynamicEmbTableOptions(output_dist="hierarchical", output_dist_capacity=...) (or "auto"), a row-wise sharded EmbeddingCollection of DynamicEmb tables gets its own input and output dist. It still works through TorchRec's sharding and awaitable interfaces.

  • NVLink inside a node, GPU-initiated InfiniBand between nodes. A node is NCCL's LSA team. On H100 systems that is an 8-GPU host. On GB200 and GB300 NVL72 with multi-node NVLink, it is the job's GPUs in one rack. Between nodes, the GPU itself sends over InfiniBand (NCCL GIN), always on its own rail.
  • Node dedup. A key requested from another node first goes over NVLink to a proxy GPU on the owner's rail. The proxy sends each (table, key) pair over InfiniBand once and receives its row once.
  • Few kernels, one host sync. The output dist is one forward kernel and one backward kernel. The input dist queues a whole call and syncs with the host once.
  • Same results as TorchRec. It gives the same forward rows, LFU counts and admission as TorchRec's dists. In the benchmark's --check runs with fp32 tables, outputs matched TorchRec's bit for bit at every scale.

The table compares the training step of DynamicEmb's EmbeddingCollection benchmark under TorchRec's dists (left) and the hierarchical dist (right). The benchmark uses 8 tables of 4M rows, dim 128, fp32, SGD, index dedup and power-law keys:

Features × samples per GPU (keys per sample and feature) Eos, 32 × H100 (4 nodes) GB300 NVL72, 16 GPUs (4 trays)
8 × 32k (1–4) 13.38 → 4.31 ms (−68%) 5.82 → 1.92 ms (−67%)
1k × 128 (1–4) 11.75 → 3.30 ms (−72%) 6.20 → 1.82 ms (−71%)
2k × 128 (1–4) 20.64 → 4.58 ms (−78%) 9.65 → 2.09 ms (−78%)
8 × 128 (1–8k) 31.25 → 14.94 ms (−52%) 11.23 → 5.13 ms (−54%)

Across all measured points (8 to 32 GPUs on Eos, 4 to 16 GPUs on GB300), the step is 48%–82% shorter. In HSTU end-to-end training on two Eos nodes, the hierarchical dist adds 1.1%–2.2% TFLOPS per GPU and cuts blocking host syncs from 27.1 to 15.6 per step.

Requirements:

  • DynamicEmb built against NCCL 2.30.7 or newer. Otherwise the build leaves the op out: "hierarchical" raises ImportError, and "auto" falls back to TorchRec's dists.
  • GPUs of compute capability 8.0 or newer.
  • Tables with dist_type roundrobin or hash_roundrobin.
  • NVLink between a node's GPUs. Otherwise "hierarchical" raises NVLinkCheckError, and "auto" falls back.

For multi-node jobs, set NCCL_CROSS_NIC=0. Before dropping a model, call dynamicemb.shard.quiesce(model) on every rank. Evaluation forwards use TorchRec's dists. See the design doc and the benchmark.

Related: #528, #550, #553, #554

DynamicEmb: table-row-wise sharding and a collection-aware planner

  • Table-row-wise sharding (#493): ShardingType.TABLE_ROW_WISE places a DynamicEmb table on one node and shards it over that node's GPUs. Pooled outputs are reduced inside the node, and only one result per node crosses the network, so cross-node output traffic drops from O(B·D·world_size) to O(B·D). Input keys still cross nodes, and each rank holds more of the table's rows. To request it, list ShardingType.TABLE_ROW_WISE in the table's sharding_types constraint or pass the table to get_planner(table_row_wise_embedding_table_names=...). The planner spreads these tables across nodes, and node_index pins a table to a specific node.
  • Tables are identified by collection and name (#504): two collections can now each declare a table with the same name, each with its own options. Use get_constraint_key(config, model) to address one of them. The planner also stopped pooling HBM budgets across collections, which could approve a model that does not fit. See Naming a table in constraints.
  • Per-table HBM budgets in fused groups (#534): each table in a fused group keeps its own local_hbm_for_values budget.

DynamicEmb: FTRL and FP8 tables

  • FTRL (#487, #500): DynamicEmbOptimType.FTRL implements FTRL-Proximal (McMahan et al., KDD 2013) with learning_rate_power, ftrl_beta, l1_reg and l2_reg. group_lasso_reg adds row-level group sparsity (G-FTRL). See FTRL.
  • FP8 table storage (#548, building on #499): embedding_dtype=torch.float8_e4m3fn stores one byte per weight, a quarter of fp32. Weights are stored as e4m3 codes of weight × 256, which covers ±1.75; larger values saturate. Every write rounds stochastically, so small optimizer steps are not lost. Only EXACT_ROWWISE_ADAGRAD is supported, and its accumulator stays in fp32. Checkpoints are written as fp16.
  • fp32 row-wise Adagrad state for 16-bit tables (#495): in fp16 tables, the row-wise Adagrad accumulator used to round to zero, so the learning rate never decayed. bf16 tables lost small increments. The accumulator is now kept in fp32 by default, and the new optimizer_state_dtype table option controls this.

DynamicEmb: admission, per-key scores and erase by score

  • Probabilistic admission (#488): ProbabilisticAdmissionStrategy(probability=...) admits a missing key with the given probability and keeps no counter. See AdmissionStrategy.

  • Per-key scores (#530): with DynamicEmbScoreStrategy.PRECISE_CUSTOMIZED, training input is a ScoredKeyedJaggedTensor that gives every key an int64 score. The scores decide which rows the table evicts. A key repeated within a batch keeps its largest score. This works for row-wise EmbeddingCollection and EmbeddingBagCollection. See Per-key score.

  • Background erase by score (#532, #536): dynamicemb.erase_by_score_async(model, score_strategy, threshold, ...) erases every key whose score is below a threshold while training continues, and returns an EraseHandle. Keys that in-flight batches are still using are spared.

    • It works on (TIMESTAMP, LFU) tables, using either score.
    • table_names= lets each table have its own threshold.
    • With strict=False, the same call works on every rank even when a table is not on every rank (table-row-wise).

    With a TIMESTAMP threshold taken from dynamicemb.device_timestamp(), this gives tables a time-to-live. On (NO_EVICTION, CUSTOMIZED) tables, it frees room so the table stops growing. See erase_by_score_async.

DynamicEmb: storage

  • A value row per key (#515): each key's value now lives in its own row, separate from its hash slot.
    • A rehash no longer moves values.
    • NO_EVICTION becomes a regular mode.
    • Row indices are int64, so tables can exceed 2^31 rows.
    • Incremental dump and replay carry (slot, row) pairs, so a replica reproduces the source's placement exactly.
  • Three storage tiers with caching (#511): with caching and an external storage, the tiers are cache → DynamicEmb storage → external storage (PS). The storage owns every key's score, and the cache ranks by recency only.

DynamicEmb: configurable exportable inference embeddings

DynamicEmb's exportable inference embedding collections are rebuilt around a configuration per collection (#478). create_inference_embedding_collection takes an InferenceEmbeddingCollectionConfig for each physical embedding collection instead of use_dynamic, and HSTU inference's conversion and export pass one too.

  • Indexers: FusedIdentityIndexer, LinearHashMapIndexer and BitConcatIndexer. BitConcatIndexer works only with hierarchical layers, and the model configuration fixes its table-ID and feature-ID bit widths.
  • NV-Embedding-Cache layers: gpu, linear_uvm, host, and hierarchical: a GPU cache, an optional host cache, and an NVHashMap or Redis parameter server.
  • Sidecars: data-backed indexer state and NVHashMap contents are stored outside the exported PT2 package. Python and C++ AOTI runtimes both load them.
  • Incremental updates: Redis-backed hierarchical tables take updates through an update coordinator and a subscriber in each runtime. The end-to-end example drives eager, Python AOTI and C++ AOTI runtimes from one coordinator.
  • Host memory and CPU-only serving: nve_layer_type="host" keeps embedding values in CPU memory, for CPU or CUDA callers, and HSTU's ranking exporters accept --host_embeddings. DYNAMICEMB_CPU_ONLY=1 builds the CPU inference bindings, the sidecar loader and the host update runtime: a deployment then needs CPU-only PyTorch, a driverless NV-Embedding-Cache and cudart, but no GPU or driver. See the HostLayer examples and the CPU-only build guide.
  • Packaging: with pynve installed, setup.py install and bdist_wheel also build and package the inference bindings. DYNAMICEMB_BUILD_INFERENCE=0 builds training only, and 1 fails if NV-Embedding-Cache is missing.

#478 verified these combinations:

Indexer NV-Embedding-Cache layer
FusedIdentityIndexer GPU
LinearHashMapIndexer LinearUVM
LinearHashMapIndexer Hierarchical, NVHashMap
LinearHashMapIndexer Hierarchical, Redis
BitConcatIndexer Hierarchical, NVHashMap
BitConcatIndexer Hierarchical, Redis

The tier bindings need NV-Embedding-Cache 26.06 or newer. Sum and mean pooling need 26.07 or newer, or the legacy 26.05; 26.06 is rejected. The image ships 26.10, plus 26.05 for compatibility. See Tier updates with stock NVE.

HSTU inference

  • Layer-wise KV-cache loading with FlexKV (#498): HSTU inference now uses FlexKV's layer-wise GET.

    • SSD→host and host→GPU transfers overlap with HSTU compute, and each layer waits only for its own KV.
    • The switch is is_layerwise / flexkv_enable_layerwise. In the benchmark, use --layerwise --layer-granularity N.
    • An optional prefetch_kvcache stages SSD blocks into host memory ahead of a GET.

    At batch size 1 with all KV on SSD (RTX PRO 6000), it lowers end-to-end latency at every history length. See the FlexKV cache benchmark.

  • Transformer backbone with FP8 DeepGEMM and FlashAttention 4 (#531, which includes #491): backbone="transformer" adds an opt-in Transformer (Pre-LN softmax attention, GELU FFN) alongside HSTU for ranking inference, Triton and AOTI export.

    • transformer_attention="fa4" replaces the padded SDPA reference with a single FlashAttention 4 varlen kernel.
    • transformer_fp8_linear=True runs the QKV, output and FFN projections as DeepGEMM FP8 GEMMs, with quantization fused into LayerNorm and GELU. Checkpoints stay bf16.

    The table shows the dense module's speedup over bf16 SDPA on an RTX PRO 6000 (3 layers, 100 candidates per user):

    Hidden Tokens bf16 SDPA FP8 FA4 FP8 + FA4
    512 16,380 10.78 ms 1.05× 2.48× 2.84×
    2048 4,095 11.16 ms 1.20× 1.63× 2.23×
    2048 16,380 38.86 ms 1.15× 1.83× 2.39×

    Both options need SM90 or newer and cannot be exported, so export mode rejects them. DeepGEMM is a new third_party submodule, and the image installs flash-attn-4==4.0.0b21. See the Transformer backbone guide.

  • Hopper paged HSTU attention (#510, #535): on SM90, paged (KV-cache) HSTU inference uses FBGEMM's CuTe DSL kernel. Training keeps the existing SM90 kernels.

SID-GR inference: faster decode, contributed by Netflix

Three new kernels are now the default. Each has a benchmark in examples/sid-gr-inference/benchmarks/:

  • Beam attention fused with its merge (#520): default on SM10x, bit-identical. At batch size 8 on B200 it is 2.4–3.3× faster than the two kernels it replaces, which makes the whole decode attention 1.3–1.7× faster. To keep the separate kernels, use --no-fused-beam-combine or fused_beam_combine=False.
  • Fused QK-norm + RoPE (#517): default on SM10x for bf16, head dim 128 and 24–32 q/k heads. Output matches within bf16 rounding. On B200 it is 1.15–1.7× faster than SGLang's two kernels from 1 to 65,536 tokens.
  • Vectorized beam-KV write (#513): default whenever K/V rows are 16-byte aligned, bit-identical. On B200 it is 2.6–3.7× faster than the PyTorch copy at batch size 8 and 1.3–1.7× faster at batch size 1. Set GR_INFERENCE_TRTLLM_BEAM_KV_WRITE_VECTORIZE=0 to turn it off.

New examples

  • AI-Search-Plugin (#539): an installable package of opt-in components that speed up RL post-training of search agents (AI search / deep research) on slime and verl. It also documents where training time goes. On five workloads, compared with each framework's fastest configuration, four train 1.17–1.56× faster (14%–36% less time) and one ties. This is an early release: the speedups were measured with the original implementation that the package was ported from, and the package's tests run on CPU only.
  • Feature-Extractor-Plugin (#540): GPU feature extraction for recommender data pipelines.
    • batch is a spark-rapids add-on. It fuses chains of row-independent expressions into one generated GPU kernel, with results bit-identical to spark-rapids.
    • streaming is a small Flink GPU operator library (window aggregation and bounded INNER lookup join) that jobs call from DataStream code.

Container and dependencies

  • Base image nvcr.io/nvidia/pytorch:26.07-py3 (was 26.05), and Triton Inference Server 26.07 with tritonclient 2.71.0 (#466).
  • FBGEMM GPU 1.8.0 (was 1.5.0) and TorchRec 1.8.0 (v1.8.0-rc1, was 1.5.0) (#466). FBGEMM GPU comes from the official CUDA 13.0 wheel on x86_64 and is built from source on arm64.
  • NV-Embedding-Cache 26.10, quack-kernels 0.4.1 on the image's CUTLASS DSL 4.5.2, and TorchX in its own virtualenv (#466).
  • New DeepGEMM submodule and FlashAttention 4 for the Transformer backbone (#531). The FBGEMM (HSTU) and FlexKV submodules are updated (#498, #510, #535).
  • DynamicEmb builds for GB300 (sm_103) by default when the CUDA compiler supports it (DEMB_ARCHS) (#528).
  • SID-GR training example: with the default backend, generate() now runs the KV-cache generate_beam_decode() path. The custom flash-attention fork with arbitrary masks is no longer needed (#466).

⚠️ Breaking changes and upgrade notes

DynamicEmb

  • get_planner moved to dynamicemb.planner: use from dynamicemb.planner import get_planner instead of dynamicemb.get_planner. DynamicEmbeddingEnumerator is removed. (#493)
  • Counter.load now requires dist_type, with optional shard_index and num_shards. Out-of-tree Counter implementations need the new signature. (#493)
  • A multi-rank load of a continuous table now raises NotImplementedError instead of silently keeping the wrong keys. (#493)
  • Checkpoint files are named after the table's own shard instead of the rank in a caller-supplied sub-group. Nothing changes for the default WORLD group. (#493)
  • The planner keys tables by TableKey(collection, name) (#504):
    • TableToPlace.name is now key, and NodePlacer.place returns Dict[TableKey, int].
    • per_rank_storage, optimizer_types and module_without_tables are keyed by TableKey.
    • fusion_key takes the owning collection as a trailing argument.
    • Two tables that share one DynamicEmbTableOptions object now raise an error instead of being silently mis-sized.
  • AdmissionStrategy is now configuration only. A custom strategy implements create_admitter(...) and returns a MultiTableAdmitter. DynamicEmbTableOptions.admission_counter is deprecated and warns; pass the counter to FrequencyAdmissionStrategy(threshold=..., counter=KVCounter(...)) instead. (#488)
  • Adam now ignores initial_accumulator_value and warns; it used to seed both moments. get/set_initial_optim_states is renamed to get/set_initial_optimizer_state. (#487)
  • Training numerics of fp16/bf16 tables can differ slightly from v26.08:
    • Row-wise Adagrad keeps its accumulator in fp32 by default. Set optimizer_state_dtype to the table dtype for the old behavior; older checkpoints still load. (#495)
    • fp16/bf16 tables no longer cast gradients down to the table dtype on the fused path. (#548)
  • Incremental dump changes (#515, #532):
    • For a row-storing table, slot_index and meta["erased_slot_index"] are [N, 2] int64 (slot, row) tensors.
    • incremental_dump skips NO_EVICTION tables, and replay_increment refuses them.
    • To make an erase replayable, use EvictedItemMode.RETAIN_KEY. Any other erase, including erase_by_score_async, marks the table erase_untracked. Its deltas are then refused until the replica is rebuilt from a full checkpoint.
  • dynamicemb.exportable_tables.create_inference_embedding_collection takes a keyword-only config: InferenceEmbeddingCollectionConfig instead of use_dynamic, and pooling_mode is now a required keyword. apply_inference_embedding_collection takes embedding_collection_configs, a mapping from table name to InferenceEmbeddingCollectionConfig, instead of dynamic_table_configs. (#478)
  • Rebuild the DynamicEmb extension: its C++ and CUDA sources changed in this release.

HSTU inference

  • In the pinned FlexKV, ModelConfig.use_mla is read-only. Set kv_dim instead (2 for separate K and V, equivalent to the old use_mla=False). (#537)

🐛 Bug fixes

DynamicEmb: memory safety

  • CUB scratch buffers were freed before CUB used them; they are now kept alive through selection. (#489)
  • When a table expanded to a size that is not a multiple of the bucket size, the value buffer was smaller than the hash capacity. It now matches the rounded-up capacity. (#490)
  • A VMMTensor now waits for queued GPU work before unmapping its memory, for example when a model is dropped right after its last step. (#526)
  • Offsets are now computed in 64 bits for pooled outputs above 2^31 elements (#555) and for optimizer kernels past 2^32 elements (#548).
  • (TIMESTAMP, LFU) tables no longer hit a device assertion when a checkpoint load, rehash or backward refresh inserts into a full bucket. (#508)

DynamicEmb: lost or wrong rows

  • Table expansion is sized to max_load_factor. Growing from init_capacity, for example at checkpoint load, used to keep only 94%–97% of the keys. (#516)
  • Reference counters are indexed per logical table, so an insert into one table can no longer evict another table's in-use keys. (#514)
  • In a prefetch pipeline, prefetched keys keep their reference counts through a rehash. (#529)
  • Training-mode forwards under torch.no_grad() now release their slot pins. (#547)
  • When the cache's insert refuses a key, DynamicEmb now raises an error instead of writing uninitialized memory back to storage. Reference-count updates are also atomic across streams. (#557)
  • Erasing a key clears all of its score words, so a reclaimed slot in a (TIMESTAMP, LFU) table is reused before live keys are evicted. (#523)
  • int32 keys behave like the int64 keys a table stores, for every dist_type and under index dedup. (#544)
  • Non-admitted rows are initialized exactly once, and the admission strategy's initializer now also applies on the prefetch paths. (#488)
  • replay_increment rechecks the layout after a cache flush, before it writes anything. (#496)

DynamicEmb: hangs, planning and checkpoints

  • Under index dedup, ranks with no keys now send empty frequency weights. Previously the KJT all-to-all could hang. (#527)
  • The planner no longer pools HBM budgets across collections, and it costs a group that fits in HBM as HBM-only. (#504)
  • Each table in a fused group keeps its own local_hbm_for_values budget. (#534)
  • A checkpoint load lists the checkpoint directory once instead of six times per table: resolving the files of 2,000 tables went from 539 s to 1.26 s. Warnings about state the caller did not request are gone. (#501)
  • Tables named <scope>/<name>, which is how TorchRec names the tables of fused feature groups, work with the new listing: dump() creates their subdirectory, and load() finds them with one listing per directory. (#509)

HSTU and inference

  • Tensor-parallel LayerNorm handles a rank whose evaluation batch is empty. (#556)
  • FlexKV no longer starts a CUDA MPS daemon, which broke CUDA for other processes on shared hosts. (#556)
  • A dense-module forward without the KV cache no longer replays the cache's CUDA graphs with the last cached request's metadata. (#531)
  • HSTU retrieval's dot-product similarity uses a backward that is safe with 64-bit indices on PyTorch 26.07. (#466)

📦 DynamicEmb wheel

A prebuilt dynamicemb-0.2.0-cp312-cp312-linux_x86_64.whl is attached to this release. DynamicEmb's package version is now 0.2.0; it was 0.1.0 from v26.03 through v26.08.

  • sha256: 3be2cb06360b05072b050af5b1705c6c536efc69b0b62a7289bd1beb26873183
  • Source: this release's DynamicEmb source.
  • Build environment: the repository's CI image on NGC PyTorch 26.07, with torch 2.13.0a0+9186a08, CUDA 13.3, NCCL 2.30.7 and Python 3.12, on x86_64.
  • CUDA targets: sm_75, sm_80, sm_90, sm_100, sm_103 (GB300) and sm_120.
  • Includes the hierarchical dist and the bindings for exportable inference embedding collections.

The extension links this exact libtorch, so install the wheel only on that torch build, for example inside nvcr.io/nvidia/pytorch:26.07-py3 or this repository's image, with TorchRec 1.8.0 and FBGEMM GPU 1.8.0:

pip install --no-deps dynamicemb-0.2.0-cp312-cp312-linux_x86_64.whl

At run time, the hierarchical dist needs NCCL 2.30.7 or newer and pynvml. The inference bindings need NV-Embedding-Cache 26.10's pynve, as installed in this repository's image. Custom eviction score functions need numba. For arm64 (for example, GB300), build from source with DEMB_ARCHS=103 python setup.py install in corelib/dynamicemb.

What's Changed

DynamicEmb: features

  • Support probabilistic admission strategy in DynamicEmb by @jiashuy in #488
  • Support FTRL optimizer in DynamicEmb by @jiashuy in #487
  • Support table-row-wise sharding in DynamicEmb by @jiashuy in #493
  • FTRL group sparsity in DynamicEmb by @jiashuy in #500
  • Cache serves as a subset of storage; external storage sits below storage by @jiashuy in #511
  • Store a value row for every key in LinearBucketTable and DynamicEmbStorage by @jiashuy in #515
  • Per-key score input through the PRECISE_CUSTOMIZED score strategy by @jiashuy in #530
  • Async erase by score by @jiashuy in #532
  • erase_by_score for lru_lfu tables, per named table by @jyuan-sc in #536
  • Hierarchical dist for row-wise sequence sharding by @shijieliu in #528
  • FP8 table storage with stochastic rounding for row-wise Adagrad by @jiashuy in #548 (building on #499 by @jyuan-sc)

DynamicEmb: fixes

  • Retain CUB scratch tensors through selection by @lidongdongbuaa in #489
  • Match expanded values to aligned hash capacity by @lidongdongbuaa in #490
  • Recheck replay layout after cache flush by @efegokdemir in #496
  • Keep row-wise Adagrad accumulator in fp32 for 16-bit tables by @jyuan-sc in #495
  • List the checkpoint directory once per load, and quiet load/dump warnings by @tiankongdeguiji in #501
  • Plan per EmbeddingCollection; identify a table by its collection and its name by @jiashuy in #504
  • Load tables named <scope>/<name> without listing per lookup by @tiankongdeguiji in #509
  • Size table expansion to max_load_factor by @nthan-sc in #516
  • Evict LruLfu tables through the ranked evictor on every insert path by @jyuan-sc in #508
  • Clear every score word in table_erase_kernel by @jiashuy in #523
  • Index reference counters across logical tables by @lidongdongbuaa in #514
  • Keep each table's local_hbm_for_values budget in fused groups by @tiankongdeguiji in #534
  • Keep prefetched keys' ref counts through a rehash by @shijieliu in #529
  • Send frequency weights for an empty input under index dedup by @shijieliu in #527
  • Wait for queued GPU work before freeing VMMTensor memory by @shijieliu in #526
  • Release slot pins after no-grad training forwards by @jiashuy in #547 (original fix by @lidongdongbuaa in #541)
  • Treat int32 keys as the int64 keys a table stores, under every dist_type and index dedup by @jiashuy in #544
  • The hierarchical dist takes a node only when NVLink connects its GPUs by @shijieliu in #553
  • Fix int32 overflow in get_dst_ptr by @lzw3232 in #555
  • The cache raises on keys its insert refuses; pins add up across streams by @shijieliu in #557

HSTU and inference

  • Pin FBGEMM Hopper SM90 paged HSTU DSL by @Clebrate in #510
  • Layer-wise KV-cache loading with FlexKV (Layerwise Design) by @Clebrate in #498
  • Set kv_dim instead of assigning ModelConfig.use_mla by @Clebrate in #537
  • Use the Hopper DSL for paged inference only by @Clebrate in #535
  • Support configurable indexing and NVE layers for exportable embedding collections by @geoffreyQiu in #478
  • CI follow-ups to #478: TP layer norm on an empty batch; FlexKV without MPS by @shijieliu in #556
  • FP8 DeepGEMM and FA4 attention for Transformer inference by @z52527 in #531 (includes the Transformer backbone from #491 by @Junyi-Zheng)

SID-GR inference

  • Fuse beam attention and merge, default on SM10x (up to 3.3×) by @bellanich in #520
  • Fuse QK-norm + RoPE and make it the default on SM10x (up to 1.7×) by @bellanich in #517
  • Vectorize the beam-KV write and make it the default (up to 3.7×) by @bellanich in #513

New examples

  • AI-Search-Plugin: drop-in components for search-agent RL post-training on slime and verl by @cb521 in #539
  • Feature-Extractor-Plugin: Spark expression fusion (batch) and Flink GPU operators (streaming) by @cb521 in #540

Build, CI and container

  • Update base image to 26.07 by @shijieliu in #466
  • Move the hierarchical dist's tests into their own CI groups by @shijieliu in #550
  • test_hier_dist no longer checks a node that is not NCCL's team by @shijieliu in #554

🙌 New contributors

Thanks to everyone who contributed to this release: @bellanich, @cb521, @Clebrate, @efegokdemir, @geoffreyQiu, @jiashuy, @Junyi-Zheng, @jyuan-sc, @lidongdongbuaa, @lzw3232, @nthan-sc, @shijieliu, @tiankongdeguiji and @z52527.

Full changelog: v26.08...v26.10