Repository navigation
This release adds a hierarchical input and output dist to DynamicEmb. Against TorchRec's dists, it shortens the training step of DynamicEmb's EmbeddingCollection benchmark by about 50%–80%. DynamicEmb also gains table-row-wise sharding, the FTRL optimizer, FP8 table storage, probabilistic admission, per-key scores, a background erase by score, and configurable exportable inference embedding collections that can live in GPU, host or Redis-backed storage and take incremental updates. HSTU inference adds layer-wise KV-cache loading with FlexKV and a Transformer backbone with FP8 DeepGEMM and FlashAttention 4. SID-GR inference gets faster default decode kernels contributed by Netflix. Two new examples join the repository, AI-Search-Plugin and Feature-Extractor-Plugin, and the container moves to NGC PyTorch 26.07. The release also includes 20 DynamicEmb fixes covering memory safety, lost keys, hangs and checkpoint loading.
🚀 Highlights
DynamicEmb: hierarchical sequence dist
With DynamicEmbTableOptions(output_dist="hierarchical", output_dist_capacity=...) (or "auto"), a row-wise sharded EmbeddingCollection of DynamicEmb tables gets its own input and output dist. It still works through TorchRec's sharding and awaitable interfaces.
- NVLink inside a node, GPU-initiated InfiniBand between nodes. A node is NCCL's LSA team. On H100 systems that is an 8-GPU host. On GB200 and GB300 NVL72 with multi-node NVLink, it is the job's GPUs in one rack. Between nodes, the GPU itself sends over InfiniBand (NCCL GIN), always on its own rail.
- Node dedup. A key requested from another node first goes over NVLink to a proxy GPU on the owner's rail. The proxy sends each (table, key) pair over InfiniBand once and receives its row once.
- Few kernels, one host sync. The output dist is one forward kernel and one backward kernel. The input dist queues a whole call and syncs with the host once.
- Same results as TorchRec. It gives the same forward rows, LFU counts and admission as TorchRec's dists. In the benchmark's
--checkruns with fp32 tables, outputs matched TorchRec's bit for bit at every scale.
The table compares the training step of DynamicEmb's EmbeddingCollection benchmark under TorchRec's dists (left) and the hierarchical dist (right). The benchmark uses 8 tables of 4M rows, dim 128, fp32, SGD, index dedup and power-law keys:
| Features × samples per GPU (keys per sample and feature) | Eos, 32 × H100 (4 nodes) | GB300 NVL72, 16 GPUs (4 trays) |
|---|---|---|
| 8 × 32k (1–4) | 13.38 → 4.31 ms (−68%) | 5.82 → 1.92 ms (−67%) |
| 1k × 128 (1–4) | 11.75 → 3.30 ms (−72%) | 6.20 → 1.82 ms (−71%) |
| 2k × 128 (1–4) | 20.64 → 4.58 ms (−78%) | 9.65 → 2.09 ms (−78%) |
| 8 × 128 (1–8k) | 31.25 → 14.94 ms (−52%) | 11.23 → 5.13 ms (−54%) |
Across all measured points (8 to 32 GPUs on Eos, 4 to 16 GPUs on GB300), the step is 48%–82% shorter. In HSTU end-to-end training on two Eos nodes, the hierarchical dist adds 1.1%–2.2% TFLOPS per GPU and cuts blocking host syncs from 27.1 to 15.6 per step.
Requirements:
- DynamicEmb built against NCCL 2.30.7 or newer. Otherwise the build leaves the op out:
"hierarchical"raisesImportError, and"auto"falls back to TorchRec's dists. - GPUs of compute capability 8.0 or newer.
- Tables with
dist_typeroundrobinorhash_roundrobin. - NVLink between a node's GPUs. Otherwise
"hierarchical"raisesNVLinkCheckError, and"auto"falls back.
For multi-node jobs, set NCCL_CROSS_NIC=0. Before dropping a model, call dynamicemb.shard.quiesce(model) on every rank. Evaluation forwards use TorchRec's dists. See the design doc and the benchmark.
Related: #528, #550, #553, #554
DynamicEmb: table-row-wise sharding and a collection-aware planner
- Table-row-wise sharding (#493):
ShardingType.TABLE_ROW_WISEplaces a DynamicEmb table on one node and shards it over that node's GPUs. Pooled outputs are reduced inside the node, and only one result per node crosses the network, so cross-node output traffic drops from O(B·D·world_size) to O(B·D). Input keys still cross nodes, and each rank holds more of the table's rows. To request it, listShardingType.TABLE_ROW_WISEin the table'ssharding_typesconstraint or pass the table toget_planner(table_row_wise_embedding_table_names=...). The planner spreads these tables across nodes, andnode_indexpins a table to a specific node. - Tables are identified by collection and name (#504): two collections can now each declare a table with the same name, each with its own options. Use
get_constraint_key(config, model)to address one of them. The planner also stopped pooling HBM budgets across collections, which could approve a model that does not fit. See Naming a table inconstraints. - Per-table HBM budgets in fused groups (#534): each table in a fused group keeps its own
local_hbm_for_valuesbudget.
DynamicEmb: FTRL and FP8 tables
- FTRL (#487, #500):
DynamicEmbOptimType.FTRLimplements FTRL-Proximal (McMahan et al., KDD 2013) withlearning_rate_power,ftrl_beta,l1_regandl2_reg.group_lasso_regadds row-level group sparsity (G-FTRL). See FTRL. - FP8 table storage (#548, building on #499):
embedding_dtype=torch.float8_e4m3fnstores one byte per weight, a quarter of fp32. Weights are stored as e4m3 codes ofweight × 256, which covers ±1.75; larger values saturate. Every write rounds stochastically, so small optimizer steps are not lost. OnlyEXACT_ROWWISE_ADAGRADis supported, and its accumulator stays in fp32. Checkpoints are written as fp16. - fp32 row-wise Adagrad state for 16-bit tables (#495): in fp16 tables, the row-wise Adagrad accumulator used to round to zero, so the learning rate never decayed. bf16 tables lost small increments. The accumulator is now kept in fp32 by default, and the new
optimizer_state_dtypetable option controls this.
DynamicEmb: admission, per-key scores and erase by score
-
Probabilistic admission (#488):
ProbabilisticAdmissionStrategy(probability=...)admits a missing key with the given probability and keeps no counter. See AdmissionStrategy. -
Per-key scores (#530): with
DynamicEmbScoreStrategy.PRECISE_CUSTOMIZED, training input is aScoredKeyedJaggedTensorthat gives every key an int64 score. The scores decide which rows the table evicts. A key repeated within a batch keeps its largest score. This works for row-wiseEmbeddingCollectionandEmbeddingBagCollection. See Per-key score. -
Background erase by score (#532, #536):
dynamicemb.erase_by_score_async(model, score_strategy, threshold, ...)erases every key whose score is below a threshold while training continues, and returns anEraseHandle. Keys that in-flight batches are still using are spared.- It works on
(TIMESTAMP, LFU)tables, using either score. table_names=lets each table have its own threshold.- With
strict=False, the same call works on every rank even when a table is not on every rank (table-row-wise).
With a
TIMESTAMPthreshold taken fromdynamicemb.device_timestamp(), this gives tables a time-to-live. On(NO_EVICTION, CUSTOMIZED)tables, it frees room so the table stops growing. See erase_by_score_async. - It works on
DynamicEmb: storage
- A value row per key (#515): each key's value now lives in its own row, separate from its hash slot.
- A rehash no longer moves values.
NO_EVICTIONbecomes a regular mode.- Row indices are int64, so tables can exceed 2^31 rows.
- Incremental dump and replay carry (slot, row) pairs, so a replica reproduces the source's placement exactly.
- Three storage tiers with caching (#511): with caching and an external storage, the tiers are cache → DynamicEmb storage → external storage (PS). The storage owns every key's score, and the cache ranks by recency only.
DynamicEmb: configurable exportable inference embeddings
DynamicEmb's exportable inference embedding collections are rebuilt around a configuration per collection (#478). create_inference_embedding_collection takes an InferenceEmbeddingCollectionConfig for each physical embedding collection instead of use_dynamic, and HSTU inference's conversion and export pass one too.
- Indexers:
FusedIdentityIndexer,LinearHashMapIndexerandBitConcatIndexer.BitConcatIndexerworks only with hierarchical layers, and the model configuration fixes its table-ID and feature-ID bit widths. - NV-Embedding-Cache layers:
gpu,linear_uvm,host, andhierarchical: a GPU cache, an optional host cache, and an NVHashMap or Redis parameter server. - Sidecars: data-backed indexer state and NVHashMap contents are stored outside the exported PT2 package. Python and C++ AOTI runtimes both load them.
- Incremental updates: Redis-backed hierarchical tables take updates through an update coordinator and a subscriber in each runtime. The end-to-end example drives eager, Python AOTI and C++ AOTI runtimes from one coordinator.
- Host memory and CPU-only serving:
nve_layer_type="host"keeps embedding values in CPU memory, for CPU or CUDA callers, and HSTU's ranking exporters accept--host_embeddings.DYNAMICEMB_CPU_ONLY=1builds the CPU inference bindings, the sidecar loader and the host update runtime: a deployment then needs CPU-only PyTorch, a driverless NV-Embedding-Cache andcudart, but no GPU or driver. See the HostLayer examples and the CPU-only build guide. - Packaging: with
pynveinstalled,setup.py installandbdist_wheelalso build and package the inference bindings.DYNAMICEMB_BUILD_INFERENCE=0builds training only, and1fails if NV-Embedding-Cache is missing.
#478 verified these combinations:
| Indexer | NV-Embedding-Cache layer |
|---|---|
FusedIdentityIndexer |
GPU |
LinearHashMapIndexer |
LinearUVM |
LinearHashMapIndexer |
Hierarchical, NVHashMap |
LinearHashMapIndexer |
Hierarchical, Redis |
BitConcatIndexer |
Hierarchical, NVHashMap |
BitConcatIndexer |
Hierarchical, Redis |
The tier bindings need NV-Embedding-Cache 26.06 or newer. Sum and mean pooling need 26.07 or newer, or the legacy 26.05; 26.06 is rejected. The image ships 26.10, plus 26.05 for compatibility. See Tier updates with stock NVE.
HSTU inference
-
Layer-wise KV-cache loading with FlexKV (#498): HSTU inference now uses FlexKV's layer-wise GET.
- SSD→host and host→GPU transfers overlap with HSTU compute, and each layer waits only for its own KV.
- The switch is
is_layerwise/flexkv_enable_layerwise. In the benchmark, use--layerwise --layer-granularity N. - An optional
prefetch_kvcachestages SSD blocks into host memory ahead of a GET.
At batch size 1 with all KV on SSD (RTX PRO 6000), it lowers end-to-end latency at every history length. See the FlexKV cache benchmark.
-
Transformer backbone with FP8 DeepGEMM and FlashAttention 4 (#531, which includes #491):
backbone="transformer"adds an opt-in Transformer (Pre-LN softmax attention, GELU FFN) alongside HSTU for ranking inference, Triton and AOTI export.transformer_attention="fa4"replaces the padded SDPA reference with a single FlashAttention 4 varlen kernel.transformer_fp8_linear=Trueruns the QKV, output and FFN projections as DeepGEMM FP8 GEMMs, with quantization fused into LayerNorm and GELU. Checkpoints stay bf16.
The table shows the dense module's speedup over bf16 SDPA on an RTX PRO 6000 (3 layers, 100 candidates per user):
Hidden Tokens bf16 SDPA FP8 FA4 FP8 + FA4 512 16,380 10.78 ms 1.05× 2.48× 2.84× 2048 4,095 11.16 ms 1.20× 1.63× 2.23× 2048 16,380 38.86 ms 1.15× 1.83× 2.39× Both options need SM90 or newer and cannot be exported, so export mode rejects them. DeepGEMM is a new
third_partysubmodule, and the image installsflash-attn-4==4.0.0b21. See the Transformer backbone guide. -
Hopper paged HSTU attention (#510, #535): on SM90, paged (KV-cache) HSTU inference uses FBGEMM's CuTe DSL kernel. Training keeps the existing SM90 kernels.
SID-GR inference: faster decode, contributed by Netflix
Three new kernels are now the default. Each has a benchmark in examples/sid-gr-inference/benchmarks/:
- Beam attention fused with its merge (#520): default on SM10x, bit-identical. At batch size 8 on B200 it is 2.4–3.3× faster than the two kernels it replaces, which makes the whole decode attention 1.3–1.7× faster. To keep the separate kernels, use
--no-fused-beam-combineorfused_beam_combine=False. - Fused QK-norm + RoPE (#517): default on SM10x for bf16, head dim 128 and 24–32 q/k heads. Output matches within bf16 rounding. On B200 it is 1.15–1.7× faster than SGLang's two kernels from 1 to 65,536 tokens.
- Vectorized beam-KV write (#513): default whenever K/V rows are 16-byte aligned, bit-identical. On B200 it is 2.6–3.7× faster than the PyTorch copy at batch size 8 and 1.3–1.7× faster at batch size 1. Set
GR_INFERENCE_TRTLLM_BEAM_KV_WRITE_VECTORIZE=0to turn it off.
New examples
- AI-Search-Plugin (#539): an installable package of opt-in components that speed up RL post-training of search agents (AI search / deep research) on slime and verl. It also documents where training time goes. On five workloads, compared with each framework's fastest configuration, four train 1.17–1.56× faster (14%–36% less time) and one ties. This is an early release: the speedups were measured with the original implementation that the package was ported from, and the package's tests run on CPU only.
- Feature-Extractor-Plugin (#540): GPU feature extraction for recommender data pipelines.
Container and dependencies
- Base image
nvcr.io/nvidia/pytorch:26.07-py3(was 26.05), and Triton Inference Server 26.07 withtritonclient2.71.0 (#466). - FBGEMM GPU 1.8.0 (was 1.5.0) and TorchRec 1.8.0 (
v1.8.0-rc1, was 1.5.0) (#466). FBGEMM GPU comes from the official CUDA 13.0 wheel on x86_64 and is built from source on arm64. - NV-Embedding-Cache 26.10,
quack-kernels0.4.1 on the image's CUTLASS DSL 4.5.2, and TorchX in its own virtualenv (#466). - New DeepGEMM submodule and FlashAttention 4 for the Transformer backbone (#531). The FBGEMM (HSTU) and FlexKV submodules are updated (#498, #510, #535).
- DynamicEmb builds for GB300 (
sm_103) by default when the CUDA compiler supports it (DEMB_ARCHS) (#528). - SID-GR training example: with the default backend,
generate()now runs the KV-cachegenerate_beam_decode()path. The custom flash-attention fork with arbitrary masks is no longer needed (#466).
⚠️ Breaking changes and upgrade notes
DynamicEmb
get_plannermoved todynamicemb.planner: usefrom dynamicemb.planner import get_plannerinstead ofdynamicemb.get_planner.DynamicEmbeddingEnumeratoris removed. (#493)Counter.loadnow requiresdist_type, with optionalshard_indexandnum_shards. Out-of-treeCounterimplementations need the new signature. (#493)- A multi-rank load of a
continuoustable now raisesNotImplementedErrorinstead of silently keeping the wrong keys. (#493) - Checkpoint files are named after the table's own shard instead of the rank in a caller-supplied sub-group. Nothing changes for the default
WORLDgroup. (#493) - The planner keys tables by
TableKey(collection, name)(#504):TableToPlace.nameis nowkey, andNodePlacer.placereturnsDict[TableKey, int].per_rank_storage,optimizer_typesandmodule_without_tablesare keyed byTableKey.fusion_keytakes the owning collection as a trailing argument.- Two tables that share one
DynamicEmbTableOptionsobject now raise an error instead of being silently mis-sized.
AdmissionStrategyis now configuration only. A custom strategy implementscreate_admitter(...)and returns aMultiTableAdmitter.DynamicEmbTableOptions.admission_counteris deprecated and warns; pass the counter toFrequencyAdmissionStrategy(threshold=..., counter=KVCounter(...))instead. (#488)- Adam now ignores
initial_accumulator_valueand warns; it used to seed both moments.get/set_initial_optim_statesis renamed toget/set_initial_optimizer_state. (#487) - Training numerics of fp16/bf16 tables can differ slightly from v26.08:
- Incremental dump changes (#515, #532):
- For a row-storing table,
slot_indexandmeta["erased_slot_index"]are[N, 2]int64 (slot, row) tensors. incremental_dumpskipsNO_EVICTIONtables, andreplay_incrementrefuses them.- To make an erase replayable, use
EvictedItemMode.RETAIN_KEY. Any other erase, includingerase_by_score_async, marks the tableerase_untracked. Its deltas are then refused until the replica is rebuilt from a full checkpoint.
- For a row-storing table,
dynamicemb.exportable_tables.create_inference_embedding_collectiontakes a keyword-onlyconfig: InferenceEmbeddingCollectionConfiginstead ofuse_dynamic, andpooling_modeis now a required keyword.apply_inference_embedding_collectiontakesembedding_collection_configs, a mapping from table name toInferenceEmbeddingCollectionConfig, instead ofdynamic_table_configs. (#478)- Rebuild the DynamicEmb extension: its C++ and CUDA sources changed in this release.
HSTU inference
- In the pinned FlexKV,
ModelConfig.use_mlais read-only. Setkv_diminstead (2for separate K and V, equivalent to the olduse_mla=False). (#537)
🐛 Bug fixes
DynamicEmb: memory safety
- CUB scratch buffers were freed before CUB used them; they are now kept alive through selection. (#489)
- When a table expanded to a size that is not a multiple of the bucket size, the value buffer was smaller than the hash capacity. It now matches the rounded-up capacity. (#490)
- A
VMMTensornow waits for queued GPU work before unmapping its memory, for example when a model is dropped right after its last step. (#526) - Offsets are now computed in 64 bits for pooled outputs above 2^31 elements (#555) and for optimizer kernels past 2^32 elements (#548).
(TIMESTAMP, LFU)tables no longer hit a device assertion when a checkpoint load, rehash or backward refresh inserts into a full bucket. (#508)
DynamicEmb: lost or wrong rows
- Table expansion is sized to
max_load_factor. Growing frominit_capacity, for example at checkpoint load, used to keep only 94%–97% of the keys. (#516) - Reference counters are indexed per logical table, so an insert into one table can no longer evict another table's in-use keys. (#514)
- In a prefetch pipeline, prefetched keys keep their reference counts through a rehash. (#529)
- Training-mode forwards under
torch.no_grad()now release their slot pins. (#547) - When the cache's insert refuses a key, DynamicEmb now raises an error instead of writing uninitialized memory back to storage. Reference-count updates are also atomic across streams. (#557)
- Erasing a key clears all of its score words, so a reclaimed slot in a
(TIMESTAMP, LFU)table is reused before live keys are evicted. (#523) - int32 keys behave like the int64 keys a table stores, for every
dist_typeand under index dedup. (#544) - Non-admitted rows are initialized exactly once, and the admission strategy's initializer now also applies on the prefetch paths. (#488)
replay_incrementrechecks the layout after a cache flush, before it writes anything. (#496)
DynamicEmb: hangs, planning and checkpoints
- Under index dedup, ranks with no keys now send empty frequency weights. Previously the KJT all-to-all could hang. (#527)
- The planner no longer pools HBM budgets across collections, and it costs a group that fits in HBM as HBM-only. (#504)
- Each table in a fused group keeps its own
local_hbm_for_valuesbudget. (#534) - A checkpoint load lists the checkpoint directory once instead of six times per table: resolving the files of 2,000 tables went from 539 s to 1.26 s. Warnings about state the caller did not request are gone. (#501)
- Tables named
<scope>/<name>, which is how TorchRec names the tables of fused feature groups, work with the new listing:dump()creates their subdirectory, andload()finds them with one listing per directory. (#509)
HSTU and inference
- Tensor-parallel LayerNorm handles a rank whose evaluation batch is empty. (#556)
- FlexKV no longer starts a CUDA MPS daemon, which broke CUDA for other processes on shared hosts. (#556)
- A dense-module forward without the KV cache no longer replays the cache's CUDA graphs with the last cached request's metadata. (#531)
- HSTU retrieval's dot-product similarity uses a backward that is safe with 64-bit indices on PyTorch 26.07. (#466)
📦 DynamicEmb wheel
A prebuilt dynamicemb-0.2.0-cp312-cp312-linux_x86_64.whl is attached to this release. DynamicEmb's package version is now 0.2.0; it was 0.1.0 from v26.03 through v26.08.
- sha256:
3be2cb06360b05072b050af5b1705c6c536efc69b0b62a7289bd1beb26873183 - Source: this release's DynamicEmb source.
- Build environment: the repository's CI image on NGC PyTorch 26.07, with torch
2.13.0a0+9186a08, CUDA 13.3, NCCL 2.30.7 and Python 3.12, on x86_64. - CUDA targets: sm_75, sm_80, sm_90, sm_100, sm_103 (GB300) and sm_120.
- Includes the hierarchical dist and the bindings for exportable inference embedding collections.
The extension links this exact libtorch, so install the wheel only on that torch build, for example inside nvcr.io/nvidia/pytorch:26.07-py3 or this repository's image, with TorchRec 1.8.0 and FBGEMM GPU 1.8.0:
pip install --no-deps dynamicemb-0.2.0-cp312-cp312-linux_x86_64.whlAt run time, the hierarchical dist needs NCCL 2.30.7 or newer and pynvml. The inference bindings need NV-Embedding-Cache 26.10's pynve, as installed in this repository's image. Custom eviction score functions need numba. For arm64 (for example, GB300), build from source with DEMB_ARCHS=103 python setup.py install in corelib/dynamicemb.
What's Changed
DynamicEmb: features
- Support probabilistic admission strategy in DynamicEmb by @jiashuy in #488
- Support FTRL optimizer in DynamicEmb by @jiashuy in #487
- Support table-row-wise sharding in DynamicEmb by @jiashuy in #493
- FTRL group sparsity in DynamicEmb by @jiashuy in #500
- Cache serves as a subset of storage; external storage sits below storage by @jiashuy in #511
- Store a value row for every key in LinearBucketTable and DynamicEmbStorage by @jiashuy in #515
- Per-key score input through the PRECISE_CUSTOMIZED score strategy by @jiashuy in #530
- Async erase by score by @jiashuy in #532
- erase_by_score for lru_lfu tables, per named table by @jyuan-sc in #536
- Hierarchical dist for row-wise sequence sharding by @shijieliu in #528
- FP8 table storage with stochastic rounding for row-wise Adagrad by @jiashuy in #548 (building on #499 by @jyuan-sc)
DynamicEmb: fixes
- Retain CUB scratch tensors through selection by @lidongdongbuaa in #489
- Match expanded values to aligned hash capacity by @lidongdongbuaa in #490
- Recheck replay layout after cache flush by @efegokdemir in #496
- Keep row-wise Adagrad accumulator in fp32 for 16-bit tables by @jyuan-sc in #495
- List the checkpoint directory once per load, and quiet load/dump warnings by @tiankongdeguiji in #501
- Plan per EmbeddingCollection; identify a table by its collection and its name by @jiashuy in #504
- Load tables named
<scope>/<name>without listing per lookup by @tiankongdeguiji in #509 - Size table expansion to max_load_factor by @nthan-sc in #516
- Evict LruLfu tables through the ranked evictor on every insert path by @jyuan-sc in #508
- Clear every score word in table_erase_kernel by @jiashuy in #523
- Index reference counters across logical tables by @lidongdongbuaa in #514
- Keep each table's local_hbm_for_values budget in fused groups by @tiankongdeguiji in #534
- Keep prefetched keys' ref counts through a rehash by @shijieliu in #529
- Send frequency weights for an empty input under index dedup by @shijieliu in #527
- Wait for queued GPU work before freeing VMMTensor memory by @shijieliu in #526
- Release slot pins after no-grad training forwards by @jiashuy in #547 (original fix by @lidongdongbuaa in #541)
- Treat int32 keys as the int64 keys a table stores, under every dist_type and index dedup by @jiashuy in #544
- The hierarchical dist takes a node only when NVLink connects its GPUs by @shijieliu in #553
- Fix int32 overflow in get_dst_ptr by @lzw3232 in #555
- The cache raises on keys its insert refuses; pins add up across streams by @shijieliu in #557
HSTU and inference
- Pin FBGEMM Hopper SM90 paged HSTU DSL by @Clebrate in #510
- Layer-wise KV-cache loading with FlexKV (Layerwise Design) by @Clebrate in #498
- Set kv_dim instead of assigning ModelConfig.use_mla by @Clebrate in #537
- Use the Hopper DSL for paged inference only by @Clebrate in #535
- Support configurable indexing and NVE layers for exportable embedding collections by @geoffreyQiu in #478
- CI follow-ups to #478: TP layer norm on an empty batch; FlexKV without MPS by @shijieliu in #556
- FP8 DeepGEMM and FA4 attention for Transformer inference by @z52527 in #531 (includes the Transformer backbone from #491 by @Junyi-Zheng)
SID-GR inference
- Fuse beam attention and merge, default on SM10x (up to 3.3×) by @bellanich in #520
- Fuse QK-norm + RoPE and make it the default on SM10x (up to 1.7×) by @bellanich in #517
- Vectorize the beam-KV write and make it the default (up to 3.7×) by @bellanich in #513
New examples
- AI-Search-Plugin: drop-in components for search-agent RL post-training on slime and verl by @cb521 in #539
- Feature-Extractor-Plugin: Spark expression fusion (batch) and Flink GPU operators (streaming) by @cb521 in #540
Build, CI and container
- Update base image to 26.07 by @shijieliu in #466
- Move the hierarchical dist's tests into their own CI groups by @shijieliu in #550
- test_hier_dist no longer checks a node that is not NCCL's team by @shijieliu in #554
🙌 New contributors
- @lidongdongbuaa made their first contribution in #489
- @efegokdemir made their first contribution in #496
- @jyuan-sc made their first contribution in #495
- @tiankongdeguiji made their first contribution in #501
- @nthan-sc made their first contribution in #516
- @bellanich made their first contribution in #520
- @lzw3232 made their first contribution in #555
- @Junyi-Zheng made their first contribution with the Transformer backbone from #491, merged in #531
Thanks to everyone who contributed to this release: @bellanich, @cb521, @Clebrate, @efegokdemir, @geoffreyQiu, @jiashuy, @Junyi-Zheng, @jyuan-sc, @lidongdongbuaa, @lzw3232, @nthan-sc, @shijieliu, @tiankongdeguiji and @z52527.
Full changelog: v26.08...v26.10