Development Roadmap(H3)

[shijieliu]

Here is the development roadmap for 2025 H3(Aug to Oct).

Focus

• Enhance the performance and core functionality of Dynamicemb.
• In HSTU example training, support larger models with increased parallelism, leveraging Dynamicemb's improved functionality and performance.
• For HSTU example inference, focus on kernel optimization and integration with Triton to enable real-world deployment.
• Update HSTU attention to support the latest architecture.
• Develop a proof of concept for the Semantic ID example.

Roadmap

	Aug Release	Sep Release	Oct Release	Long-Term
Dynamicemb	GPU cache and hot embedding migration (milestone 1) [FEA] dynamic embedding training cache #63	Refactor load/dump to support distributed dumping [FEA] Distributed embedding dumping for dynamicemb #108 GPU cache and hot embedding migration (milestone 2) [FEA] dynamic embedding training cache #63 LFU bug fixing [BUG] dynamicemb's LFU mode only counts the frequency of unique keys. #143	Embedding admission [FEA] Embedding adimission #111 LRU dumping score [FEA] dynamicemb LRU support dumping score #158	NVEmbedding backend integration InputDist upstream Dynamic growing Fuse multiple tables with same dimension
HSTU attention	Arbitrary mask support [FEA] Support arbitrary HSTU mask #118		Blackwell support
HSTU example training		Dynamicemb prefetch pipeline integration [FEA] hstu example support dynamicemb prefetch #159	HSTU + FFN support [FEA] Harness TransformerLayer from megatron to enable flexible structure #133 Activation offloading [FEA] activation offloading #48	Context parallelism [FEA] Support HSTU Context parallelism in training #7 Sequence parallelism [FEA] Enable Sequence Parallelism #130
HSTU example inference	NVEmbedding integration [FEA] HSTU ranking training and inference e2e example #109 E2E example [FEA] HSTU ranking training and inference e2e example #109	HSTU layer kernel optimization and fusing [FEA] HSTU layer inference kernel optimization #160	NVIDIA Triton HSTU model support [FEA] HSTU inference TritonServer support #161	Multi-stream KVCache manager support Model serialization & torch cpp runtime reference KVCache manager upstream
Semantic ID example training & inference			PoC

[mia1460]

Hi,

I am currently using HSTU for ranking tasks and encountered an issue with the kvcache feature when using the bf16 data type. Specifically, I noticed discrepancies that lower the AUC of my tasks. I also came across the following error during runtime when trying to use float32:

RuntimeError: HSTU only supports fp16 and bf16 data type

Since HSTU currently does not support float32, I wanted to ask if there are any plans to add support for float32 in the future. In recommendation systems, precision is often crucial, and supporting float32 could help address the current issues with bf16 in certain use cases.

Any insights or updates on this would be greatly appreciated!

[shijieliu]

Hi @mia1460 thanks for your attention to our roadmap!

We do not plan to support FP32. Instead, we are focusing on lower precisions such as FP8 in the roadmap. For models like HSTU, which have extremely high compute density, FP32 does not meet the computational requirements and cannot fully leverage the GPU performance for recommender systems.

Regarding your concerns about kvcache precision, we take this issue seriously. Could you please create issues and describe the specific problem you encountered? @geoffreyQiu will assist in investigating it. I also recommend using PyTorch for the reference implementation to help debug the issue efficiently if necessary.

[mia1460]

Hi, @shijieliu
Understood about the roadmap, thanks for clarifying.
The related issue has already been filed here: #152. I will update the issue with my recent results later.
Thanks also for the suggestion, I will give it a try!