[TRTLLM-12316][feat] Integrate FP4 indexer for DSv4 by mikeiovine · Pull Request #13575 · NVIDIA/TensorRT-LLM

mikeiovine · 2026-04-28T21:37:39Z

Description

Integrate the fp4 indexer kernels from DeepGEMM into DSV4. It can be used by setting DeepSeekV4SparseAttentionConfig(indexer_k_dtype="fp4").

Test Coverage

Existing tests plus a few new new unit tests. I also manually tested accuracy - confirmed that there is no drop in gsm8k for both Flash and Pro (a small drop is expected due to the lower precision):

Flash (FP8 Blockwise Indexer)
|Tasks|Version|     Filter     |n-shot|  Metric   |   | Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|------:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |94.9962|±  |0.6005|
|     |       |strict-match    |     5|exact_match|↑  |95.0720|±  |0.5962|

Flash (FP4 Indexer)
|Tasks|Version|     Filter     |n-shot|  Metric   |   | Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|------:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |94.3897|±  |0.6339|
|     |       |strict-match    |     5|exact_match|↑  |94.4655|±  |0.6298|

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

mikeiovine · 2026-05-14T19:45:14Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-14T19:52:27Z

PR_Github #48423 [ run ] triggered by Bot. Commit: 9f6f0d5 Link to invocation

tensorrt-cicd · 2026-05-14T23:06:33Z

PR_Github #48423 [ run ] completed with state SUCCESS. Commit: 9f6f0d5
/LLM/main/L0_MergeRequest_PR pipeline #38224 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Mike Iovine <miovine@nvidia.com>

tensorrt-cicd · 2026-05-15T19:54:19Z

PR_Github #48610 [ run ] completed with state ABORTED. Commit: d6d986c

Link to invocation

tensorrt-cicd · 2026-05-15T23:56:26Z

PR_Github #48633 [ run ] completed with state FAILURE. Commit: ba16ae9
/LLM/main/L0_MergeRequest_PR pipeline #38415 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

mikeiovine · 2026-05-15T23:59:11Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-16T00:05:21Z

PR_Github #48648 [ run ] triggered by Bot. Commit: ba16ae9 Link to invocation

tensorrt-cicd · 2026-05-16T03:17:51Z

PR_Github #48648 [ run ] completed with state FAILURE. Commit: ba16ae9
/LLM/main/L0_MergeRequest_PR pipeline #38429 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

mikeiovine · 2026-05-16T15:28:05Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-16T15:33:54Z

PR_Github #48693 [ run ] triggered by Bot. Commit: ba16ae9 Link to invocation

tensorrt-cicd · 2026-05-16T16:26:17Z

PR_Github #48693 [ run ] completed with state FAILURE. Commit: ba16ae9
/LLM/main/L0_MergeRequest_PR pipeline #38467 completed with status: 'ABORTED'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>

mikeiovine · 2026-05-16T16:53:41Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-16T17:00:10Z

PR_Github #48701 [ run ] triggered by Bot. Commit: fded951 Link to invocation

tensorrt-cicd · 2026-05-16T18:12:02Z

PR_Github #48701 [ run ] completed with state FAILURE. Commit: fded951
/LLM/main/L0_MergeRequest_PR pipeline #38472 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

mikeiovine · 2026-05-17T03:57:47Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-17T04:04:55Z

PR_Github #48730 [ run ] triggered by Bot. Commit: fded951 Link to invocation

Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>

lfr-0531 · 2026-05-17T05:06:43Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-17T05:12:19Z

PR_Github #48734 [ run ] triggered by Bot. Commit: d07d383 Link to invocation

tensorrt-cicd · 2026-05-17T05:12:42Z

PR_Github #48730 [ run ] completed with state ABORTED. Commit: fded951

Link to invocation

tensorrt-cicd · 2026-05-17T07:40:04Z

PR_Github #48734 [ run ] completed with state SUCCESS. Commit: d07d383
/LLM/main/L0_MergeRequest_PR pipeline #38500 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lfr-0531 · 2026-05-17T16:00:25Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-17T16:06:45Z

PR_Github #48772 [ run ] triggered by Bot. Commit: d07d383 Link to invocation

tensorrt-cicd · 2026-05-17T16:37:32Z

PR_Github #48772 [ run ] completed with state FAILURE. Commit: d07d383
/LLM/main/L0_MergeRequest_PR pipeline #38539 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

mikeiovine · 2026-05-17T19:50:40Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-17T19:58:57Z

PR_Github #48782 [ run ] triggered by Bot. Commit: d07d383 Link to invocation

tensorrt-cicd · 2026-05-17T21:10:53Z

PR_Github #48782 [ run ] completed with state SUCCESS. Commit: d07d383
/LLM/main/L0_MergeRequest_PR pipeline #38547 completed with status: 'SUCCESS'

CI Report

Link to invocation

Signed-off-by: Mike Iovine <miovine@nvidia.com> Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com> Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> Co-authored-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> (cherry picked from commit 7fbe349) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

github-actions Bot assigned mikeiovine Apr 28, 2026

lfr-0531 added the deepseek-v4 label Apr 29, 2026

lfr-0531 force-pushed the feat/deepseek_v4 branch from 67b0b17 to 9c57516 Compare May 7, 2026 16:29

mikeiovine force-pushed the fp4-indexer branch from 59906f8 to 1a8b70c Compare May 13, 2026 19:39

lfr-0531 force-pushed the feat/deepseek_v4 branch from 0a93d10 to 118e7a5 Compare May 14, 2026 07:44

mikeiovine force-pushed the fp4-indexer branch from 1a8b70c to f9d4660 Compare May 14, 2026 14:39

mikeiovine requested a review from lfr-0531 May 14, 2026 19:19

mikeiovine changed the title ~~[TRTLLM-12316][feat] Support FP4 indexer~~ [TRTLLM-12316][feat] Integrate FP4 indexer for DSv4 May 14, 2026

mikeiovine marked this pull request as ready for review May 14, 2026 19:45

mikeiovine requested review from a team as code owners May 14, 2026 19:45

mikeiovine requested review from QiJune, liji-nv and schetlur-nv and removed request for a team May 14, 2026 19:45

mikeiovine commented May 14, 2026

View reviewed changes

Comment thread tests/unittest/_torch/attention/sparse/rocketkv/test_rocketkv.py

mikeiovine force-pushed the fp4-indexer branch from 0a62ca6 to 9f6f0d5 Compare May 14, 2026 19:49

yingguo-trt approved these changes May 15, 2026

View reviewed changes

jiaganc reviewed May 15, 2026

View reviewed changes

FP4 indexer

3021dc9

Signed-off-by: Mike Iovine <miovine@nvidia.com>

mikeiovine commented May 16, 2026

View reviewed changes

Comment thread tests/integration/test_lists/waives.txt Outdated

Waive failing test

fded951

Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>

mikeiovine force-pushed the fp4-indexer branch from cc0a862 to fded951 Compare May 16, 2026 16:50

Merge branch 'feat/deepseek_v4' into fp4-indexer

d07d383

Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>

lfr-0531 merged commit 7fbe349 into NVIDIA:feat/deepseek_v4 May 18, 2026
6 checks passed

mikeiovine deleted the fp4-indexer branch May 18, 2026 14:50

Conversation

mikeiovine commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

mikeiovine commented May 14, 2026

Uh oh!

Uh oh!

tensorrt-cicd commented May 14, 2026

Uh oh!

tensorrt-cicd commented May 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tensorrt-cicd commented May 15, 2026

Uh oh!

tensorrt-cicd commented May 15, 2026

Uh oh!

mikeiovine commented May 15, 2026

Uh oh!

tensorrt-cicd commented May 16, 2026

Uh oh!

tensorrt-cicd commented May 16, 2026

Uh oh!

mikeiovine commented May 16, 2026

Uh oh!

tensorrt-cicd commented May 16, 2026

Uh oh!

tensorrt-cicd commented May 16, 2026

Uh oh!

Uh oh!

mikeiovine commented May 16, 2026

Uh oh!

tensorrt-cicd commented May 16, 2026

Uh oh!

tensorrt-cicd commented May 16, 2026

Uh oh!

mikeiovine commented May 17, 2026

Uh oh!

tensorrt-cicd commented May 17, 2026

Uh oh!

lfr-0531 commented May 17, 2026

Uh oh!

tensorrt-cicd commented May 17, 2026

Uh oh!

tensorrt-cicd commented May 17, 2026

Uh oh!

tensorrt-cicd commented May 17, 2026

Uh oh!

lfr-0531 commented May 17, 2026

Uh oh!

tensorrt-cicd commented May 17, 2026

Uh oh!

tensorrt-cicd commented May 17, 2026

Uh oh!

mikeiovine commented May 17, 2026

Uh oh!

tensorrt-cicd commented May 17, 2026

Uh oh!

tensorrt-cicd commented May 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mikeiovine commented Apr 28, 2026 •

edited

Loading