Skip to content

[DNM - test PR, waiting for nvidia/tensorrtllm to accept the fusedmhc patch] Update DSV4 TRT fused MHC image #1270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Oseltamivir wants to merge 3 commits into
base: main
Choose a base branch
from
Open

[DNM - test PR, waiting for nvidia/tensorrtllm to accept the fusedmhc patch] Update DSV4 TRT fused MHC image #1270

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1a3b417
update dsv4 trt fused mhc image
Oseltamivir May 3, 2026
1ce0e22
Update perf-changelog.yaml
Oseltamivir May 3, 2026
660a3b7
all, again
Oseltamivir May 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2707,7 +2707,7 @@ dsv4-fp4-b300-vllm:
- { tp: 8, ep: 8, dp-attn: true, conc-start: 2048, conc-end: 2048 }

dsv4-fp4-b300-trt:
image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-4999884
image: ghcr.io#semianalysisai/trtllm-deepseek-v4:fix-mhc7168-eb20e9e
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: b300
Expand Down
7 changes: 2 additions & 5 deletions benchmarks/single_node/dsv4_fp4_b300_trt.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/usr/bin/env bash

# DeepSeek-V4-Pro single-node TRTLLM recipe for B300. The configured image
# already contains NVIDIA/TensorRT-LLM@feat/deepseek_v4; do not build TRTLLM at
# already contains a TensorRT-LLM DeepSeek-V4 build; do not build TRTLLM at
# runtime from this benchmark path.

source "$(dirname "$0")/../benchmark_lib.sh"
Expand Down Expand Up @@ -101,10 +101,7 @@ if [ "${EVAL_ONLY}" = "true" ]; then
MAX_NUM_TOKENS="$EVAL_MAX_MODEL_LEN"
fi

# DeepSeek-V4-Pro has hidden size 7168. The current TRTLLM fused-HC MHC
# path corrupts eval generations for this shape; keep eval servers on the
# unfused path until the fused kernel is guarded or supports 7168.
export TRTLLM_MHC_ENABLE_FUSED_HC=0
export TRTLLM_MHC_ENABLE_FUSED_HC="${TRTLLM_MHC_ENABLE_FUSED_HC:-1}"
echo "TRTLLM_MHC_ENABLE_FUSED_HC: $TRTLLM_MHC_ENABLE_FUSED_HC"

start_gpu_monitor --output "$PWD/gpu_metrics.csv"
Expand Down
7 changes: 7 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2148,3 +2148,10 @@
- "Disable TRTLLM fused MHC hyper-connection for eval servers via TRTLLM_MHC_ENABLE_FUSED_HC=0 because the current fused kernel corrupts DeepSeek-V4-Pro hidden size 7168 generations"
- "Keep this as eval-only PR validation until the TensorRT-LLM fused MHC kernel is guarded or supports hidden size 7168"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1233

- config-keys:
- dsv4-fp4-b300-trt
description:
- "Update the TensorRT-LLM DeepSeek-V4-Pro image to ghcr.io/semianalysisai/trtllm-deepseek-v4:fix-mhc7168-eb20e9e"
- "Enable TRTLLM fused MHC by default now that the image includes the hidden-size 7168 fused-HC fix"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1270
Loading