Skip to content

[testing][nv] h100: mtp configs sglang - #644

Merged
ishandhanani merged 15 commits into
mainfrom
nv/h100-mtp-sglang
Feb 13, 2026
Merged

[testing][nv] h100: mtp configs sglang#644
ishandhanani merged 15 commits into
mainfrom
nv/h100-mtp-sglang

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Feb 5, 2026 •
edited by Klaud-Cold
Loading

Copy link
Copy Markdown
Collaborator

Add MTP (Multi-Token Prediction) configurations for DeepSeek-R1 FP8 on H100 using SGLang with Dynamo disaggregation.

Configuration Details

  • Config key: dsr1-fp8-h100-dynamo-sglang
  • Image: lmsysorg/sglang:v0.5.8-cu130
  • Model: deepseek-ai/DeepSeek-R1-0528
  • Runner: h100-multinode-slurm (multinode + disagg enabled)
  • Precision: FP8
  • Framework: dynamo-sglang

Search Space

1k1k (ISL=1024, OSL=1024)

  • MTP Max throughput TEP (1P2D): 1 prefill worker, 2 decode workers, TP16
  • MTP Max throughput DEP (1P1D): 1 prefill worker, 1 decode worker with dp-attention, EP16

1k8k (ISL=1024, OSL=8192)

  • MTP Max throughput TEP (1P2D): 1 prefill worker, 2 decode workers, TP16
  • MTP Max throughput DEP (1P1D): 1 prefill worker, 1 decode worker with dp-attention, EP16

8k1k (ISL=8192, OSL=1024)

  • MTP Max throughput TEP (1P1D): 1 prefill worker, 1 decode worker, TP16
  • MTP Max throughput DEP (1P1D): 1 prefill worker, 1 decode worker with dp-attention, EP16

Concurrency Ranges

  • TEP configs: 1, 2, 4, 8, 16, 32, 64, 128
  • DEP configs: 1, 2, 4, 8, 16, 32, 64

Recipe Files

Uses CONFIG_FILE-based recipes under recipes/h100/*/mtp/ for each sequence length configuration.

@ishandhanani
ishandhanani requested a review from a team as a code owner February 5, 2026 01:53
@cquil11

cquil11 commented Feb 5, 2026

Copy link
Copy Markdown
Collaborator

@claude Please update PR description and add an entry to perf-changelog.yaml Reference other PRs if you need to.

@Klaud-Cold

Klaud-Cold commented Feb 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Claude finished @cquil11's task in 1m 27s —— View job

PR Description and Changelog Update Complete

  • Read the changed files to understand the PR changes
  • Review similar PRs for reference
  • Update PR description
  • Add entry to perf-changelog.yaml
  • Commit and push changes

Summary:

  • Updated PR description with detailed configuration information including:
    • Config key, image, model, runner, precision, and framework details
    • Search space breakdown for 1k1k, 1k8k, and 8k1k sequence lengths
    • TEP and DEP mode configurations with worker and TP/EP settings
    • Concurrency ranges and recipe file information
  • Added entry to perf-changelog.yaml documenting the new dsr1-fp8-h100-dynamo-sglang configuration
    |

@github-actions @cquil11
docs: add perf-changelog entry for H100 MTP SGLang configs
603daca 
Add changelog entry for dsr1-fp8-h100-dynamo-sglang configuration
with MTP disaggregated multinode support.

Co-authored-by: Cameron Quilici <cquil11@users.noreply.github.com>
@ishandhanani
ishandhanani marked this pull request as draft February 6, 2026 00:56
@kedarpotdar-nv
kedarpotdar-nv marked this pull request as ready for review February 8, 2026 18:12
@claude

claude Bot commented Feb 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Claude encountered an error —— View job

I'll analyze this and get back to you.

kedarpotdar-nv and others added 4 commits February 8, 2026 10:17
@kedarpotdar-nv
Fix perf-changelog whitespace to satisfy process_changelog (additions…
161fec1 
…-only)
@kedarpotdar-nv
typo
78f6aa6
@ishandhanani
Resolve merge conflict in perf-changelog.yaml
43c9574
@ishandhanani
Add SGLang support to H100 runner
4a3dec7 
- Add framework-aware model path and container configuration
- Map SGLang container to /mnt/nfs/lustre/containers/lmsysorg_sglang_v0.5.8.post1-cu130.sqsh
- Add nginx container for SGLang at /mnt/nfs/lustre/containers/nginx_1.27.4.sqsh
- Update srtslurm.yaml with dynamo-sglang and nginx-sqsh entries
@kedarpotdar-nv

kedarpotdar-nv commented Feb 9, 2026

Copy link
Copy Markdown
Collaborator

why did this sweep not auto-start? github outage?

@kedarpotdar-nv

kedarpotdar-nv commented Feb 9, 2026

Copy link
Copy Markdown
Collaborator

/sweep

@github-actions

github-actions Bot commented Feb 9, 2026

Copy link
Copy Markdown
Contributor

@kedarpotdar-nv Kicking off a sweep.

Run: https://github.com/InferenceMAX/InferenceMAX/actions/runs/21840473862
Command: ``
Pinned ref: 4a3dec7
Approval: not required (trusted collaborator).

@ishandhanani
Resolve merge conflicts from main
d4cb995
@ishandhanani
Add STP configs alongside MTP for H100 SGLang
35a467c 
- Add STP entries (without spec-decoding) for all ISL/OSL combinations
- Keep MTP entries (with spec-decoding: mtp) as before
- Now both STP and MTP will run when testing dsr1-fp8-h100-dynamo-sglang
@ishandhanani
Restore B200 MTP SGLang config lost in merge
2b1ea75 
The dsr1-fp8-b200-dynamo-sglang-mtp config was accidentally removed
during merge conflict resolution.
@ishandhanani
Add perf-changelog entry for H100 STP SGLang (PR #643)
75e7bf6
@ishandhanani
Comment out H100 SGLang STP configs for MTP-only CI testing
bf0ceb6
@ishandhanani

ishandhanani commented Feb 10, 2026

Copy link
Copy Markdown
Collaborator Author

Ive commented out the STP results so I can run a test on only the MTP ones. Please don't merge my PRs until I/Kedar explicitly say they are ready to merge

@ishandhanani ishandhanani changed the title [nv] h100: mtp configs sglang [testing][nv] h100: mtp configs sglang Feb 10, 2026
@ishandhanani

ishandhanani commented Feb 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Everything succeeded. 1 of the jobs timed out before the final concurrency. Increasing the time limit and restarting the run

https://github.com/InferenceMAX/InferenceMAX/actions/runs/21922011918

@ishandhanani

ishandhanani commented Feb 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Good to go @cquil11 @functionstackx

@ishandhanani
ishandhanani enabled auto-merge (squash) February 13, 2026 02:25
functionstackx
functionstackx approved these changes Feb 13, 2026
View reviewed changes

@functionstackx functionstackx left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

@ishandhanani
ishandhanani merged commit 78d4861 into main Feb 13, 2026
7 of 11 checks passed
@ishandhanani
ishandhanani deleted the nv/h100-mtp-sglang branch February 13, 2026 03:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@functionstackx functionstackx functionstackx approved these changes

Assignees

No one assigned

Labels

NVIDIA sweep-enabled

Projects

InferenceMAX Board
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ishandhanani @cquil11 @Klaud-Cold @kedarpotdar-nv @functionstackx