Skip to content

[None][refactor] Clean up model paths and remove deprecated configurations in performance tests - #17057

Merged
yufeiwu-nv merged 6 commits into
NVIDIA:mainfrom
yufeiwu-nv:fp4
Jul 31, 2026
Merged

[None][refactor] Clean up model paths and remove deprecated configurations in performance tests#17057
yufeiwu-nv merged 6 commits into
NVIDIA:mainfrom
yufeiwu-nv:fp4

Conversation

@yufeiwu-nv

@yufeiwu-nv yufeiwu-nv commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Dev Engineer Review

  • Refactored performance-test model configuration paths by pruning unused/obsolete model aliases in tests/integration/defs/perf/_model_paths.py (including various Llama/Qwen/Phi-Bielik/Nemotron/DeepSeek R1 variants).
  • Updated PyTorch performance model config generation in tests/integration/defs/perf/pytorch_model_config.py:
    • DeepSeek V4 Flash-Base MoE: removed an explicit moe_config.backend: WIDEEP override while keeping attention DP enablement.
    • DeepSeek FP8 block-scale allowlist: removed deepseek_r1_fp8 and added deepseek_r1_0528_fp8 to ensure the DEEPGEMM MoE backend override applies on SM100+.
    • Removed large DeepSeek R1-specific pattern/config variations (including MTP/speculative-decoding-related pattern entries).
    • Simplified/contracted several pattern matchers (MiniMax/Llama/GPT-OSS/Gemma3/Nemotron), including narrowing cuda-graph padding to the “Llama-v4 Scout” FP4 pattern and pruning specific GPT-OSS request/connection variants.
  • Tightened sampler option behavior in tests/integration/defs/perf/sampler_options_config.py:
    • model_label matching is now based on exact equality to PerfTestConfig.to_string() output.
    • Default sampler parameter injection is restricted to a narrower set of label(s) instead of the previous broader multi-label list.

QA Engineer Review

  • No test functions were added, modified, or removed.
  • Config-only changes under tests/:
    • tests/integration/defs/perf/_model_paths.py
    • tests/integration/defs/perf/pytorch_model_config.py
    • tests/integration/defs/perf/sampler_options_config.py
  • No changes were identified under tests/integration/test_lists/.
  • Verdict: needs follow-up (renamed/removed performance model labels and narrowed pattern scopes may impact CI/CBTS coverage for benchmark combinations).

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

…n model YAML (NVIDIA#16935)

Signed-off-by: yufeiwu-nv <230315618+yufeiwu-nv@users.noreply.github.com>
…tions in performance tests

- Removed unused model paths from _model_paths.py.
- Updated sampler_options_config.py to reflect changes in model labels.
- Cleaned up pytorch_model_config.py by removing obsolete DeepSeek R1 model configurations.

These changes streamline the model configuration files and ensure they only contain relevant entries.

Signed-off-by: yufeiwu-nv <230315618+yufeiwu-nv@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Performance integration definitions update model path mappings, backend overrides, benchmark pattern configurations, and sampler-label matching for selected model variants.

Changes

Performance configuration

Layer / File(s) Summary
Model path mapping updates
tests/integration/defs/perf/_model_paths.py
Llama, DeepSeek, Phi, and Bielik model path entries are narrowed or replaced with selected instruct, FP8, FP4, reasoning, and multimodal variants.
DeepSeek backend and pattern routing
tests/integration/defs/perf/pytorch_model_config.py
DeepSeek FP8 allowlists, MoE backend selection, and DeepSeek R1 benchmark configurations are updated.
Benchmark pattern configuration
tests/integration/defs/perf/pytorch_model_config.py
MiniMax, Llama, GPT-OSS, Gemma3, and Nemotron pattern matching and variants are revised.
Sampler label matching
tests/integration/defs/perf/sampler_options_config.py
Default sampler options now apply to a revised set of exact model labels, with matching behavior documented inline.

Estimated code review effort: 4 (Complex) | ~40 minutes

Possibly related PRs

  • NVIDIA/TensorRT-LLM#17047: Updates the same performance model mappings and benchmark configuration routing for DeepSeek and low-precision models.

Suggested reviewers: bowenfu, qijune, emmaqiaoch, mlefeb01, ruodil

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description is still the template stub and omits the required issue/solution and test coverage details. Replace the stub with a short description of the change, why it was needed, and list the relevant tests or validation steps.
✅ Passed checks (4 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title is concise and accurately describes the main cleanup and deprecation-removal changes in performance tests.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Fix failing CI checks
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

Signed-off-by: yufeiwu-nv <230315618+yufeiwu-nv@users.noreply.github.com>

# Conflicts:
#	tests/integration/defs/perf/_model_paths.py
@yufeiwu-nv

Copy link
Copy Markdown
Collaborator Author

/bot skip --comment "only test modify"

@yufeiwu-nv
yufeiwu-nv enabled auto-merge (squash) July 30, 2026 08:16
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62738 [ skip ] triggered by Bot. Commit: 56fa694 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62738 [ skip ] completed with state SUCCESS. Commit: 56fa694
Skipping testing for commit 56fa694

Link to invocation

@yufeiwu-nv

Copy link
Copy Markdown
Collaborator Author

/bot skip --comment "only test modify"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62752 [ skip ] triggered by Bot. Commit: 5491972 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62752 [ skip ] completed with state SUCCESS. Commit: 5491972
Skipping testing for commit 5491972

Link to invocation

@BowenFu

BowenFu commented Jul 30, 2026

Copy link
Copy Markdown

The removed-key audit is clean. I checked all 14 keys dropped from MODEL_PATH_DICT: none is referenced anywhere under tests/integration/test_lists/**; the three that remain in allowed_configs.py (qwen2_7b_instruct, qwen_14b_chat, starcoder2_3b) still validate via the get_allowed_models() fallback at test_perf.py:757-761; and no removed key has a scheduled pp_size > 1 case that would trip the assert at :804.

What gives me pause is that this isn't only deletion. Three of the replacement blocks swap an explicit label list for a bare model-name pattern, and matching is substring, not equality — pytorch_model_config.py:624-625, if pattern in model_label.lower(). So they widen:

  • 'patterns': ['minimax_m2.5_fp8'] replaces a list of 12 fully-qualified labels. It now matches every scheduled minimax_m2.5_fp8 case, so roughly ten more of them newly get enable_attention_dp: True.
  • The minimax_m3_mxfp8 8000,1000 entries move max_seq_len 4096 → 9216 for the scheduled cases they match.
  • 'patterns': ['gemma_3'] and 'patterns': ['llama_v4_scout_17b_16e_instruct_fp4'] broaden the same way — gemma_3 matches gemma_3_1b_it, gemma_3_12b_it, gemma_3_27b_it and their fp8/fp4 variants alike.

The sampler change has the same shape: the retained label now canonicalizes to what qa/llm_perf_core.yml:59 actually emits, so that case newly picks up top_k=4 / top_p=0.5 / temperature=0.5 where previously nothing matched at all.

In every one of these the test identity is unchanged, so they keep their existing recorded perf baselines while their runtime configuration changes underneath. That's what I'd want covered before this lands — either refreshed baselines or a before/after run showing the current thresholds still hold. If the deltas are known to be within noise, say so and I'm happy; I just can't tell that from the diff.

To be clear, the to_string() comment and the label fix are genuine improvements — get_sampler_options_config was silently matching nothing, and that's worth having found.

@yufeiwu-nv

Copy link
Copy Markdown
Collaborator Author

/bot skip --comment "only test modify"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62930 [ skip ] triggered by Bot. Commit: 06d7859 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62930 [ skip ] completed with state SUCCESS. Commit: 06d7859
Skipping testing for commit 06d7859

Link to invocation

@yufeiwu-nv
yufeiwu-nv merged commit 8e602fa into NVIDIA:main Jul 31, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants