[None][chore] AutoDeploy: Added Qwen3.5 accuracy test for NVFP4#13014
[None][chore] AutoDeploy: Added Qwen3.5 accuracy test for NVFP4#13014taylor-yb-lee merged 4 commits intoNVIDIA:mainfrom
Conversation
Signed-off-by: Taylor Yeonbok Lee <249374542+taylor-yb-lee@users.noreply.github.com>
Signed-off-by: Taylor Yeonbok Lee <249374542+taylor-yb-lee@users.noreply.github.com>
Signed-off-by: Taylor Yeonbok Lee <249374542+taylor-yb-lee@users.noreply.github.com>
taylor-yb-lee
force-pushed
the
taylor/qwen3.5_nvfp4_acctest
branch
from
a977fea to
484e3bb
Compare
📝 WalkthroughWalkthroughThe PR updates test infrastructure and model registry configurations for a new NVFP4 quantized variant of the Qwen3.5-397B model. It introduces a new test case, adds model-specific evaluation configuration, updates model path mappings, and disables a transform setting in the model registry configuration. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 1 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (1 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tests/test_common/llm_data.py (1)
1-1:⚠️ Potential issue | 🟡 MinorUpdate the copyright year on this modified file.
This file is being edited in 2026, but the header still ends at 2025.
As per coding guidelines, "Add NVIDIA copyright header on ALL new files and update year on modified files".
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/test_common/llm_data.py` at line 1, Update the copyright header SPDX line at the top of the file (the SPDX-FileCopyrightText comment) to include 2026 as the end year instead of 2025 so the header reflects the current modification year; locate the SPDX-FileCopyrightText line in tests/test_common/llm_data.py and change the year range to 2022-2026.
🧹 Nitpick comments (1)
tests/integration/defs/accuracy/test_llm_api_autodeploy.py (1)
812-816: Prefer generating these evaluator kwargs per test invocation.
EXTRA_EVALUATOR_KWARGSis a shared nested dict on the class. If any evaluator mutates it, state can leak between the BF16 and NVFP4 runs. A tiny helper returning a fresh dict avoids that coupling.♻️ Suggested change
- EXTRA_EVALUATOR_KWARGS = dict( - apply_chat_template=True, - fewshot_as_multiturn=True, - chat_template_kwargs=dict(enable_thinking=False), - ) + `@staticmethod` + def _extra_evaluator_kwargs(): + return { + "apply_chat_template": True, + "fewshot_as_multiturn": True, + "chat_template_kwargs": {"enable_thinking": False}, + }Then update both
task.evaluate(..., extra_evaluator_kwargs=...)call sites to passself._extra_evaluator_kwargs().🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/integration/defs/accuracy/test_llm_api_autodeploy.py` around lines 812 - 816, EXTRA_EVALUATOR_KWARGS is a shared mutable dict that can leak state between tests; replace it with a small helper method (e.g., def _extra_evaluator_kwargs(self): ...) that returns a fresh dict with apply_chat_template=True, fewshot_as_multiturn=True, and chat_template_kwargs={'enable_thinking': False}, and update both task.evaluate(..., extra_evaluator_kwargs=...) call sites to pass self._extra_evaluator_kwargs() instead of the module/class-level EXTRA_EVALUATOR_KWARGS.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@tests/integration/defs/accuracy/test_llm_api_autodeploy.py`:
- Around line 865-866: In test_nvfp4, the registry config is fetched for the
base model name; update the call to _get_registry_yaml_extra to use the NVFP4
variant by replacing self.MODEL_NAME with self.MODEL_NAME_NVFP4 so the test
loads the correct registry YAML for the NVFP4 model (look for the
_get_registry_yaml_extra(...) invocation in test_nvfp4 and change the argument
to self.MODEL_NAME_NVFP4).
---
Outside diff comments:
In `@tests/test_common/llm_data.py`:
- Line 1: Update the copyright header SPDX line at the top of the file (the
SPDX-FileCopyrightText comment) to include 2026 as the end year instead of 2025
so the header reflects the current modification year; locate the
SPDX-FileCopyrightText line in tests/test_common/llm_data.py and change the year
range to 2022-2026.
---
Nitpick comments:
In `@tests/integration/defs/accuracy/test_llm_api_autodeploy.py`:
- Around line 812-816: EXTRA_EVALUATOR_KWARGS is a shared mutable dict that can
leak state between tests; replace it with a small helper method (e.g., def
_extra_evaluator_kwargs(self): ...) that returns a fresh dict with
apply_chat_template=True, fewshot_as_multiturn=True, and
chat_template_kwargs={'enable_thinking': False}, and update both
task.evaluate(..., extra_evaluator_kwargs=...) call sites to pass
self._extra_evaluator_kwargs() instead of the module/class-level
EXTRA_EVALUATOR_KWARGS.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 8db2a9df-4ed9-4c2f-a742-c0a515b63bea
📒 Files selected for processing (3)
examples/auto_deploy/model_registry/configs/qwen3.5_moe_400b.yamltests/integration/defs/accuracy/test_llm_api_autodeploy.pytests/test_common/llm_data.py
|
@taylor-yb-lee If this model needs to be in the CI, you might want to add this test to Please also update the |
Signed-off-by: Taylor Yeonbok Lee <249374542+taylor-yb-lee@users.noreply.github.com>
|
/bot skip "added only post-merge tests" |
GitHub Bot Help
Provide a user friendly way for developers to interact with a Jenkins server. Run See details below for each supported subcommand. Details
Launch build/test pipelines. All previously running jobs will be killed.
kill
Kill all running builds associated with pull request. skip
Skip testing for latest commit on pull request. reuse-pipeline
Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break. |
|
/bot skip --comment "added only post-merge tests" |
|
PR_Github #43181 [ skip ] triggered by Bot. Commit: |
|
PR_Github #43181 [ skip ] completed with state |
3 participants
Summary by CodeRabbit
New Features
Tests
Description
Added Qwen3.5 NVFP4 accuracy test for AutoDeploy
Measured score :
MMLU: 85
GSM8K:
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.