TEST: update qwen3.5 397b test by littlegy · Pull Request #4607 · InternLM/lmdeploy

littlegy · 2026-05-21T08:13:56Z

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Please describe the motivation of this PR and the goal you want to achieve through this PR.

Modification

Please briefly describe what modification is made in this PR.

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
If the modification has a dependency on downstream projects of a newer version, this PR should be tested with all supported versions of downstream projects.
The documentation has been modified accordingly, like docstring or example tutorials.

lvhan028 · 2026-05-23T09:32:58Z

+    if eval_subpath is None:
+        if eval_config_name == 'default':
+            if 'gpt' in run_config.get('model', '').lower():
+                eval_config_name = 'gpt'
+            elif 'intern-s1-pro' in run_config.get('model', '').lower():
+                eval_config_name = 'intern-s1-pro'
+            elif 'qwen3.5' in run_config.get('model', '').lower():
+                eval_config_name = 'qwen3.5'


Can we use run_config['model'].lower()

lvhan028 · 2026-05-23T09:33:56Z


-    if str(config.get('env_tag')) == 'ascend':
-        eval_config_name = f'{eval_config_name}-2batch'
+        if str(config.get('env_tag')) == 'ascend':


str(config['env_tag'])

lvhan028 · 2026-05-23T09:38:46Z

-def run_eval_test(config, run_config, worker_id, test_type='infer', eval_config_name='default'):
+def run_eval_test(config, run_config, worker_id, test_type='infer', eval_config_name='default', eval_subpath=None):
+    if eval_config_name == 'default':
+        if 'qwen3.5' in run_config.get('model', '').lower():


if 'qwen3.5' in run_config['model'].lower():

lvhan028 · 2026-05-25T04:24:13Z

+@contextmanager
+def _mmengine_lazy_allow_lazyattr_call():
+    try:
+        from mmengine.config import lazy as mm_lazy
+    except ImportError:
+        yield
+        return
+    cls = mm_lazy.LazyAttr
+    orig = cls.__call__
+
+    def _call(self, *args, **kwargs):
+        fn = self.build()
+        if not callable(fn):
+            raise RuntimeError()
+        return fn(*args, **kwargs)
+
+    cls.__call__ = _call
+    try:
+        yield
+    finally:
+        cls.__call__ = orig


May let us know the reason to bring it in

lvhan028 · 2026-05-25T11:38:38Z

+def _sync_ruler_tokenizer_model(cfg, model_path):
+    tokenizer = os.environ.get('TOKENIZER_MODEL', model_path)
+    if not getattr(cfg, 'datasets', None):
+        return
+    for dataset in cfg.datasets:
+        if isinstance(dataset, dict) and 'tokenizer_model' in dataset:
+            dataset['tokenizer_model'] = tokenizer


Could you let us know the reason to sync tokenizer?

Copilot

Pull request overview

This PR updates the autotest harness to better support Qwen3.5-397B(-FP8) evaluation, including new dp/ep distributed test coverage and more consistent evaluation preset selection based on model name + env_tag.

Changes:

Add Qwen3.5-397B-A17B (and FP8 variants) to autotest YAML matrices and adjust runtime params (e.g., batch size).
Centralize eval preset selection via resolve_eval_config_name() and apply it across API + MLLM evaluation tests.
Improve evaluation stability/diagnostics (mmengine LazyAttr loading workaround, tokenizer_model sync, richer traceback, longer MLLM infer timeout) and refine proxy distributed readiness checks for dp-sized clusters.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
autotest/utils/proxy_distributed_utils.py	Update proxy node readiness reporting and expected node counting for dp-sized distributed runs.
autotest/utils/evaluate_utils.py	Add mmengine LazyAttr workaround, tokenizer sync, conditional NUM_WORKERS override, and improved error reporting; extend MLLM infer timeout.
autotest/utils/config_utils.py	Add fp8 model-format case generation, tune Qwen3.5-397B params, generalize MTP params, add `resolve_eval_config_name()`, and make case parsing more robust.
autotest/evaluate/test_mllm_api_evaluate.py	Use centralized eval preset resolution; add proxy-distributed dp4ep8 MLLM infer test; support eval output subpaths.
autotest/evaluate/test_api_evaluate.py	Replace inline preset selection with `resolve_eval_config_name()`; add dp4ep8 distributed + longtext variants; pass eval preset into `eval_test()`.
autotest/config_h.yml	Extend model lists/parallel configs/quantization settings to include Qwen3.5-397B-A17B and FP8 variants.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+
+
+def _apply_eval_config_env_suffix(config: dict[str, Any], name: str) -> str:
+    env_tag = str(config['env_tag'])


+    def _call(self, *args, **kwargs):
+        fn = self.build()
+        if not callable(fn):
+            raise RuntimeError()


* TEST: update qwen3.5 397b test * fix comments

TEST: update qwen3.5 397b test

acd4a66

lvhan028 reviewed May 23, 2026

View reviewed changes

lvhan028 reviewed May 25, 2026

View reviewed changes

littlegy added 2 commits May 25, 2026 16:50

fix comments

6452af2

Merge branch 'InternLM:main' into qwen397b_test

7e14ac2

lvhan028 approved these changes May 25, 2026

View reviewed changes

lvhan028 reviewed May 25, 2026

View reviewed changes

lvhan028 approved these changes May 25, 2026

View reviewed changes

lvhan028 requested a review from Copilot May 25, 2026 12:21

Copilot started reviewing on behalf of lvhan028 May 25, 2026 12:21 View session

Copilot AI reviewed May 25, 2026

View reviewed changes

lvhan028 merged commit 0cc6dfb into InternLM:main May 25, 2026
4 checks passed

lvhan028 pushed a commit to lvhan028/lmdeploy that referenced this pull request May 27, 2026

TEST: update qwen3.5 397b test (InternLM#4607)

ab230b2

* TEST: update qwen3.5 397b test * fix comments

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TEST: update qwen3.5 397b test#4607

TEST: update qwen3.5 397b test#4607
lvhan028 merged 3 commits into
InternLM:mainfrom
littlegy:qwen397b_test

littlegy commented May 21, 2026

Uh oh!

lvhan028 May 23, 2026

Uh oh!

lvhan028 May 23, 2026

Uh oh!

lvhan028 May 23, 2026

Uh oh!

lvhan028 May 25, 2026

Uh oh!

lvhan028 May 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants



		def _apply_eval_config_env_suffix(config: dict[str, Any], name: str) -> str:
		env_tag = str(config['env_tag'])

Conversation

littlegy commented May 21, 2026

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

Uh oh!

lvhan028 May 23, 2026

Choose a reason for hiding this comment

Uh oh!

lvhan028 May 23, 2026

Choose a reason for hiding this comment

Uh oh!

lvhan028 May 23, 2026

Choose a reason for hiding this comment

Uh oh!

lvhan028 May 25, 2026

Choose a reason for hiding this comment

Uh oh!

lvhan028 May 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants