test: add vLLM deployment tests for checkpoint robustness by adil-a · Pull Request #1656 · NVIDIA-NeMo/Automodel

adil-a · 2026-04-02T16:00:56Z

Summary

vLLM deployment verification tests that load consolidated checkpoints and compare greedy output token-for-token against HuggingFace
Supports both full token comparison and smoke test mode (--vllm_smoke_test)
14 model configs covering Llama, Qwen, Nemotron, Phi-4, Gemma, Baichuan, Mistral, GPT-OSS

Test plan

Depends on checkpoint robustness PR test: add checkpoint robustness functional tests #1606 (produces the checkpoints vLLM loads)
Run L2_vLLM_Deploy_*.sh scripts after checkpoint robustness tests complete

🤖 Generated with Claude Code

vLLM deployment verification tests that load consolidated checkpoints and compare greedy output token-for-token against HuggingFace. Supports both full comparison and smoke test mode. Depends on checkpoint robustness PR #1606. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: adil-a <adil.asif2000@hotmail.com>

copy-pr-bot · 2026-04-02T16:01:00Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com>

thomasdhc · 2026-04-03T20:32:33Z

@adil-a I refactored similar to what we have for checkpoint robustness have a look when you get a chance: Ministral-3-3B-Instruct-2512, Llama-3.1-Nemotron-Nano-8B-v1, Qwen3-30B-A3B aren't recipes we have in examples. Are these needed?

thomasdhc · 2026-04-03T20:33:35Z

tests/functional_tests/checkpoint_robustness/test_checkpoint_vllm_deploy_runner.py

+TEST_FOLDER = "checkpoint_robustness"
+
+
+class TestVLLMDeploy:


I've updated the checkpoints to be the ones from the finetune job, should it be from the robustness job?

After we make that update I think we can delete this file

akoumpa · 2026-04-06T20:15:55Z

@adil-a can we merge this?

thomasdhc · 2026-04-06T20:52:55Z

@akoumpa Not yet this pathway has not been tested yet

Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com>

thomasdhc · 2026-04-08T21:49:01Z

/ok to test 26054f2

* test: add vLLM deployment tests for checkpoint robustness vLLM deployment verification tests that load consolidated checkpoints and compare greedy output token-for-token against HuggingFace. Supports both full comparison and smoke test mode. Depends on checkpoint robustness PR #1606. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: adil-a <adil.asif2000@hotmail.com> * Create deploy-test dependency group Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Revert deploy test group Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Move configs to recipes and create vllm_launcher Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Setup deploy environment Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Remove duplicate keys Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Add scope to vllm deploy test Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Drop needs dependency Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Use finetune test name for ckpt dir Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Make ckpt checking more robust Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Pass arguments correctly Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Update arguments Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Remove unused file Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> --------- Signed-off-by: adil-a <adil.asif2000@hotmail.com> Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>

test: add vLLM deployment tests for checkpoint robustness (#1656) * test: add vLLM deployment tests for checkpoint robustness vLLM deployment verification tests that load consolidated checkpoints and compare greedy output token-for-token against HuggingFace. Supports both full comparison and smoke test mode. Depends on checkpoint robustness PR #1606. * Create deploy-test dependency group * Revert deploy test group * Move configs to recipes and create vllm_launcher * Setup deploy environment * Remove duplicate keys * Add scope to vllm deploy test * Drop needs dependency * Use finetune test name for ckpt dir * Make ckpt checking more robust * Pass arguments correctly * Update arguments * Remove unused file --------- Signed-off-by: adil-a <adil.asif2000@hotmail.com> Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Adil <47084919+adil-a@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>

* test: add vLLM deployment tests for checkpoint robustness vLLM deployment verification tests that load consolidated checkpoints and compare greedy output token-for-token against HuggingFace. Supports both full comparison and smoke test mode. Depends on checkpoint robustness PR #1606. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: adil-a <adil.asif2000@hotmail.com> * Create deploy-test dependency group Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Revert deploy test group Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Move configs to recipes and create vllm_launcher Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Setup deploy environment Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Remove duplicate keys Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Add scope to vllm deploy test Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Drop needs dependency Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Use finetune test name for ckpt dir Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Make ckpt checking more robust Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Pass arguments correctly Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Update arguments Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> * Remove unused file Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> --------- Signed-off-by: adil-a <adil.asif2000@hotmail.com> Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>

adil-a mentioned this pull request Apr 2, 2026

test: add checkpoint robustness functional tests #1606

Merged

9 tasks

Create deploy-test dependency group

f071d84

Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com>

thomasdhc requested a review from a team as a code owner April 3, 2026 19:32

thomasdhc added 2 commits April 3, 2026 15:39

Revert deploy test group

56525b3

Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com>

Move configs to recipes and create vllm_launcher

5363342

Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com>

thomasdhc requested review from HuiyingLi, ZhiyuLi-Nvidia, akoumpa, hemildesai and pthombre as code owners April 3, 2026 20:30

thomasdhc reviewed Apr 3, 2026

View reviewed changes

thomasdhc added the r0.4.0 Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge. label Apr 7, 2026

akoumpa and others added 11 commits April 6, 2026 23:38

Merge branch 'main' into adil-a/checkpoint-vllm-deploy

975b354

Setup deploy environment

c735eae

Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com>

Remove duplicate keys

dfb8890

Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com>

Add scope to vllm deploy test

476e5d3

Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com>

Drop needs dependency

f5b2914

Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com>

Use finetune test name for ckpt dir

4040a0f

Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com>

Make ckpt checking more robust

06c29b0

Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com>

Pass arguments correctly

22eaa53

Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com>

Update arguments

e1902f6

Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com>

Merge branch 'main' into adil-a/checkpoint-vllm-deploy

ce7bec4

Remove unused file

26054f2

Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com>

thomasdhc approved these changes Apr 8, 2026

View reviewed changes

copy-pr-bot bot temporarily deployed to test April 8, 2026 21:49 Inactive

copy-pr-bot bot temporarily deployed to nemo-ci April 8, 2026 21:49 Inactive

copy-pr-bot bot temporarily deployed to nemo-ci April 8, 2026 22:07 Inactive

copy-pr-bot bot temporarily deployed to nemo-ci April 8, 2026 22:20 Inactive

copy-pr-bot bot temporarily deployed to nemo-ci April 8, 2026 22:38 Inactive

akoumpa approved these changes Apr 9, 2026

View reviewed changes

akoumpa merged commit 3a3f685 into main Apr 9, 2026
56 checks passed

akoumpa deleted the adil-a/checkpoint-vllm-deploy branch April 9, 2026 07:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: add vLLM deployment tests for checkpoint robustness#1656

test: add vLLM deployment tests for checkpoint robustness#1656
akoumpa merged 15 commits intomainfrom
adil-a/checkpoint-vllm-deploy

adil-a commented Apr 2, 2026

Uh oh!

copy-pr-bot bot commented Apr 2, 2026

Uh oh!

thomasdhc commented Apr 3, 2026

Uh oh!

thomasdhc Apr 3, 2026

Uh oh!

thomasdhc Apr 3, 2026

Uh oh!

akoumpa commented Apr 6, 2026

Uh oh!

thomasdhc commented Apr 6, 2026

Uh oh!

thomasdhc commented Apr 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		TEST_FOLDER = "checkpoint_robustness"


		class TestVLLMDeploy:

Conversation

adil-a commented Apr 2, 2026

Summary

Test plan

Uh oh!

copy-pr-bot bot commented Apr 2, 2026

Uh oh!

thomasdhc commented Apr 3, 2026

Uh oh!

thomasdhc Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

thomasdhc Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

akoumpa commented Apr 6, 2026

Uh oh!

thomasdhc commented Apr 6, 2026

Uh oh!

thomasdhc commented Apr 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants