Skip to content

test(hybrid): AUT-971 quarantine Nemotron QAD functional test#6013

Merged
ko3n1g merged 1 commit into
NVIDIA:mainfrom
svcnemo-autobot:fix/quarantine-nemotron-qad
Jul 24, 2026
Merged

test(hybrid): AUT-971 quarantine Nemotron QAD functional test#6013
ko3n1g merged 1 commit into
NVIDIA:mainfrom
svcnemo-autobot:fix/quarantine-nemotron-qad

Conversation

@svcnemo-autobot

Copy link
Copy Markdown
Collaborator

Background

The Nemotron QAD functional test failed exact total loss comparison in merge-queue run 30045775203, while the approximate check passed. The failure repeated three times with the same trajectory and needs isolated investigation.

What changed

  • Temporarily move the H100 QAD recipe to broken scopes.
  • Keep the test definition in-tree for straightforward re-enable.

Details

tests/test_utils/recipes/h100/mamba.yaml changes mr and mr-github to their -broken counterparts. No model code, golden values, tolerances, or test configuration changed.

Five sequential fresh-pod reproductions will determine whether the test is flaky or its exact golden is stale.

Tested

  • Autoformat check passed.
  • Recipe parser emitted no active workflow for the exact QAD case.
  • Gates passed in the failing run's CI image on pod ci-events-agent-write-646e17de.

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
svcnemo-autobot requested a review from a team as a code owner July 24, 2026 00:11
@svcnemo-autobot
svcnemo-autobot requested a review from ko3n1g July 24, 2026 00:11
@github-actions

Copy link
Copy Markdown
Contributor

This PR has been automatically converted to draft because all PRs must start as drafts.

When you are ready for review, click Ready for Review to begin the review process. This will:

  1. Add the oncall reviewer (optional reviewer)
  2. Add required review teams based on your changes

See the contribution guide for more details.

@svcnvidia-nemo-ci
svcnvidia-nemo-ci marked this pull request as draft July 24, 2026 00:11
@copy-pr-bot

copy-pr-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@ko3n1g
ko3n1g marked this pull request as ready for review July 24, 2026 00:12
@ko3n1g
ko3n1g merged commit ff60360 into NVIDIA:main Jul 24, 2026
23 of 29 checks passed
@svcnemo-autobot

Copy link
Copy Markdown
Collaborator Author

/ok to test 44a1a4b

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants