fix(auditor): Update auditor usages#221
Merged
anastasia-nesterenko merged 6 commits intoJun 10, 2026
Merged
Conversation
Signed-off-by: anastasia-nesterenko <anesterenko@nvidia.com>
anastasia-nesterenko
force-pushed
the
anesterenko/aircore-739-nvbug-310-baseline-tasks-fail-auditor-cli-command-mismatch
branch
from
June 9, 2026 00:15
67373b9 to
1974684
Compare
…asks-fail-auditor-cli-command-mismatch
Contributor
|
Contributor
📝 WalkthroughWalkthroughThis PR migrates the auditor CLI workflow from job-based ( ChangesAuditor CLI Workflow Migration
Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
tests/agentic-use/auditor-default-job-cli-easy/instruction.md (1)
1-1:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winUpdate title to match current workflow.
Title says "Run Default Audit Job (CLI)" but workflow no longer creates audit jobs. Consider "Run Default Audit (CLI)" or "Run Default Audit with CLI".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/agentic-use/auditor-default-job-cli-easy/instruction.md` at line 1, Update the document title in instruction.md by replacing the current heading text "Run Default Audit Job (CLI)" with a title that reflects the updated workflow (suggestions: "Run Default Audit (CLI)" or "Run Default Audit with CLI"); ensure the H1 heading line at the top of the file is changed accordingly so the file's title matches the current behavior.tests/agentic-use/auditor-default-job-cli/instruction.md (1)
1-1:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winUpdate title to match current workflow.
Title says "Run Default Audit Job (CLI)" but workflow no longer creates audit jobs. Consider "Run Default Audit (CLI)" or "Run Default Audit with CLI".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/agentic-use/auditor-default-job-cli/instruction.md` at line 1, Update the markdown title string "Run Default Audit Job (CLI)" to reflect the current workflow that no longer creates audit jobs—replace it with a clearer title such as "Run Default Audit (CLI)" or "Run Default Audit with CLI" by editing the top-level heading in instruction.md (the line containing "Run Default Audit Job (CLI)"); ensure the new title remains a top-level heading (prefixed with "# ") and matches other documentation naming conventions.
🧹 Nitpick comments (6)
web/packages/studio/src/routes/agents/AgentSuggestionsRoute/utils.spec.ts (1)
463-467: ⚡ Quick winTest does not enforce the required auditor run command.
Line 463-467 should also assert
suggested_actionsincludesnemo auditor audit run --specso the migrated workflow contract is covered.Proposed test update
expect(newModel?.suggested_actions?.[0]).toContain('nemo auditor targets create'); expect(newModel?.suggested_actions?.[0]).toContain('"model": "new-model-7b"'); + expect( + newModel?.suggested_actions?.some((a) => a.includes('nemo auditor audit run --spec')) + ).toBe(true); expect(newModel?.suggested_actions?.[1]).toContain( 'nemo evaluation jobs create --model new-model-7b' );🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/src/routes/agents/AgentSuggestionsRoute/utils.spec.ts` around lines 463 - 467, The test is missing an assertion that the migrated model's suggested_actions includes the required auditor run command; update the spec in utils.spec.ts to assert that newModel?.suggested_actions contains the string "nemo auditor audit run --spec" (e.g., add expect(newModel?.suggested_actions).toContain('nemo auditor audit run --spec') or expect(newModel?.suggested_actions?.[2]).toContain('nemo auditor audit run --spec') alongside the existing checks against newModel and suggested_actions).tests/agentic-use/auditor-default-job-cli-easy/tests/test_outputs.py (1)
50-61: 💤 Low valueAdd explicit return type annotations.
Test functions lack
-> Nonereturn type annotations. Per coding guidelines, always use concrete type hints.Also applies to: 67-77, 83-92, 95-110
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/agentic-use/auditor-default-job-cli-easy/tests/test_outputs.py` around lines 50 - 61, Several test functions (e.g., test_audit_target_exists, test_audit_target_model and the other tests around lines 67-77, 83-92, 95-110) are missing explicit return type annotations; update each test function signature to include the concrete return type "-> None". Locate the functions by their names (test_audit_target_exists, test_audit_target_model, and the other test_* functions mentioned) and add "-> None" to their defs so all tests conform to the project's typing guideline.Source: Coding guidelines
tests/agentic-use/auditor-default-job-cli/instruction.md (1)
13-27: ⚡ Quick winConsider adding CLI command examples.
Other instruction variants (auditor-default-job-cli-easy and auditor-custom-probes-cli) provide concrete CLI command examples. This variant only lists high-level steps without showing commands. Per coding guidelines, use concrete examples with real commands.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/agentic-use/auditor-default-job-cli/instruction.md` around lines 13 - 27, Add concrete nmp CLI command examples showing how to create the audit target named audit-target (referencing model aws/anthropic/bedrock-claude-sonnet-4-5-v1 via provider nvidia-inference), how to create the audit config named default with representative flags/options, and how to invoke the audit run using the default config and audit-target; include the exact nmp subcommands and arguments (e.g., create-target/create-config/run or the project’s equivalent) so reviewers can copy-paste and verify that audit-target and default are created and that the audit run command is executed with both.Source: Coding guidelines
tests/agentic-use/auditor-custom-probes-cli/tests/test_outputs.py (1)
53-57: 💤 Low valueAdd explicit return type annotations.
Test functions lack
-> Nonereturn type annotations. Per coding guidelines, always use concrete type hints.Example fix
-def test_audit_target_exists(client: NeMoPlatform): +def test_audit_target_exists(client: NeMoPlatform) -> None:Also applies to: 60-64, 70-74, 77-82, 102-112, 117-133, 146-162, 164-179
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/agentic-use/auditor-custom-probes-cli/tests/test_outputs.py` around lines 53 - 57, Several test functions are missing explicit return type annotations; update each test function (e.g., test_audit_target_exists and the other test functions in the ranges referenced) to include an explicit "-> None" return type. Locate each top-level test function definition (def test_*(...):) and change the signature to add "-> None" before the colon so all tests follow the concrete type-hint guideline.Source: Coding guidelines
tests/agentic-use/auditor-custom-probes-cli/README.md (1)
21-22: 💤 Low valueOptional: Avoid word repetition.
"right probes" and "right model" in close proximity. Consider "correct model" or "expected model".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/agentic-use/auditor-custom-probes-cli/README.md` around lines 21 - 22, The sentence under the "Setup correctness" bullet uses repeated wording ("right probes" and "right model"); edit that bullet (the "Setup correctness" line) to replace one or both instances with alternatives such as "correct probes" or "expected model" so the phrase reads smoothly (for example: "config has the correct probes, target references the expected model, and the audit command references both config + target").tests/agentic-use/auditor-default-job-cli/tests/test_outputs.py (1)
50-61: 💤 Low valueAdd explicit return type annotations.
Test functions lack
-> Nonereturn type annotations. Per coding guidelines, always use concrete type hints.Also applies to: 67-77, 83-92, 95-110
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/agentic-use/auditor-default-job-cli/tests/test_outputs.py` around lines 50 - 61, Add explicit return type annotations "-> None" to the test functions; specifically update the signatures of test_audit_target_exists and test_audit_target_model (and any other test_... functions in this test file noted by the reviewer) to include "-> None" so they read e.g. def test_audit_target_exists(client: NeMoPlatform) -> None: and def test_audit_target_model(client: NeMoPlatform) -> None:.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@web/packages/studio/src/routes/agents/AgentSuggestionsRoute/utils.ts`:
- Around line 371-374: The suggested_actions array in utils.ts for the new-model
suggestion flow is missing the audit execution step; update the
suggested_actions entry (the array named suggested_actions) to include a nemo
auditor audit run --spec command (using the appropriate spec or placeholder tied
to the created target/model) so the workflow runs the audit after creating the
target (e.g. insert a `nemo auditor audit run --spec <spec>`-style command in
the array, placed after the target creation entry and before or alongside the
evaluation jobs entry) to ensure audits are executed as part of the migrated
flow.
---
Outside diff comments:
In `@tests/agentic-use/auditor-default-job-cli-easy/instruction.md`:
- Line 1: Update the document title in instruction.md by replacing the current
heading text "Run Default Audit Job (CLI)" with a title that reflects the
updated workflow (suggestions: "Run Default Audit (CLI)" or "Run Default Audit
with CLI"); ensure the H1 heading line at the top of the file is changed
accordingly so the file's title matches the current behavior.
In `@tests/agentic-use/auditor-default-job-cli/instruction.md`:
- Line 1: Update the markdown title string "Run Default Audit Job (CLI)" to
reflect the current workflow that no longer creates audit jobs—replace it with a
clearer title such as "Run Default Audit (CLI)" or "Run Default Audit with CLI"
by editing the top-level heading in instruction.md (the line containing "Run
Default Audit Job (CLI)"); ensure the new title remains a top-level heading
(prefixed with "# ") and matches other documentation naming conventions.
---
Nitpick comments:
In `@tests/agentic-use/auditor-custom-probes-cli/README.md`:
- Around line 21-22: The sentence under the "Setup correctness" bullet uses
repeated wording ("right probes" and "right model"); edit that bullet (the
"Setup correctness" line) to replace one or both instances with alternatives
such as "correct probes" or "expected model" so the phrase reads smoothly (for
example: "config has the correct probes, target references the expected model,
and the audit command references both config + target").
In `@tests/agentic-use/auditor-custom-probes-cli/tests/test_outputs.py`:
- Around line 53-57: Several test functions are missing explicit return type
annotations; update each test function (e.g., test_audit_target_exists and the
other test functions in the ranges referenced) to include an explicit "-> None"
return type. Locate each top-level test function definition (def test_*(...):)
and change the signature to add "-> None" before the colon so all tests follow
the concrete type-hint guideline.
In `@tests/agentic-use/auditor-default-job-cli-easy/tests/test_outputs.py`:
- Around line 50-61: Several test functions (e.g., test_audit_target_exists,
test_audit_target_model and the other tests around lines 67-77, 83-92, 95-110)
are missing explicit return type annotations; update each test function
signature to include the concrete return type "-> None". Locate the functions by
their names (test_audit_target_exists, test_audit_target_model, and the other
test_* functions mentioned) and add "-> None" to their defs so all tests conform
to the project's typing guideline.
In `@tests/agentic-use/auditor-default-job-cli/instruction.md`:
- Around line 13-27: Add concrete nmp CLI command examples showing how to create
the audit target named audit-target (referencing model
aws/anthropic/bedrock-claude-sonnet-4-5-v1 via provider nvidia-inference), how
to create the audit config named default with representative flags/options, and
how to invoke the audit run using the default config and audit-target; include
the exact nmp subcommands and arguments (e.g., create-target/create-config/run
or the project’s equivalent) so reviewers can copy-paste and verify that
audit-target and default are created and that the audit run command is executed
with both.
In `@tests/agentic-use/auditor-default-job-cli/tests/test_outputs.py`:
- Around line 50-61: Add explicit return type annotations "-> None" to the test
functions; specifically update the signatures of test_audit_target_exists and
test_audit_target_model (and any other test_... functions in this test file
noted by the reviewer) to include "-> None" so they read e.g. def
test_audit_target_exists(client: NeMoPlatform) -> None: and def
test_audit_target_model(client: NeMoPlatform) -> None:.
In `@web/packages/studio/src/routes/agents/AgentSuggestionsRoute/utils.spec.ts`:
- Around line 463-467: The test is missing an assertion that the migrated
model's suggested_actions includes the required auditor run command; update the
spec in utils.spec.ts to assert that newModel?.suggested_actions contains the
string "nemo auditor audit run --spec" (e.g., add
expect(newModel?.suggested_actions).toContain('nemo auditor audit run --spec')
or expect(newModel?.suggested_actions?.[2]).toContain('nemo auditor audit run
--spec') alongside the existing checks against newModel and suggested_actions).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: ff1efd7b-e31e-4d1f-8525-605e5b383771
⛔ Files ignored due to path filters (2)
sdk/python/nemo-platform/README.mdis excluded by!sdk/**sdk/python/overrides/nemo-platform/README/03_usage.mdis excluded by!sdk/**
📒 Files selected for processing (11)
plugins/nemo-agents/src/nemo_agents_plugin/skills/agents-optimize/SKILL.mdpytest.initests/agentic-use/auditor-custom-probes-cli/README.mdtests/agentic-use/auditor-custom-probes-cli/instruction.mdtests/agentic-use/auditor-custom-probes-cli/tests/test_outputs.pytests/agentic-use/auditor-default-job-cli-easy/instruction.mdtests/agentic-use/auditor-default-job-cli-easy/tests/test_outputs.pytests/agentic-use/auditor-default-job-cli/instruction.mdtests/agentic-use/auditor-default-job-cli/tests/test_outputs.pyweb/packages/studio/src/routes/agents/AgentSuggestionsRoute/utils.spec.tsweb/packages/studio/src/routes/agents/AgentSuggestionsRoute/utils.ts
💤 Files with no reviewable changes (1)
- pytest.ini
anastasia-nesterenko
changed the title
…asks-fail-auditor-cli-command-mismatch
…asks-fail-auditor-cli-command-mismatch
tylersbray
approved these changes
Jun 9, 2026
tylersbray
left a comment
tylersbray
left a comment
Contributor
There was a problem hiding this comment.
Looks good. I did a pass with an AI code review, there were two things that popped out:
- Run at least one updated task in Harbor (e.g. auditor-default-job-cli-easy) to confirm:
- client.auditor.* calls succeed against the eval container API
- Agents following new instructions pass the verifier
- Could be a little tighter in auditor-default-job-cli-easy/tests/test_outputs.py. There is this snippet:
has_audit_run = any(
all(p in cmd for p in ["auditor", "audit", "run", CONFIG_NAME, TARGET_NAME])
for cmd in commands
)
It quibbled about these issues:
- CONFIG_NAME = "default" is an extremely common substring — can false-pass on unrelated commands containing "default".
- No _has_token() — custom-probes task uses word-boundary regex; default-job does not.
- Does not verify qualified spec — instructions say default/audit-target; test only checks bare name substrings.
- No target-create trajectory check — only config interaction is trajectory-tested.
Recommendation: Port _has_token() from custom-probes and require audit-target + either default/audit-target or default/default in the run command.
I'm not sure how set up harbor is at this point, I think we were waiting on some NAT changes at one point? Tightening up the tests might be good to do a little more than just look for strings.
…asks-fail-auditor-cli-command-mismatch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes auditor agentic-use task instructions and verifiers to use the current
nemo auditorCLI surface instead of the removednemo audit/ audit-job commands.Linear: AIRCORE-739
Changes
audit jobs create/get-statustoauditor audit runclient.auditor, for auditor configs and targetsservices/auditor/testspytest discovery pathTesting
Closes: AIRCORE-739
Summary by CodeRabbit
Release Notes
Documentation
Tests