feat(inference): add declarative DGX Spark llama.cpp profile - #8224
Conversation
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughAdds a declarative Nemotron 3 Nano llama.cpp recipe and explicit DGX Spark preset. Extends llama.cpp contracts, schema validation, readiness matching, adapter registries, catalog loading, managed-catalog projection, and compiler coverage. Changesllama.cpp serving integration
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant compiledServingCatalog as compiled serving catalog
participant catalogLoader as catalog loader
participant managedProjection as managed catalog projection
participant servingAPI as serving lookup API
compiledServingCatalog->>catalogLoader: load and cache full catalog
catalogLoader->>managedProjection: select managed vLLM definitions
managedProjection->>managedProjection: filter, recalculate digest, validate
servingAPI->>catalogLoader: request recipe or preset by ID
catalogLoader-->>servingAPI: return compiled definition
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
PR Review Advisor — No blocking findings reportedAdvisor assessment: No blocking advisor findings reported Model lanes
Second-opinion terminology and E2E selections are advisory. They do not change the primary assessment or E2E / PR Gate. 1 semantic terminology decisionTerminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.
E2E guidanceAdvisory only. E2E / PR Gate selects and runs jobs independently. Recommended E2E: 1 optional E2E recommendation
This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
src/lib/inference/serving/catalog-loader.test.ts (1)
99-111: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winTest retention of a managed definition.
This test only proves that an all-host-local catalog becomes empty. A predicate that removes every recipe and preset would also pass.
Add one registered vLLM recipe and preset to this fixture. Assert that the managed definition remains while
HOST_LOCAL_RECIPEandHOST_LOCAL_PRESETare excluded.As per path instructions, “Tests must prove that public entrypoints reach the new path and that the old path is deleted or cannot execute.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lib/inference/serving/catalog-loader.test.ts` around lines 99 - 111, Extend the test case around managedInferenceCatalogFromServingCatalog by adding a registered vLLM recipe and preset alongside HOST_LOCAL_RECIPE and HOST_LOCAL_PRESET. Assert that the managed catalog retains the registered definitions while excluding both host-local definitions, and update the digest assertion as needed to reflect the retained managed content.Sources: Coding guidelines, Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@managed-inference/schemas/recipe.schema.json`:
- Around line 1088-1102: Close the llama.cpp-specific schemas, including the
nested objects covered by the corresponding properties branch, so unsupported
generic fields are rejected at compile time. Update the compiler handling for
LlamaCppServingRecipe to reject bindings, execution.nodeCount,
runtime.networkMode, serve.executable, and each other generic-only field class
absent from the llama.cpp contract, while preserving the fixed single-host
profile.
In `@src/lib/inference/serving/catalog-loader.ts`:
- Around line 128-134: Add an entrypoint-level test in catalog-loader.test.ts
that calls loadManagedInferenceCatalog() and verifies host-local definitions
such as llama.cpp are excluded from the returned managed catalog. Keep the
existing direct projection test, and assert the filtering through the public
loader entrypoint rather than calling managedInferenceCatalogFromServingCatalog
directly.
---
Nitpick comments:
In `@src/lib/inference/serving/catalog-loader.test.ts`:
- Around line 99-111: Extend the test case around
managedInferenceCatalogFromServingCatalog by adding a registered vLLM recipe and
preset alongside HOST_LOCAL_RECIPE and HOST_LOCAL_PRESET. Assert that the
managed catalog retains the registered definitions while excluding both
host-local definitions, and update the digest assertion as needed to reflect the
retained managed content.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 26618257-fdcd-422e-add7-6feb8fe4dc4b
📒 Files selected for processing (10)
managed-inference/presets/llama-cpp.dgx-spark-gb10.single.nemotron-3-nano-30b-a3b.yamlmanaged-inference/recipes/llama-cpp.nemotron-3-nano-30b-a3b.spark-single.v1.yamlmanaged-inference/schemas/recipe.schema.jsonsrc/lib/inference/serving/adapter-registry.tssrc/lib/inference/serving/catalog-loader.test.tssrc/lib/inference/serving/catalog-loader.tssrc/lib/inference/serving/catalog.test.tssrc/lib/inference/serving/catalog.tssrc/lib/inference/serving/types.tstest/managed-inference-catalog-compiler.test.ts
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in commit df13c02 in the TypeScript / code-coverage/cliThe overall coverage in commit df13c02 in the Show a code coverage summary of the most impacted files.
Updated |
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
There was a problem hiding this comment.
🧹 Nitpick comments (1)
src/lib/inference/serving/catalog-loader.test.ts (1)
88-96: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winAssert the complete projected definition sets.
expectOnlyManagedVllmDefinitions()checks only non-empty collections and backend labels. It does not verify that every managed vLLM recipe and preset is retained, that host-local definitions are absent, or that unrelatedsourcesare removed. A projection regression can therefore pass both tests.Assert the expected recipe, preset, and source ID sets from the fixtures. For the public loader test, compare the result with the expected projected catalog. Do not copy the production filter into the test.
As per path instructions, tests must prove observable behavior. The PR objective also requires retaining managed definitions and filtering their sources.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lib/inference/serving/catalog-loader.test.ts` around lines 88 - 96, Expand expectOnlyManagedVllmDefinitions() to assert the exact recipe and preset ID sets expected by the fixtures, ensuring all managed vLLM definitions are retained and host-local definitions are excluded. Also assert the exact projected source ID set, excluding unrelated sources, and update the public loader test to compare its result with the expected projected catalog without duplicating the production filtering logic.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/lib/inference/serving/catalog-loader.test.ts`:
- Around line 88-96: Expand expectOnlyManagedVllmDefinitions() to assert the
exact recipe and preset ID sets expected by the fixtures, ensuring all managed
vLLM definitions are retained and host-local definitions are excluded. Also
assert the exact projected source ID set, excluding unrelated sources, and
update the public loader test to compare its result with the expected projected
catalog without duplicating the production filtering logic.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 8848862c-602d-4e1c-904d-f2ae11d52057
📒 Files selected for processing (4)
managed-inference/schemas/recipe.schema.jsonsrc/lib/inference/serving/catalog-loader.test.tssrc/lib/inference/serving/catalog-loader.tssrc/lib/inference/serving/catalog.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- src/lib/inference/serving/catalog-loader.ts
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
wscurran
added
area: inference
2 participants
Summary
Add an explicit-only DGX Spark llama.cpp profile for NVIDIA Nemotron to the canonical managed-inference YAML catalog. Preserve the automatic vLLM profile and exclude host-local recipes from the active managed-cluster runtime until later lifecycle and qualification slices land.
Related Issue
Fixes #8173
Changes
Type of Change
Quality Gates
Documentation Writer Review
no-docs-neededdf13c021eresolves the writing findings. The profile remains explicit-only and excluded from the active managed-cluster runtime, so this PR adds no user-facing workflow to document.DGX Station Hardware Evidence
Verification
Signed-off-by:line and every commit appears asVerifiedin GitHubpre-commit,commit-msg, andpre-pushhooks passed, ornpm run validate:prpassed after refreshingorigin/mainwhen hooks were skipped or unavailablenpm testfor broad runtime/test-harness changes;npm run checkfor repo-wide validation/coverage changes — command/result:npm run build:clipassed. No broad runtime or repository-wide validation surface changed.npm run docsbuilds without warnings (doc changes only)Signed-off-by: Aaron Erickson aerickson@nvidia.com
Summary by CodeRabbit
New Features
Bug Fixes
Validation