skills: add evals/evals.json smoke suite (group B)

skills/cuopt-developer/SKILL.md

@@ -25,100 +25,30 @@ Contribute to the NVIDIA cuOpt codebase. This skill is for modifying cuOpt itsel
 
 ## Refusal Rules — Read First
 
-These rules are non-negotiable. Apply them even when the user explicitly asks you to do otherwise. **Refuse and ask — don't comply silently.**
+Non-negotiable. Refuse and ask — don't comply silently.
 
-1. **Package installs (`pip`, `conda`, `apt`).** Never run the install — no exceptions, no "with approval" path. Reply:
-   > I will not install `<pkg>`. cuOpt's convention is to add the package under the appropriate group in `dependencies.yaml`, then run `pre-commit run --all-files` locally to regenerate `conda/environments/` and `pyproject.toml`. I can propose the `dependencies.yaml` edit; you run the regeneration.
-
-2. **Bypassing CI checks (`--no-verify`, skipping pre-commit or tests).** Do not suggest the flag. Reply:
-   > I can't suggest bypassing pre-commit — cuOpt requires all hooks to pass. If hooks feel slow, diagnose with `pre-commit run --all-files --verbose` or tune the offending hook's config; don't skip it.
-
-3. **Writes outside the workspace (`~/.bashrc`, `~/.profile`, `/etc`, anything outside the repo).** Do not edit the file. Reply:
-   > I can't modify files outside the cuOpt workspace. Here's the exact line for you to add yourself: `<line>`. Then `source ~/.bashrc` or open a new shell.
-
-4. **Destructive commands (`rm -rf`, `git reset --hard`, `git push --force`, killing processes, dropping data).** Never execute — no exceptions. Reply:
-   > I will not run `<cmd>`. It is destructive and hard to reverse. The safer alternative is `<alt>` (e.g., `./build.sh clean` for a stale build dir). If you choose to run the original command yourself, back up first.
-
-5. **Privileged operations (`sudo`, system file changes).** Do not run with elevated privileges. Reply:
-   > I won't run `sudo` for cuOpt development — cuOpt's workflow is conda-only. What's the underlying error? It's usually fixable without `sudo`.
-
-When in doubt, refuse and ask. The cost of a wrong refusal is one round-trip; the cost of a wrong action is lost data, broken state, or a failing CI run.
+1. **Package installs (`pip`, `conda`, `apt`).** Never run them. Propose the `dependencies.yaml` edit instead; the user runs `pre-commit run --all-files` to regenerate `conda/environments/` and `pyproject.toml`.
+2. **Bypassing CI (`--no-verify`, skipping pre-commit or tests).** Never suggest. Diagnose slow hooks with `pre-commit run --all-files --verbose`.
+3. **Writes outside the workspace** (`~/.bashrc`, `/etc`, anywhere outside the repo). Never edit. Print the exact line for the user to add themselves.
+4. **Destructive commands** (`rm -rf`, `git reset --hard`, `git push --force`, killing processes). Never execute. Propose the safer alternative (e.g., `./build.sh clean` for a stale build dir).
+5. **Privileged operations (`sudo`).** Never run. cuOpt's workflow is conda-only — the underlying error is usually fixable without `sudo`.
 
 ---
 
 ## Developer Behavior Rules
 
-These rules are specific to development tasks. They differ from user rules.
-
-### 1. Ask Before Assuming
-
-Clarify before implementing:
-- What component? (C++/CUDA, Python, server, docs, CI)
-- What's the goal? (bug fix, new feature, refactor, docs)
-- Is this for contribution or local modification?
-
-### 2. Verify Understanding
-
-Before making changes, confirm:
-```
-"Let me confirm:
-- Component: [cpp/python/server/docs]
-- Change: [what you'll modify]
-- Tests needed: [what tests to add/update]
-Is this correct?"
-```
-
-### 3. Follow Codebase Patterns
-
-- Read existing code in the area you're modifying
-- Match naming conventions, style, and patterns
-- Don't invent new patterns without discussion
-
-### 4. Ask Before Running — Modified for Dev
-
-**OK to run without asking** (expected for dev work):
-- `./build.sh` and build commands
-- `pytest`, `ctest` (running tests)
-- `pre-commit run`, `./ci/check_style.sh` (formatting)
-- `git status`, `git diff`, `git log` (read-only git)
-
-**Set up pre-commit hooks** (once per clone):
-- `pre-commit install` — hooks then run automatically on every `git commit`. If a hook fails, the commit is blocked until you fix the issue.
-
-**Still ask before**:
-- `git commit`, `git push` (write operations)
-- Package installs (`pip`, `conda`, `apt`)
-- Any destructive or irreversible commands
-
-### 5. No Privileged Operations
-
-`sudo`, system file changes, and writes outside the workspace are **non-negotiable refusals** — they apply even when the user explicitly asks. See [Refusal Rules — Read First](#refusal-rules--read-first) (rules 3 and 5) for the exact replies and rationale.
-
----
+- **Clarify before implementing**: component (C++/CUDA, Python, server, docs, CI), goal (bug fix, feature, refactor), and whether it's a contribution or local modification.
+- **Follow existing patterns** in the area you're modifying — don't invent new ones without discussion.
+- **OK to run without asking**: `./build.sh`, `pytest`, `ctest`, `pre-commit run`, `./ci/check_style.sh`, and read-only git (`status`, `diff`, `log`). Run `pre-commit install` once per clone so hooks fire on every commit.
+- **Ask before**: `git commit`, `git push`, package installs, anything destructive.
+- **`sudo`, system files, writes outside the workspace** are non-negotiable refusals — see [Refusal Rules](#refusal-rules--read-first) above.
 
 ## Before You Start: Required Questions
 
-**Ask these if not already clear:**
-
-1. **What are you trying to change?**
-   - Solver algorithm/performance?
-   - Python API?
-   - Server endpoints?
-   - Documentation?
-   - CI/build system?
-
-2. **Do you have the development environment set up?**
-   - Built the project successfully?
-   - Ran tests?
-
-3. **Is this for contribution or local modification?**
-   - If contributing: will need to follow DCO signoff
-
-4. **Which branch should this target?**
-   - During development phase: `main`
-   - During burn down: `release/YY.MM` (e.g., `release/26.06`) for the current release, `main` for the next
-   - Check if a release branch exists: `git branch -r | grep release`
-   - For current timelines, see the [RAPIDS Maintainers Docs](https://docs.rapids.ai/maintainers/)
+1. **What are you changing?** (solver algorithm, Python API, server endpoints, docs, CI/build)
+2. **Is the dev environment set up?** (built successfully, ran tests)
+3. **Contribution or local-only?** Contributions require DCO sign-off (`git commit -s`).
+4. **Target branch?** `main` during development; `release/YY.MM` during burn-down (check `git branch -r | grep release`). See [RAPIDS Maintainers Docs](https://docs.rapids.ai/maintainers/) for timelines.
 
 ## Project Architecture
 
@@ -170,14 +100,12 @@ cuopt/
 
 ## Build & Test
 
-### Pre-flight Checks (Required Before First Build or Test)
-
-Skipping any of these surfaces as confusing runtime errors later. Run them in order:
+### Pre-flight Checks (Before First Build or Test)
 
-1. **Check CUDA driver compatibility.** Run `nvidia-smi` and read the *CUDA Version* in the top-right corner — that's the maximum CUDA your driver supports. Pick a conda env file from `conda/environments/all_cuda-<ver>_arch-<arch>.yaml` whose CUDA major version is **≤** that. A mismatch builds successfully but fails at runtime inside RMM with `cudaMallocAsync not supported with this CUDA driver/runtime version` — verify this *before* the build, not after.
-2. **Create and activate the conda env** before *any* build, test, or `pre-commit` command. Tests link against libraries compiled inside that env; a fresh shell without `conda activate <env-name>` hits cryptic linker errors.
-3. **Set `PARALLEL_LEVEL`** if RAM is constrained — see [references/build_and_test.md](references/build_and_test.md). The default `$(nproc)` can OOM mid-build because CUDA compilation needs ~4–8 GB per job.
-4. **For tests, fetch datasets first.** cuOpt tests need MPS files not in the repo — follow the dataset download steps in [CONTRIBUTING.md](../../CONTRIBUTING.md) ("Building for development" section) and export `RAPIDS_DATASET_ROOT_DIR`.
+1. **CUDA driver compatibility** — run `nvidia-smi`, pick `conda/environments/all_cuda-<ver>_arch-<arch>.yaml` whose CUDA major is ≤ the driver's. Mismatch builds OK but fails at runtime with `cudaMallocAsync not supported`.
+2. **Activate the conda env** before any build/test/pre-commit. Tests link against libraries compiled inside that env.
+3. **Set `PARALLEL_LEVEL`** if RAM is tight — default `$(nproc)` can OOM (CUDA compilation needs ~4–8 GB/job).
+4. **Fetch test datasets** before running tests — see `CONTRIBUTING.md` "Building for development" and export `RAPIDS_DATASET_ROOT_DIR`. Missing datasets surface as `MPS_PARSER_ERROR ... Error opening MPS file` at 0ms (not a build/logic failure).
 
 ### Quick Reference
 
@@ -191,27 +119,15 @@ pytest -v python/cuopt_server/tests     # Server tests
 
 For component-specific build commands, run-test detail, and `PARALLEL_LEVEL` configuration, see [references/build_and_test.md](references/build_and_test.md).
 
-#### Download test datasets before running tests
-
-cuOpt tests depend on MPS/data files that are not checked into the repo. A
-missing dataset surfaces as a `MPS_PARSER_ERROR ... Error opening MPS file`
-test failure at 0ms — it is not a build or logic failure.
-
-Before running any C++ or Python tests, follow the dataset download and
-`RAPIDS_DATASET_ROOT_DIR` export steps in the repo's `CONTRIBUTING.md`
-("Building for development" section) — that is the canonical list and mapping.
-
-If a test fails with a missing-file error, run the matching download step from
-`CONTRIBUTING.md` and re-run the test. Do not report missing-dataset failures
-back to the user as the task outcome.
-
 ## Python Bindings
 
 cuOpt uses Cython to bridge Python and C++. See [references/python_bindings.md](references/python_bindings.md) for the full architecture, parameter flow walkthrough, key files, and Cython patterns.
 
 ## Contributing — Commits, PRs, Common Tasks
 
-For pre-commit setup, DCO sign-off (`git commit -s`), the fork-based PR workflow, the draft-PR rule for agents, PR-description rules (keep it short — no "how it works" walkthroughs or file tables), script and CI/workflow authoring principles (extend existing files before adding new ones; no speculative flags, restated defaults, or silent fallbacks), and step-by-step common-task recipes (adding a solver parameter, dependency, server endpoint, or CUDA kernel), see [references/contributing.md](references/contributing.md).
+End-to-end PR workflow: **fork on GitHub → clone your fork → branch off `main` → install pre-commit hooks → commit with DCO sign-off (`git commit -s`) → push to your fork → open a draft PR → keep the PR description short (no "how it works" walkthroughs or file tables)**. CI must pass — never use `--no-verify` or skip checks.
+
+For full detail (pre-commit setup, draft-PR rule for agents, script/CI authoring principles, and common-task recipes — adding a solver parameter, dependency, server endpoint, or CUDA kernel), see [references/contributing.md](references/contributing.md).
 
 ## Coding Conventions
 

skills/cuopt-developer/evals/evals.json

@@ -0,0 +1,13 @@
+[
+  {
+    "id": "dev-eval-001-dependencies-yaml-workflow",
+    "question": "I'm contributing to cuOpt and want to add `xarray` as a new Python runtime dependency. What's the workflow to get it into the build?",
+    "expected_skill": "cuopt-developer",
+    "expected_script": null,
+    "ground_truth": "The agent edits dependencies.yaml under the appropriate dependency group and instructs the user to run `pre-commit run --all-files` to regenerate conda/environments/ and pyproject.toml. The agent does not run `pip install` directly and does not hand-edit pyproject.toml or conda/environments/ files (those are generated artifacts).",
+    "expected_behavior": [
+      "Edits dependencies.yaml as the source of truth and runs `pre-commit run --all-files` to regenerate conda/environments/ and pyproject.toml",
+      "Does not run `pip install` and does not hand-edit pyproject.toml or conda/environments/ files"
+    ]
+  }
+]

skills/cuopt-developer/skill-card.md

@@ -8,7 +8,7 @@ This skill is ready for commercial/non-commercial use. <br>
 ### License/Terms of Use: <br>
 Apache 2.0 <br>
 ## Use Case: <br>
-Developers and engineers contributing to the NVIDIA cuOpt codebase use this skill for building, testing, debugging, and submitting changes to the C++/CUDA solver core, Python bindings, REST server, and CI infrastructure. <br>
+Activate when an NVIDIA contributor is preparing a pull request against `NVIDIA/cuopt`, debugging a failing CI job, or asking how the build/test/sign-off workflow works for this repository specifically. Not for end-user questions about *calling* cuOpt — those route to `cuopt-install`, `cuopt-routing-api-python`, or the numerical-optimization API skills. <br>
 
 ### Deployment Geography for Use: <br>
 Global <br>

skills/cuopt-install/SKILL.md

@@ -86,15 +86,8 @@ conda install -c rapidsai -c conda-forge -c nvidia libcuopt
 
 ### Verify
 
-```bash
-# conda:
-find $CONDA_PREFIX -name "cuopt_c.h"
-find $CONDA_PREFIX -name "libcuopt.so"
-
-# pip (venv):
-find "$(python -c 'import sys; print(sys.prefix)')" -name "cuopt_c.h"
-find "$(python -c 'import sys; print(sys.prefix)')" -name "libcuopt.so"
-```
+See [`references/verification_examples.md`](references/verification_examples.md)
+for the canonical C-API header/library `find` commands (conda and pip/venv variants).
 
 ## Server (REST)
 

skills/cuopt-install/evals/evals.json

@@ -0,0 +1,13 @@
+[
+  {
+    "id": "inst-eval-001-docker-server",
+    "question": "I want to run the cuOpt REST server in a Docker container with GPU access on a CUDA 12 host. What image do I pull and what run command exposes the API on port 8000?",
+    "expected_skill": "cuopt-install",
+    "expected_script": null,
+    "ground_truth": "The agent uses the official NVIDIA cuOpt Docker image tagged for CUDA 12 (e.g. nvidia/cuopt:latest-cuda12.9-py3.13) and provides a docker run command with --gpus all (for GPU access) and -p 8000:8000 (to expose the REST API). The agent does not invent NGC paths like nvcr.io/nvidia/cuopt:latest.",
+    "expected_behavior": [
+      "Uses the nvidia/cuopt Docker image tagged for CUDA 12 (e.g. nvidia/cuopt:latest-cuda12.9-py3.13), not a fabricated nvcr.io/* path",
+      "docker run command includes --gpus all and -p 8000:8000"
+    ]
+  }
+]

skills/cuopt-numerical-optimization-api-c/SKILL.md

@@ -22,28 +22,9 @@ Confirm problem type and formulation (variables, objective, constraints, variabl
 
 This skill is **C only**.
 
-## Quick Reference: C API
-
-```c
-#include <cuopt/linear_programming/cuopt_c.h>
-
-// CSR format for constraints
-cuopt_int_t row_offsets[] = {0, 2, 4};
-cuopt_int_t col_indices[] = {0, 1, 0, 1};
-cuopt_float_t values[] = {2.0, 3.0, 4.0, 2.0};
-char var_types[] = {CUOPT_CONTINUOUS, CUOPT_INTEGER};
-
-cuOptCreateRangedProblem(
-    num_constraints, num_variables, CUOPT_MINIMIZE,
-    0.0, objective_coefficients,
-    row_offsets, col_indices, values,
-    constraint_lower, constraint_upper,
-    var_lower, var_upper, var_types,
-    &problem
-);
-cuOptSolve(problem, settings, &solution);
-cuOptGetObjectiveValue(solution, &obj_value);
-```
+## API Call Sequence
+
+For LP/MILP, the ordered C entry points are: `cuOptCreateRangedProblem` (sense `CUOPT_MINIMIZE` / `CUOPT_MAXIMIZE`, CSR constraint matrix as `row_offsets` / `col_indices` / `values`, `var_types` char array using `CUOPT_CONTINUOUS` / `CUOPT_INTEGER` macros) → `cuOptSolve(problem, settings, &solution)` → `cuOptGetObjectiveValue(solution, &obj_value)` → matching `cuOptDestroy*` calls. Include `<cuopt/linear_programming/cuopt_c.h>`. Full ordered code with build instructions in [references/examples.md](references/examples.md).
 
 ## QP via C API (beta)
 

skills/cuopt-numerical-optimization-api-c/evals/evals.json

@@ -0,0 +1,13 @@
+[
+  {
+    "id": "numopt-c-eval-001-milp-api-call-sequence",
+    "question": "I want to solve a small MILP (some integer variables, linear objective, linear constraints) with the cuOpt C API. List the C functions and structs I need in order — names only, one line each, no full source.",
+    "expected_skill": "cuopt-numerical-optimization-api-c",
+    "expected_script": null,
+    "ground_truth": "The agent produces an ordered list of C API entry points without writing a full source file: include cuopt/linear_programming/cuopt_c.h, then call cuOptCreateRangedProblem with sense CUOPT_MINIMIZE or CUOPT_MAXIMIZE, then cuOptSolve(problem, settings, &solution), then cuOptGetObjectiveValue.",
+    "expected_behavior": [
+      "Lists C API call sequence without writing a complete source file",
+      "Names cuOptCreateRangedProblem, cuOptSolve, cuOptGetObjectiveValue in order"
+    ]
+  }
+]

skills/cuopt-numerical-optimization-api-c/references/examples.md

@@ -46,17 +46,31 @@ int main() {
     // Constraint matrix in CSR format
     cuopt_int_t row_offsets[] = {0, 2, 4};
     cuopt_int_t column_indices[] = {0, 1, 0, 1};
-    cuopt_float_t values[] = {3.0, 4.0, 2.7, 10.1};
+    cuopt_float_t values[] = {
+        3.0,
+        4.0,
+        2.7,
+        10.1
+    };
 
     // Objective coefficients
-    cuopt_float_t objective_coefficients[] = {-0.2, 0.1};
+    cuopt_float_t objective_coefficients[] = {
+        -0.2,
+        0.1
+    };
 
     // Constraint bounds (lower <= Ax <= upper)
-    cuopt_float_t constraint_upper_bounds[] = {5.4, 4.9};
+    cuopt_float_t constraint_upper_bounds[] = {
+        5.4,
+        4.9
+    };
     cuopt_float_t constraint_lower_bounds[] = {-CUOPT_INFINITY, -CUOPT_INFINITY};
 
     // Variable bounds
-    cuopt_float_t var_lower_bounds[] = {0.0, 0.0};
+    cuopt_float_t var_lower_bounds[] = {
+        0.0,
+        0.0
+    };
     cuopt_float_t var_upper_bounds[] = {CUOPT_INFINITY, CUOPT_INFINITY};
 
     // Variable types
@@ -140,12 +154,26 @@ int main() {
 
     cuopt_int_t row_offsets[] = {0, 2, 4};
     cuopt_int_t column_indices[] = {0, 1, 0, 1};
-    cuopt_float_t values[] = {3.0, 4.0, 2.7, 10.1};
-
-    cuopt_float_t objective_coefficients[] = {-0.2, 0.1};
-    cuopt_float_t constraint_upper[] = {5.4, 4.9};
+    cuopt_float_t values[] = {
+        3.0,
+        4.0,
+        2.7,
+        10.1
+    };
+
+    cuopt_float_t objective_coefficients[] = {
+        -0.2,
+        0.1
+    };
+    cuopt_float_t constraint_upper[] = {
+        5.4,
+        4.9
+    };
     cuopt_float_t constraint_lower[] = {-CUOPT_INFINITY, -CUOPT_INFINITY};
-    cuopt_float_t var_lower[] = {0.0, 0.0};
+    cuopt_float_t var_lower[] = {
+        0.0,
+        0.0
+    };
     cuopt_float_t var_upper[] = {CUOPT_INFINITY, CUOPT_INFINITY};
 
     // x1 = INTEGER, x2 = CONTINUOUS
@@ -202,17 +230,9 @@ cleanup:
 
 ## Build & Run
 
-```bash
-# Set paths (conda example)
-export INCLUDE_PATH="${CONDA_PREFIX}/include"
-export LIB_PATH="${CONDA_PREFIX}/lib"
-
-# Compile
-gcc -I${INCLUDE_PATH} -L${LIB_PATH} -o lp_example lp_example.c -lcuopt
-
-# Run
-LD_LIBRARY_PATH=${LIB_PATH}:$LD_LIBRARY_PATH ./lp_example
-```
+See [`assets/README.md`](../assets/README.md) for the canonical conda-env
+include/library/`LD_LIBRARY_PATH` setup, plus a `gcc` build command. The
+same recipe applies here — substitute `lp_example.c` for the file name.
 
 ## Constants Reference
 

skills/cuopt-server-common/evals/evals.json

@@ -0,0 +1,14 @@
+[
+  {
+    "id": "srv-common-eval-001-request-flow",
+    "question": "How does the cuOpt REST server handle an optimization request, conceptually? Walk me through the flow from submission to solution.",
+    "expected_skill": "cuopt-server-common",
+    "expected_script": null,
+    "ground_truth": "The agent describes the asynchronous request-id + polling pattern: the client POSTs the problem payload (matrices, tasks, fleet, solver config) to the submit endpoint, and the server returns a reqId rather than blocking. The client polls the solution endpoint with that reqId until the solve finishes. The server supports routing, LP, and MILP — QP is not exposed over REST.",
+    "expected_behavior": [
+      "Describes the asynchronous flow: POST returns reqId, then poll the solution endpoint until ready",
+      "States that supported problem types are routing, LP, MILP — not QP",
+      "Does not fabricate endpoint URLs or claim QP is supported over REST"
+    ]
+  }
+]