feat(scicode): per-rollout subtask_accuracy + across-run std of both headline metrics

[laszkiewiczp]

What

Two related fixes to SciCode's aggregate metrics:

1. subtask_accuracy gets real statistics. Today it is only a pooled scalar injected at aggregate time by the agent's compute_metrics — it is never a per-rollout field, so the generic RewardProfiler computes no statistics for it at all (no mean/, std/, min/, max/), unlike reward/problem_accuracy. ScicodeVerifyResponse now carries a per-rollout subtask_accuracy (that rollout's sub-step pass fraction), so the profiler describes it like any other numeric field.

2. Across-run (repeat-to-repeat) std for both headline metrics. With num_repeats > 1 (the benchmark config default is 3), the agent's compute_metrics now emits:

   • mean/problem_accuracy/std_dev_across_runs — sample std-dev (ddof=1) over per-run means of problem_accuracy
   • subtask_accuracy/std_dev_across_runs — sample std-dev over per-run sub-step-weighted pools, i.e. the exact micro-average definition of the headline subtask_accuracy, computed per run

   Repeats are aligned by _ng_rollout_index; the run count is the minimum repeat count across problems (rectangular under partial outputs); single-repeat collections emit nothing extra.

Why

Benchmark dashboards want an explicit across-run variability artifact next to each declared main metric (same convention as compute_pass_majority_metrics' pass@1[avg-of-k]/*/std_dev_across_runs). SciCode's declared metrics are mean/problem_accuracy and subtask_accuracy; neither had an across-run statistic, and subtask_accuracy had no statistics of any kind — a symptom of the pooled-scalar-only definition this PR fixes.

Compatibility

• The pooled subtask_accuracy value and its definition are unchanged (still sub-step-weighted micro-average, matching nemo-skills parity noted in the benchmark README).
• get_key_metrics moves from an exact-key special case to a prefix match so the new subtask_accuracy/* stats surface as headline metrics; mean/* behavior unchanged.
• Note: the new per-rollout field also yields mean/subtask_accuracy (a macro-average over rollouts), which is intentionally distinct from the pooled micro-average subtask_accuracy.

Tests

• resources_servers/scicode/tests/test_app.py: verify-path asserts for the new field (all-pass → 1.0, all-fail → 0.0, out-of-context partial → 0.5).
• responses_api_agents/scicode_agent/tests/test_app.py: 6 new tests — hand-computed 2-problems×3-repeats case for both metrics, single-repeat no-op, rollout-index alignment vs arrival order, min-k rectangularization, zero-variance case, and key-metrics surfacing.
• Both files: 37 passed.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[laszkiewiczp]

/ok to test cf9a693