Skip to content

Fix Harbor reward details output - #75

Merged
adithya-s-k merged 1 commit into
huggingface:mainfrom
neubig:fix/harbor-reward-details
Jul 13, 2026
Merged

Fix Harbor reward details output#75
adithya-s-k merged 1 commit into
huggingface:mainfrom
neubig:fix/harbor-reward-details

Conversation

@neubig

@neubig neubig commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Keep reward.txt as the Harbor-readable scalar result.
  • Write pr_runtime diagnostics to Harbor's reward-details.json sidecar.

Test plan

  • uv run pytest -q tests/test_pr_runtime_verifier.py tests/test_pipeline_pr_runtime.py
  • uv run ruff check src/repo2rlenv/pipelines/_pr_runtime_verifier.py src/repo2rlenv/pipelines/pr_runtime.py tests/test_pr_runtime_verifier.py tests/test_pipeline_pr_runtime.py
  • uv run ruff format --check src/repo2rlenv/pipelines/_pr_runtime_verifier.py src/repo2rlenv/pipelines/pr_runtime.py tests/test_pr_runtime_verifier.py tests/test_pipeline_pr_runtime.py

Out of scope

  • Regenerating published Hub dataset revisions.

Closes #74

Live evidence

Run on 2026-07-13 UTC with the real standalone production _pr_runtime_verifier.py CLI in each checkout. Both runs used the identical parsed pytest log and F2P/P2P oracle files:

python src/repo2rlenv/pipelines/_pr_runtime_verifier.py \
  --log pytest.log \
  --f2p f2p.json \
  --p2p p2p.json \
  --runner pytest \
  --exit-code 0 \
  --out-dir "$OUT_DIR"

The input contained one passing FAIL_TO_PASS test and one passing PASS_TO_PASS test, so the production verifier computed a scalar reward of 1.0 and resolved=true.

Current main: Harbor-preferred file has an invalid schema

Commit: a8cf74d71a0f6de6bbc9adfb43a00249108610c1

The CLI wrote both reward.txt and reward.json. Although reward.txt contained the valid scalar 1.000000, Harbor gives reward.json precedence. The emitted JSON was neither a number nor a flat numeric map: it contained booleans (resolved, command_resolved), arrays (regressions, untracked_failed), and strings (parse_status, runner). Applying Harbor's numeric-or-flat-numeric-map contract produced:

{
  "harbor_schema_valid": false,
  "resolved": true,
  "command_resolved": true,
  "parse_status": "ok"
}

This demonstrates the issue directly: a correctly resolved verifier run still presents Harbor with a higher-precedence structured reward file that violates its reward schema.

PR head: Harbor consumes the scalar and details remain available

Commit: 056632c92d4d4d01a33ab2c337b21f8024297b94

The identical CLI invocation wrote reward.txt and reward-details.json, with no reward.json present:

files=reward.txt reward-details.json
reward.txt=1.000000
reward.json present=false
harbor_scalar_valid=true

The sidecar retained the structured evaluation data without competing for Harbor reward precedence:

{
  "reward": 1.0,
  "resolved": true,
  "command_resolved": true,
  "parse_status": "ok",
  "runner": "pytest"
}

This validates the changed scalar-plus-sidecar output contract through the actual standalone verifier process. Regenerating published Hub dataset revisions remains explicitly out of scope for this code-only PR.

@neubig
neubig marked this pull request as ready for review July 10, 2026 21:32
@adithya-s-k
adithya-s-k merged commit e2af476 into huggingface:main Jul 13, 2026
adithya-s-k added a commit that referenced this pull request Jul 13, 2026
…ar (#76)

PR #75 (Neubig) fixed the Harbor reward-output spec violation on the shared
_pr_runtime_verifier.py — pr_runtime / commit_runtime / cve_patches all
inherited it. But `_pr_diff_verifier.py` writes its own reward.json with
nested `components` / `weights` dicts + `judge_model` string + `judge_status`
string, all of which violate Harbor's "flat map of floats/ints" schema
(<https://harborframework.com/docs/tasks>). Ports the same split:

- reward.txt              — scalar (Harbor reads this)
- reward-details.json     — full 6-component breakdown + judge metadata

Also updates the doc/card refs across hub.py + pr_diff.py + the e2e test to
point at reward-details.json (previously named /logs/verifier/reward.json in
dataset-card prose that gets rendered into the published READMEs).

Version bump 0.8.5 → 0.8.5.post1 covers the shipped fix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pr_runtime: keep structured details out of reward.json

2 participants