Fix Harbor reward details output - #75
Merged
Merged
Conversation
neubig
marked this pull request as ready for review
July 10, 2026 21:32
adithya-s-k
added a commit
that referenced
this pull request
Jul 13, 2026
…ar (#76) PR #75 (Neubig) fixed the Harbor reward-output spec violation on the shared _pr_runtime_verifier.py — pr_runtime / commit_runtime / cve_patches all inherited it. But `_pr_diff_verifier.py` writes its own reward.json with nested `components` / `weights` dicts + `judge_model` string + `judge_status` string, all of which violate Harbor's "flat map of floats/ints" schema (<https://harborframework.com/docs/tasks>). Ports the same split: - reward.txt — scalar (Harbor reads this) - reward-details.json — full 6-component breakdown + judge metadata Also updates the doc/card refs across hub.py + pr_diff.py + the e2e test to point at reward-details.json (previously named /logs/verifier/reward.json in dataset-card prose that gets rendered into the published READMEs). Version bump 0.8.5 → 0.8.5.post1 covers the shipped fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
reward.txtas the Harbor-readable scalar result.pr_runtimediagnostics to Harbor'sreward-details.jsonsidecar.Test plan
uv run pytest -q tests/test_pr_runtime_verifier.py tests/test_pipeline_pr_runtime.pyuv run ruff check src/repo2rlenv/pipelines/_pr_runtime_verifier.py src/repo2rlenv/pipelines/pr_runtime.py tests/test_pr_runtime_verifier.py tests/test_pipeline_pr_runtime.pyuv run ruff format --check src/repo2rlenv/pipelines/_pr_runtime_verifier.py src/repo2rlenv/pipelines/pr_runtime.py tests/test_pr_runtime_verifier.py tests/test_pipeline_pr_runtime.pyOut of scope
Closes #74
Live evidence
Run on 2026-07-13 UTC with the real standalone production
_pr_runtime_verifier.pyCLI in each checkout. Both runs used the identical parsed pytest log and F2P/P2P oracle files:python src/repo2rlenv/pipelines/_pr_runtime_verifier.py \ --log pytest.log \ --f2p f2p.json \ --p2p p2p.json \ --runner pytest \ --exit-code 0 \ --out-dir "$OUT_DIR"The input contained one passing FAIL_TO_PASS test and one passing PASS_TO_PASS test, so the production verifier computed a scalar reward of 1.0 and
resolved=true.Current
main: Harbor-preferred file has an invalid schemaCommit:
a8cf74d71a0f6de6bbc9adfb43a00249108610c1The CLI wrote both
reward.txtandreward.json. Althoughreward.txtcontained the valid scalar1.000000, Harbor givesreward.jsonprecedence. The emitted JSON was neither a number nor a flat numeric map: it contained booleans (resolved,command_resolved), arrays (regressions,untracked_failed), and strings (parse_status,runner). Applying Harbor's numeric-or-flat-numeric-map contract produced:{ "harbor_schema_valid": false, "resolved": true, "command_resolved": true, "parse_status": "ok" }This demonstrates the issue directly: a correctly resolved verifier run still presents Harbor with a higher-precedence structured reward file that violates its reward schema.
PR head: Harbor consumes the scalar and details remain available
Commit:
056632c92d4d4d01a33ab2c337b21f8024297b94The identical CLI invocation wrote
reward.txtandreward-details.json, with noreward.jsonpresent:The sidecar retained the structured evaluation data without competing for Harbor reward precedence:
{ "reward": 1.0, "resolved": true, "command_resolved": true, "parse_status": "ok", "runner": "pytest" }This validates the changed scalar-plus-sidecar output contract through the actual standalone verifier process. Regenerating published Hub dataset revisions remains explicitly out of scope for this code-only PR.