Allow evaluating an engine branch and only modified entries - #849
Allow evaluating an engine branch and only modified entries#849Wenjie Fan (gggdttt) wants to merge 1 commit into
Conversation
Add optional engine-ref and modified-only inputs to pr-review-evaluation so an unmerged BCQuality article can be scored via a BC-ALAgents branch, without touching BC-Bench's BCQuality isolation. Both default to today's behaviour. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7a9dff24-8942-4810-8359-18904eaa4da4
|
Closing this in favour of the repository's established practice. The BC-ALAgents revision under evaluation is pinned in This pull request instead made the pin overridable at dispatch time (
|
Why
Today
pr-review-evaluation.ymlcan only score the engine commit hard-coded ininstall-agent-harnesses, over the wholecodereview.jsonldataset.That makes it impossible to answer the question the self-improvement loop keeps
asking: does this not-yet-merged BCQuality article actually suppress the false
positive we wrote a gold answer for? The article only takes effect once
bcquality.config.yamlpoints at it, and that file lives in BC-ALAgents - soyou need to evaluate an engine branch, not the pinned baseline.
BC-Bench deliberately isolates BCQuality (
_environment_without_bcquality_overridesin
agent/pr_review/agent.pystrips everyBCQUALITY_*variable, andPrepare-BCQualityRoot.ps1reads the ref only from the engine's own config).That isolation is correct and is left untouched here. Instead this PR makes the
engine selectable, so the normal engine -> BCQuality chain does the rest:
bcquality.config.yamlrefpoints at that PR head,engine-refset to that branch.What
Two optional
workflow_dispatchinputs, both defaulting to today's behaviour:engine-ref- BC-ALAgents ref to evaluate. Blank keeps the pinnedbaseline commit. Plumbed through a new
bc-alagents-refinput on theinstall-agent-harnessescomposite action, which falls back to the samepinned SHA when the value is empty (a composite action's
defaultdoes notapply to an explicitly-passed empty string, hence the
||fallback).It is also forwarded through
requeue-evaluationso repeated runs stay on thesame engine.
modified-only- forwarded toget-entries.yml, which already supportsit but was never given the value. Scores only the dataset entries changed
versus
origin/main, which is what you want when validating a handful of newgold answers instead of burning credits on all 251.
Because a
modified-onlyrun is a biased subset, it is excluded from anythingthat would treat it as a baseline:
skip-leaderboardis set,requeueisskipped, and it gets its own concurrency group so it does not queue behind a
full run.
Validation
uv run ruff format/ruff checkcleanuv run pytest -q- 871 passed, 2 skippedtest_review_workflows.pyupdated: the harness test now asserts the baselineSHA is still pinned and that the override input exists.
No behaviour change when both inputs are left blank, which is every scheduled
and requeued run today.