Skip to content

Allow evaluating an engine branch and only modified entries - #849

Closed
Wenjie Fan (gggdttt) wants to merge 1 commit into
mainfrom
bench/pr-dataset-eval-gate
Closed

Allow evaluating an engine branch and only modified entries#849
Wenjie Fan (gggdttt) wants to merge 1 commit into
mainfrom
bench/pr-dataset-eval-gate

Conversation

@gggdttt

Copy link
Copy Markdown
Collaborator

Why

Today pr-review-evaluation.yml can only score the engine commit hard-coded in
install-agent-harnesses, over the whole codereview.jsonl dataset.

That makes it impossible to answer the question the self-improvement loop keeps
asking: does this not-yet-merged BCQuality article actually suppress the false
positive we wrote a gold answer for?
The article only takes effect once
bcquality.config.yaml points at it, and that file lives in BC-ALAgents - so
you need to evaluate an engine branch, not the pinned baseline.

BC-Bench deliberately isolates BCQuality (_environment_without_bcquality_overrides
in agent/pr_review/agent.py strips every BCQUALITY_* variable, and
Prepare-BCQualityRoot.ps1 reads the ref only from the engine's own config).
That isolation is correct and is left untouched here. Instead this PR makes the
engine selectable, so the normal engine -> BCQuality chain does the rest:

  1. open the BCQuality PR with the new article,
  2. push a BC-ALAgents branch whose bcquality.config.yaml ref points at that PR head,
  3. run this workflow with engine-ref set to that branch.

What

Two optional workflow_dispatch inputs, both defaulting to today's behaviour:

  • engine-ref - BC-ALAgents ref to evaluate. Blank keeps the pinned
    baseline commit. Plumbed through a new bc-alagents-ref input on the
    install-agent-harnesses composite action, which falls back to the same
    pinned SHA when the value is empty (a composite action's default does not
    apply to an explicitly-passed empty string, hence the || fallback).
    It is also forwarded through requeue-evaluation so repeated runs stay on the
    same engine.

  • modified-only - forwarded to get-entries.yml, which already supports
    it but was never given the value. Scores only the dataset entries changed
    versus origin/main, which is what you want when validating a handful of new
    gold answers instead of burning credits on all 251.

Because a modified-only run is a biased subset, it is excluded from anything
that would treat it as a baseline: skip-leaderboard is set, requeue is
skipped, and it gets its own concurrency group so it does not queue behind a
full run.

Validation

  • uv run ruff format / ruff check clean
  • uv run pytest -q - 871 passed, 2 skipped
  • test_review_workflows.py updated: the harness test now asserts the baseline
    SHA is still pinned and that the override input exists.
  • YAML of both changed files parses.

No behaviour change when both inputs are left blank, which is every scheduled
and requeued run today.

Add optional engine-ref and modified-only inputs to pr-review-evaluation so an unmerged BCQuality article can be scored via a BC-ALAgents branch, without touching BC-Bench's BCQuality isolation. Both default to today's behaviour.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 7a9dff24-8942-4810-8359-18904eaa4da4
@gggdttt
Wenjie Fan (gggdttt) marked this pull request as draft September 3, 2026 11:24
@gggdttt
Wenjie Fan (gggdttt) marked this pull request as ready for review September 3, 2026 13:44
@gggdttt

Copy link
Copy Markdown
Collaborator Author

Closing this in favour of the repository's established practice.

The BC-ALAgents revision under evaluation is pinned in .github/actions/install-agent-harnesses/action.yml, and changing it is done by editing that pin in a pull request — see #843 (Refresh PR Review baseline and model leaderboard), and #789, which added a dedicated skill for the harness version bump. Evaluating an unmerged BCQuality change is therefore already possible without a runtime override: point an engine branch's bcquality.config.yaml at it and move the pin, or run the evaluation locally.

This pull request instead made the pin overridable at dispatch time (ref: ${{ inputs.bc-alagents-ref || '<sha>' }}). That works against #851, which is currently exporting bc-alagents-commit, bcquality-repository, bcquality-commit and bcquality-version as harness outputs so every evaluation result carries the exact identity of what produced it. A runtime override lets the recorded pin and the engine that actually ran diverge, and this pull request only propagated an engine-ref string rather than the full identity set. The two changes also touch the same three files, so they conflict textually as well as in design.

modified-only is the one independently useful piece here, and its consumer was microsoft/BC-ALAgentsInternal#25, which is being closed alongside this. If it is wanted for manual evaluation runs it can be reproposed on its own after #851 merges, without touching the pin.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant