Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 94 additions & 0 deletions .github/workflows/opencode-review-corpus-quality-ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
name: OpenCode Review Corpus Quality CI

on:
pull_request:
branches:
- main
- feat/opencode-review-quality-gate
paths:
- ".github/workflows/opencode-review-corpus-quality-ci.yml"
- "scripts/ci/opencode_review_sample.py"
- "scripts/ci/opencode_review_adjudicate.py"
- "scripts/ci/opencode_review_adjudication_primitives.py"
- "scripts/ci/opencode_review_adjudication_annotations.py"
- "scripts/ci/opencode_review_adjudication_decisions.py"
- "tests/test_opencode_review_sample.py"
- "tests/test_opencode_review_sample_coverage.py"
- "tests/test_opencode_review_adjudicate.py"
- "tests/test_opencode_review_adjudicate_coverage.py"
- "docs/doctoring/opencode-review-annotation-guide.md"
- "benchmarks/opencode_review/head_matched_v1/**"
- "CHANGELOG.md"

permissions:
contents: read

concurrency:
group: opencode-review-corpus-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

jobs:
gold-corpus-quality:
name: gold-corpus-quality
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
- name: Checkout exact source revision
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
persist-credentials: false

- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"

- name: Install exact hash-verified test runner dependencies
env:
PIP_DISABLE_PIP_VERSION_CHECK: "1"
PIP_NO_INPUT: "1"
shell: bash --noprofile --norc -e -o pipefail {0}
run: |
cat >"${RUNNER_TEMP}/opencode-review-corpus-requirements.txt" <<'REQEOF'
coverage==7.15.2 --hash=sha256:b9a6367e4aff723e8ee8190836836124284e8fcd4265e307c844010cfa074f3f
iniconfig==2.1.0 --hash=sha256:9deba5723312380e77435581c6bf4935c94cbfab9b1ed33ef8d238ea168eb760
packaging==26.2 --hash=sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e
pluggy==1.6.0 --hash=sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746
pygments==2.20.0 --hash=sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176
pytest==9.1.1 --hash=sha256:37a86b45efb9a47a61a36449063e8e18d0cab3161329fc099eb21783169c4f0c
REQEOF
python -m pip install \
--only-binary=:all: \
--require-hashes \
-r "${RUNNER_TEMP}/opencode-review-corpus-requirements.txt"

- name: Verify head-matched corpus contracts
shell: bash --noprofile --norc -e -o pipefail {0}
run: |
test "$(git rev-parse HEAD)" = "${{ github.event.pull_request.head.sha || github.sha }}"
python -m coverage run \
--branch \
--source=scripts/ci \
-m pytest \
tests/test_opencode_review_sample.py \
tests/test_opencode_review_sample_coverage.py \
tests/test_opencode_review_adjudicate.py \
tests/test_opencode_review_adjudicate_coverage.py \
-q
python -m coverage report \
--include='scripts/ci/opencode_review_sample.py,scripts/ci/opencode_review_adjudicate.py,scripts/ci/opencode_review_adjudication_primitives.py,scripts/ci/opencode_review_adjudication_annotations.py,scripts/ci/opencode_review_adjudication_decisions.py' \
--fail-under=100 \
--show-missing
python -m compileall -q \
scripts/ci/opencode_review_sample.py \
scripts/ci/opencode_review_adjudicate.py \
scripts/ci/opencode_review_adjudication_primitives.py \
scripts/ci/opencode_review_adjudication_annotations.py \
scripts/ci/opencode_review_adjudication_decisions.py \
tests/test_opencode_review_sample.py \
tests/test_opencode_review_sample_coverage.py \
tests/test_opencode_review_adjudicate.py \
tests/test_opencode_review_adjudicate_coverage.py
git diff --exit-code
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ Semantic Versioning where the repository publishes a release.

### Added

- Added deterministic exact-head corpus sampling and blinded two-expert-plus-adjudicator gold-freeze tooling, with strict JSON, immutable evidence receipts, hard language/size/risk/defect coverage, atomic outputs, stable failure classes, and permanent 100% production statement/branch/docstring evidence.
- Added an empirical OpenCode review-quality benchmark, fail-closed scorer, exact-head quality workflow, and APA 7th doctoring that keep lifecycle-yield evidence separate from head-matched expert-gold precision and recall, require Wilson-bound non-inferiority before any CodeRabbit-parity claim, and preserve 100% production statement/branch/docstring evidence.
- Added a trusted pull-request comment router for `@cwl-noema-review` and review-only `@opencode-agent` dispatches, with an organization sweep, exact-head receipts, repository allowlisting, fixed runners, immutable checkout pins, and a permanent 100% statement/branch/docstring quality gate.
- Added exact-base `uv.lock` materialization that reconstructs standalone nested projects with a checksum-pinned official `uv` exporter, isolated frozen/offline execution, strict exact-pin and SHA-256 output validation, and complete Python 3.10/3.14 quality evidence.
Expand Down
95 changes: 95 additions & 0 deletions benchmarks/opencode_review/head_matched_v1/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# Head-matched OpenCode review corpus v1

This directory is the versioned landing area for real, exact-head code-review
quality evidence. It intentionally contains no fabricated expert annotations or
synthetic parity result.

## Required lifecycle

```text
trusted exact-head inventory
→ deterministic policy-complete sample
→ two independent blinded expert annotations
→ third-party blinded adjudication
→ immutable gold freeze
→ same-head OpenCode and CodeRabbit capture
→ scorer input assembly
→ frozen benchmark report
```

Gold must be frozen before either automated reviewer output is disclosed to the
experts or adjudicator. Lifecycle pilot data under `../pilot_baseline_v1.json`
remains operational telemetry and must not be copied into this expert-gold
answer key.

## Expected layout

```text
head_matched_v1/
├── README.md
├── inventory.json
├── sample.json
├── partition_manifest.json
└── cases/
└── case_001/
├── context_manifest.json
├── expert_a.json
├── expert_b.json
├── adjudication.json
├── gold.json
├── opencode_review.json
└── coderabbit_review.json
```

The repository may store large or access-restricted context bundles outside Git
when required. `context_manifest.json` must then retain immutable object
identities, content digests, access classification, and reproduction
instructions without embedding credentials.

## Tooling

Select an eligible sample:

```bash
python scripts/ci/opencode_review_sample.py \
--input benchmarks/opencode_review/head_matched_v1/inventory.json \
--output benchmarks/opencode_review/head_matched_v1/sample.json \
--seed opencode-review-head-matched-v1
```

Freeze one adjudicated case:

```bash
python scripts/ci/opencode_review_adjudicate.py \
--expert-a benchmarks/opencode_review/head_matched_v1/cases/case_001/expert_a.json \
--expert-b benchmarks/opencode_review/head_matched_v1/cases/case_001/expert_b.json \
--adjudication benchmarks/opencode_review/head_matched_v1/cases/case_001/adjudication.json \
--output benchmarks/opencode_review/head_matched_v1/cases/case_001/gold.json
```

Both tools are offline, strict-JSON, deterministic utilities. They do not call
GitHub, execute repository code, inspect model credentials, or create human
judgments.

## Admission rules

- Same repository, pull request, base SHA, head SHA, diff digest, and context
digest across every record.
- Complete repository context available to both experts.
- Same-head OpenCode and CodeRabbit review possible.
- Two distinct independent expert annotators and a distinct adjudicator.
- Required language, diff-size, risk, and defect-class strata represented.
- Every expert finding adjudicated exactly once.
- Every accepted gold finding carries path, positive line, class, severity,
trigger, impact, root cause, fix direction, and regression target.
- No secrets, cookies, tokens, or unrelated personal data.
- No related-change family leakage across development, calibration, and held-out
partitions.

## Current status

`EMPTY_PENDING_REAL_COLLECTION`

The absence of cases is not a passing parity result. OpenCode Review remains
`INSUFFICIENT_EVIDENCE` until a frozen corpus meets the configured minimum case
and gold-finding floors and the separately versioned statistical gate passes.
Loading
Loading