Skip to content

feat(gitchameleon): ready the funded evidence run - #27

Merged
tcballard merged 2 commits into
mainfrom
codex/gitchameleon-funded-run
Aug 2, 2026
Merged

feat(gitchameleon): ready the funded evidence run#27
tcballard merged 2 commits into
mainfrom
codex/gitchameleon-funded-run

Conversation

@tcballard

@tcballard tcballard commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Outcome

Makes the GitChameleon evidence run operationally ready without claiming that the funded run has happened.

  • pins voyage-4-large as the concrete naive_rag baseline over the same per-example Markdown corpus as As Decided
  • names the product-backed experimental arm asdecided throughout CLI arguments, records, schemas, fixtures, generated filenames, and documentation; the retired rac label is deliberately rejected
  • pins dataset commit 799a6a33e572a07a8985914e7251f5dea54b0ac4 and upstream scorer commit 3a1b6045a6b2a276bd24d715589cb041f8eccb93
  • fetches the immutable raw dataset JSONL at the selected commit, rather than a mutable viewer projection
  • adds bounded provider retries, one-example shakedowns, explicit overwrite protection, and provenance-checked resume
  • records hashes of the exact prompt and grounding used for every completion
  • refuses incomplete or duplicate paired resolution records before statistics
  • records the execution contract as accepted GCB-ADR-0003 and freezes the first-run inputs in run-config.json

Evidence

  • 71 repository tests passed on the funded-run implementation
  • 20 focused GitChameleon tests pass after the arm rename
  • Ruff check passed for all changed Python files (EXE001 ignored because repository Python entrypoints are intentionally mode 0644)
  • all 3 GitChameleon decisions validate under As Decided v0.26.2
  • live dataset fetch reproduced 328 raw rows at the pinned revision
  • generated 328 per-example corpora / 1,968 decision artifacts; 1,968 valid, 0 invalid, OKF conformant
  • generated 656 no_grounding + asdecided bundles; As Decided ranked the governing version pin first in 328/328 product-arm cases
  • mechanical contribution preflight passed with no blockers

Explicit boundary

No benchmark result is claimed here. ANTHROPIC_API_KEY and VOYAGE_API_KEY are not present on this machine, and the locally built upstream Docker scorer still requires a functioning Docker daemon. The paid 984-answer run and upstream executable scoring remain the completion gate for #10.

Progresses #10.

Authored under my direction with Codex.

Pin the Voyage baseline and upstream scorer, add immutable dataset fetches, safe resume semantics, provenance hashes, completeness gates, and the accepted execution decision.

Authored under my direction with Codex.
@tcballard
tcballard merged commit 2c8df7a into main Aug 2, 2026
3 checks passed
@tcballard
tcballard deleted the codex/gitchameleon-funded-run branch August 2, 2026 07:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant