Skip to content

chameleon 2.13.2

Choose a tag to compare

@github-actions github-actions released this 12 Jun 04:22

The effectiveness eval harness (arc 2): a repeatable A/B measurement of whether chameleon improves agent output, plus one runtime fix it surfaced. First measurement run (24 cells, $8.58, zero errors) is seeded as the tier-ci baseline.

Added

  • Effectiveness eval harness (tests/effectiveness/, local-only, never CI). Real claude -p sessions run identical task prompts under matched arms (off / shadow / enforce, plus --toggle <enforcement key> paired arms for feature-level experiments), in per-cell git worktrees with bootstrap-once profile cloning. Deterministic-first scoring (convention lint, crossfile callers-updated from the calls index, duplication body-hash, verification from the exec log) under a strict metrics-or-unscored contract; a blind pairwise judge panel for the subjective remainder; run.json / run.md scoreboards with committed baselines and a direction-aware regression banner. Two committed convention-rich fixtures (TS + Rails) carry 8 ci tasks; tier-full task packs target the env-pointed real repos. See tests/effectiveness/README.md.

Fixed

  • ruby -Itest was never classified as a test command. The exec-log classifier's \b(?:-Itest|minitest)\b had an unreachable first branch (no word boundary between whitespace and -), so the standard minitest invocation never recorded test_command_seen and the Stop-gate test nudge fired even after a passing suite. The pattern now uses a lookbehind, with negative cases pinned.