Skip to content

Add evidence-driven Value Map evaluation - #51

Merged
bigboateng merged 7 commits into
mainfrom
codex/value-map-evidence
Jul 18, 2026
Merged

Add evidence-driven Value Map evaluation#51
bigboateng merged 7 commits into
mainfrom
codex/value-map-evidence

Conversation

@bigboateng

Copy link
Copy Markdown
Contributor

Summary

Adds an evidence-driven evaluation layer for Value Map without changing the canonical public skill.

  • introduces a bounded evidence controller and explicit information/control-theory metrics
  • adds deterministic dev and holdout evaluations with promotion gates
  • adds a three-call Gemini Flash inverse evaluation over two controlled micro-repositories
  • adds a separate consumer-skill smoke test that invokes the canonical skill through Codex
  • narrows the publisher workflow to canonical skill changes so research-only changes cannot trigger a release

Why

Value Map needs to preserve one external claim while allowing repository truth to change the grounded conclusion. Comparing generated prose or asking another model to judge it would not verify that mapping boundary.

This PR tests a controlled inverse relation instead: the exact same message is projected against one repository that only implements isEven and another that implements both isEven and isOdd. Deterministic gates require the first result to retain a testable gap and the second to surface a blocking contradiction, with valid path:line evidence throughout.

The ZCA slice is one source claim, two explicit code truths, and one mapping operator. The value emerges when source meaning stays stable while the code-grounded conclusion changes correctly.

Implementation

  • Adds relation distortion, evidence coverage/resolution, contradiction recall, unsupported-claim count, and bounded inspection cost.
  • Adds one-probe adaptive evidence control focused on unresolved value relations.
  • Makes structured tool use explicit across Anthropic and Gemini providers.
  • Canonicalizes harmless ordinal relation aliases while rejecting ambiguous lineage.
  • Extracts reusable live source and repository projection functions shared by the CLI and evaluator.
  • Adds generated fixtures with capability declarations, implementations, and tests.
  • Adds negative controls for swapped repositories, removed contradictions, and swapped consumer responses.
  • Prints live results to stdout only; model responses are not committed or cached.

Verification

  • pnpm typecheck
  • pnpm build
  • pnpm lab:value-map
  • dev candidate evaluation: 8/8 expected verdicts, all promotion gates pass
  • holdout candidate evaluation: 8/8 expected verdicts, all promotion gates pass
  • live inverse evaluation with gemini-2.5-flash: all 11 gates and both negative controls pass using exactly 3 API calls (2,253 input / 2,743 output tokens)
  • canonical consumer-skill smoke test with gpt-5.6-sol: both controlled repositories produce the expected grounded distinction; all 9 gates and the swapped-response negative control pass (2 sessions, 19,645 host tokens)
  • both generated fixture assertions pass
  • git diff --check

Limits

These are controlled smoke tests, not population-level accuracy or product-value claims. The fixtures are synthetic by design, and the consumer smoke test checks stable semantic invariants rather than byte-for-byte prose.

Publishing impact

The canonical value-map/SKILL.md is unchanged. The workflow path filter now ensures candidate research, fixtures, and benchmark changes do not publish a public Value Map release.

@bigboateng
bigboateng marked this pull request as ready for review July 18, 2026 20:14
@bigboateng
bigboateng merged commit ea59dab into main Jul 18, 2026
1 check passed
@bigboateng
bigboateng deleted the codex/value-map-evidence branch July 18, 2026 20:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant