Skip to content

[codex] qualify the evidence compiler against primitives - #2109

Merged
TheGreenCedar merged 1 commit into
dev/codestory-nextfrom
codex/2108-v018-builder-ablation
Sep 2, 2026
Merged

[codex] qualify the evidence compiler against primitives#2109
TheGreenCedar merged 1 commit into
dev/codestory-nextfrom
codex/2108-v018-builder-ablation

Conversation

@TheGreenCedar

@TheGreenCedar TheGreenCedar commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Context

PR #2107 landed the repository-derived packet compiler, but the packet remains a product hypothesis. This slice adds the frozen builder-visible experiment that must compare it with native tools and CodeStory's exact primitives before any 0.18 usefulness claim.

The historical language-expansion tasks are contaminated development inputs. They cannot authorize a release.

What changed

  • Adds five fixed arms: native tools, exact identity/source, exact plus relations, packet without dense candidates, and packet with dense candidates.
  • Adds a benchmark-only dense-stage switch. Product builds neither expose nor accept it.
  • Records actual descriptor-stage execution and binds optional continuations to the exact offered parent, option set, request, and publication.
  • Holds typed continuations dense-disabled in both packet treatments, so only the withheld initial packet call differs between the semantic arms.
  • Runs fresh rows in balanced order with opaque treatment-blind ephemeral Codex homes and one checksum-bound CLI.
  • Allows adaptive local source inspection while rejecting wrappers, compound commands, MCP, cross-arm CodeStory operations, remote context, reuse, task hints, and hidden answer-shape input.
  • Allowlists exact-tool arguments, binds the pinned project and prepared search namespace, and rejects cache, refresh, foreign-project, and output-path overrides.
  • Keeps task, arm, repeat, and measurement artifact identities out of the model prompt.
  • Partitions one observed local repository-context action sequence at the matched first successful CodeStory action. Unknown readers count when they expose output, failing exits cannot erase shown context, and parallel exploration cannot fall between first-action and post-CodeStory accounting.
  • Blinds and independently adjudicates all 96 CodeStory-arm answers for critical factual and unsupported-relation claims.
  • Emits the packet advance / revise_once / stop decision and enforced keep-or-disable decisions for graph and dense-semantic defaults.

Review focus

  • Can any agent command escape the direct checksum-bound CLI, prepared state, or pinned repository, or combine CodeStory with source inspection?
  • Can task IDs, arm names, repeats, scorer fields, treatment labels, or artifact paths reach the model?
  • Can a continuation use a parent or option not offered by the initial packet, fail, or change request/publication without invalidating the row?
  • Can repository context exposed by an unfamiliar, extensionless, globbed, failing, or parallel command/tool evade either side of the observed-action partition?
  • Can missing, reused, dirty, incompletely adjudicated, or candidate-only critical evidence pass a decision?
  • Does semantic-off execute the normal descriptor pipeline with only Stage1b removed?

Focused verification

  • JavaScript analyzer and ablation contracts: 210 passed.
  • CLI packet feature lane: 59 passed.
  • Retrieval benchmark tests: 2 passed.
  • Runtime benchmark proof test: 1 passed.
  • Value-score tests: 16 passed.
  • Plugin static tests: 140 passed, 1 Windows-only skip.
  • Packet generalization boundary, workflow policy, documentation links, formatting, and diff checks: passed.

Non-claims

  • The 120-row builder ablation has not run.
  • This does not establish packet usefulness, release accuracy, SLO compliance, installed-product acceptance, or 0.18 readiness.
  • No version, calibration constant, package, workflow, or release surface changed.

Closes #2108
Refs #2098

@TheGreenCedar
TheGreenCedar force-pushed the codex/2108-v018-builder-ablation branch from 8f2e699 to 3d6b1cb Compare September 2, 2026 13:53
@TheGreenCedar
TheGreenCedar marked this pull request as ready for review September 2, 2026 13:58
@TheGreenCedar
TheGreenCedar merged commit 114542c into dev/codestory-next Sep 2, 2026
8 checks passed
@TheGreenCedar
TheGreenCedar deleted the codex/2108-v018-builder-ablation branch September 2, 2026 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

1 participant