docs: publish the 29-application breadth corpus beside the two head-to-heads - #415
Merged
Conversation
…o-heads benchmark/reliability/summary.json records 17 verified successes, 10 safe halts, and 2 wrong actions over 29 applications. The site published the two single-task comparisons and not this one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
✅ Deploy Preview for cosmic-klepon-3c693c ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
This was referenced Aug 28, 2026
abrichr
added a commit
that referenced
this pull request
Aug 28, 2026
…418) main went red on the first sentence to arrive after the universal guard landed. #415 added "all 29 compiled, 17 replays reached a verified success, 10 halted safely, and 2 reported success while the external oracle disagreed" to /research, and the guard refused it because nothing said what makes it true. That corpus lives only in openadapt-flow, so the sentence cannot be checked against data/benchmark.json. Rather than attest it against a URL nobody opens, vendor benchmark/reliability/summary.json at 7472d19d with its sha256 and read it. Evidence records may now name an upstream source, and an exhaustive record may list the counts quoted in the same sentence, so "all 29 compiled" is 29 against 29 and the three outcome counts are bound field by field. A universal usually travels with the numbers it summarises, and half-binding it leaves the other half free to drift. Two tests: a tampered snapshot fails the digest, and a quoted count that stops matching the artifact fails by name. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A selection problem rather than a wrong number.
openadapt-flow/benchmark/reliability/summary.jsonrecords one record-compile-replay observation for each of 29 public web applications: all 29 compiled, 17 replays verified, 10 halted safely, 2 reported success while the independent oracle disagreed. It is the least flattering result we have measured.It was cited in the paper body and nowhere else. Not on
/research, not inllms.txtorllms-full.txt, not on the blog or the org profile./researchlisted the OpenEMR head-to-head, the drift repair and the identity ladder, all of which read well, and skipped this one./researchnow carries a bullet for it, second in the list so it sits next to the OpenEMR result rather than at the bottom.llms-full.txtcarries the paragraph version with the failure breakdown and the caveats, right where the two head-to-heads are already stated honestly.Companion PRs put the same corpus in
openadapt-flow's README,docs/BENCHMARKS.mdand the paper abstract (openadapt-flow#425).npm test(261),npm run check:versions,npm run check:linksandnpm run buildall pass.