Repository navigation
v1.0.0
Immutable
release. Only release title and notes can be modified.
Added
- Settings checks:
tiers:can't putsecuritybelow Sonnet (a lower tier is raised to Sonnet, with a warning), and askip:glob that matches nearly every path, such as**, and aCODE_REVIEW_SHADOWname that matches no reviewer are warnings in the check run (#94). - An eval case can record the author association of each issue its description closes, as
issueAssociations(a member by default), and the evals leave out a non-writer's issue as the review step does (#109). - Timing instrumentation: each model call's record in the trace has when it started (
startedAt) and when its first response came (firstResponseMs), the trace has the engine's stage times (timings),node tools/timings.tssummarizes downloaded trace artifacts,npm run bench:reviewtimes a review on a fake agent with production's call durations, andnpm run bench:teststimes each test file (#128). - The README's Interface section writes down the caller–engine contract v1.0.0 will freeze: the action's inputs, the triggers and types a caller needs, each job's permissions, environment and secrets, and the
basecheckout (#57). - The Eval workflow runs a named eval subset (
smokeorre-review) by hand frommain, with verification on or off, aCODE_REVIEW_TIERSoverride, up to two runs and a cap of $5, $10 or $25 a run, in thecode-reviewenvironment, and uploads the outputs after a secret scan, so evals need no local Bedrock key.CODE_REVIEW_TIERStakessynthesis=<tier>andCODE_REVIEW_SYNTHESIS_EFFORTsets the synthesis call's effort, to measure synthesis on a cheaper setup (#135). - The smoke eval's first baseline,
evals/baseline-smoke.json, from the run onmainafter the decisions rule was narrowed: caught 2 of 3, the must-not case quiet (#52).
Changed
- The tests cover how a review builds each reviewer's prompt on a diff in parts (its own part's diff, rules and charter, with the other parts' files marked as in another part), run the repository's own reviewers on the tier its config gives them, and rerun the selection and context calls on Bedrock when the API credits run out (#96).
- This repository's own workflow pins v0.4.0 and triggers on
closed, so its pull requests are reviewed by the current engine and record each finding's outcome when they close. - When the Anthropic credits are used up and Bedrock is throttling the key or its quota is used up (HTTP 429,
ThrottlingExceptionorServiceQuotaExceededException), the check run says the review budget is used up, neutral, instead of starting a review Bedrock would refuse partway (#92). - The light review no longer takes an SVG, which can carry script, or a
.txtfile outside the repository root anddoc/anddocs/folders, such as a fixture, a data input or a license list: they get the standard review and are no longer ordered last in an oversized diff, and an SVG's diff reaches the reviewers instead of a generated-file stub (#91). - A pull request counts as a template adoption or update, whose findings on the template's files go upstream, only when it adds
.copier-answers.ymlor changes its_commit; any other edit to the answers file is reviewed as the project's. The light review reads the template's version files from the same list as the rest of the engine (#90). - The
api-compatfloor also readsIsPackablefromDirectory.Build.propsandDirectory.Build.targets, in MSBuild's order: the props from the root down, then the.csproj, then the targets, the last that sets it winning (#94). - Verification no longer drops a refuted blocker or
securityfinding: it stays as a question one severity lower, with the refutation in its evidence. Any refutation must cite a line of the checkout outside the lines the pull request adds; one that rests only on the change's own lines keeps the finding as a question (#87). - Verification stops at the run's cost ceiling like the reviewer calls: its calls that haven't started are skipped, their findings stay unverified, and the summary says how many were skipped (#87).
- A re-review also runs
securitywhen what changed since the last review touches a filesensitive:names or code the selection floor gives tosecurity, such as a path withauth,tokenorcryptin it, and a change to asensitive:file makes it a deep review:correctnessandsecurityrun on Opus, unlessCODE_REVIEW_TIERSnames them, verification checks the findings in sensitive files on Opus, and the summary and the check run say so. Each of its reviewers checks the earlier findings, and the most cautious status counts; when one of them fails, the re-review posts no findings and the next one covers those changes again (#86). - The tests check that the evals' review cost leaves out the concern judge's and precision judge's calls, which the case cost counts, so
eval:comparenever reports the judges' spend as review cost (#107). - The tests check what a
/review explainrun leaves for the publish step: onlyanswer.jsonfor an unknown finding, without a model call,trace.jsontoo when the call ran, theansweredoutput, and neverresults.json(#108). - The evals read the issues a case's pull request closes with the review step's own reader, so the two can't drift apart (#111).
- The
api-compatfloor places each changed C# line in its member by reading the file's braces, with comments, strings and preprocessor lines left out, instead of its indentation, so unusual formatting no longer misplaces a change; a file whose braces don't balance counts as a public change (#112). - A re-review posts a problem
correctnessandsecurityboth report, in the same file within three lines and with the same title words, once, at the higher severity and naming both reviewers, instead of two threads; its log counts these merges and the findings of the two that sit that close without matching, for the evals to measure (#116). - The evals have a re-review case whose changes touch a file the selection floor gives to
securityand one whose changes touch asensitive:path, which a case can now set for a repository that has none;--subset re-reviewruns the four re-review cases, whose output has each one's cost (#117). - The smoke eval fails when
evals/baseline-smoke.jsonis missing after it has been committed, naming the commit that deleted it, so deleting the baseline can't turn the gate off; until the first baseline is committed it still passes with a notice. Its workflow checks out the full history for this (#93). - The tests check that the smoke eval workflow uploads its outputs only after the secret scan passes and fails the job when the run or the check fails, and that
evals/run.tswrites its output before it fails a run over--max-cost(#93). - A private organization member's comment counts as the maintainer's for the loop limit, their open issues can track a finding, and the bot answers their
Autopilot:replies: these use the engine's writer check, the author's association or else their permission on the repository, and a lookup that fails counts as no write access. The README says how such a member can use/review, whose gate still reads only the association (#89). - Most GitHub requests go through one client in
src/github.tswith a 30-second time limit, to the address the runner names (GITHUB_API_URLandGITHUB_GRAPHQL_URL, set on a GitHub Enterprise Server) or github.com; re-reviews and outcomes read review threads with one shared reader, and the bot's comments neutralize @mentions and link text with shared helpers (#95). - The engine's code is reorganized, with no change in behavior: the C# contract checks move from
gather.tstocontracts.ts,stoppedBytochecks.ts(withNoProviderErrorin its ownno-provider.ts, so the acknowledgement still loads no provider code), and reading a finding comment's title and text tothreads.ts;review()takes an options object and runs a light review in its ownreviewLight(); the eval output file has one type its writer and readers share; one helper counts the findings that belong upstream; and the workflows read the Node version from.node-version(#95). - The rest of the engine's GitHub reads go through the one client in
src/github.ts, with its time limit and the runner's GitHub address: the open issues synthesis reads, the issues a description closes, permission lookups and the training's guidance issues. Outcomes and the publish step's dismissals use the engine's writer check, so who counts as a writer can't drift between them; a lookup GitHub answers with 404 counts as no write access without a warning. The review step reads a pull request's changes with git once for gather and the template check, and the finding marker is read in one place (#95). .github/code-review/config.ymland a project reviewer's frontmatter are read with theyamllibrary, pinned to an exact version, accepting the same subset as before except that a file whose keys are all indented the same now reads and a double-quoted string takes every YAML escape. An error names its line followed by the library's own words (#95).- A review's reviewer calls start sooner: each part's calls on one model start once the first of them has its first model response, which has written the shared prompt cache, instead of after every first call has finished, and waiting calls start longest first. In a simulation on production's call durations a full review takes about 10 s less (#128).
- The selection call runs beside the context call instead of after it, about 5 s sooner;
conventionsstill joins the reviewers when the context call picked rules for it (#128). - The context, selection, synthesis, re-review and
/review explaincalls cache their prompts: each takes at least two model requests, and the second no longer pays for the whole prompt again (#128). - A closed pull request's run skips
npm ci: recording outcomes loads no npm package, since the engine loads the Agent SDK only for a model call. These runs take about 5 s less (#128). - The
Enable-CodeReviewtests run concurrently, which shortens CI's test step by several seconds (#128). - The tests run faster on Windows:
test/rereview.test.tsbuilds each git repository its tests only read once instead of once per test, and the fetch time-limit test's server drops its connection after the limit, so the killed git'sgit-remote-httpno longer holds the folder through the cleanup's retries; that test now checks the folder is removed on Windows too (#136). - Callers record outcomes in a job of their own,
outcomes, which runs only when a pull request closes, outside thecode-reviewenvironment and withoutid-token: write, the Bedrock key or a checkout, since it makes no model call; the review job no longer runs onclosed. This repository's workflow and the README's caller have it, and the README's examples pin v0.4.0 (#47). - Reviewers no longer question a decision the pull request adds for a scope or limit it states, such as leaving existing data as it is: they question it only where it contradicts the code or another decision. The smoke eval's must-not case flagged such a question (#52).
- Verification is on by default;
CODE_REVIEW_VERIFICATION=offturns it off. On the smoke evals it caught as much as without it, with no must-not flag, at 5 blockers and should-fixes against 7 and 8, for about 3% more. The Eval workflow runs it by default too (#41). - The README's Evals section records the 2026-11-01 results: verification on and off, synthesis on Sonnet and at medium effort, which stays on Opus at the default effort since Sonnet posted more serious findings, and the re-review cases' cost, with no overlapping findings for
mergeOverlappingto merge (#123, #135).
Fixed
- A re-review no longer fails when GitHub's answer for the earlier review threads isn't JSON or its connection drops partway: it goes on without the earlier findings and won't approve, as when GitHub refuses the request (#92).
- A closed issue that can't be read, such as on a network failure, no longer fails the review step: the reviewers go without it and the log says why. A failed lookup of an author's permission, which leaves their description out, warns with the reason too (#92).
- The log warns, with the reason, when the pull request's comments can't be read, so a run that couldn't tell which commit it reviewed last, and so reviews in full, says why (#92).
- The evals build each case's pull request as the review step does: a case can name its author and association (a member by default), a description from an author without write access is left out with the issues it closes, and an issue that can't be read warns instead of vanishing (#92).
- Findings an open issue already tracks no longer count toward the verdict or the finding totals, so a pull request doesn't show request-changes for problems tracked elsewhere. The summary comment and the check run list them apart, under "Tracked in open issues" (#98).
- A template adoption or update leaves a changed file to the template only when it matches NCodeGroup/project-template at the
_committhe answers name, a commit the template's default branch or one of its tags reaches, so a file edited by hand in the same pull request, or in a fork of the template, is reviewed as the project's, and every such file is when the template can't be read. A file rendered from a.jinjasource counts only when it is that source's exact rendering with the answers' values, so one whose source has a block, a filter or a name the answers don't hold is the project's too (#113). - A summary no longer says verification couldn't settle a finding it refuted but kept as a question, a refuted blocker on a re-review or a refutation that cites only the lines the pull request adds: it says the finding was refuted and kept (#114).
- A re-review shows the reviewers the replies in the earlier findings' threads from an organization member whose membership is private, and a finding's outcome counts their replies: when a reply's author association doesn't name a writer, both ask the repository's permissions, as the description's check does, looking up at most 20 repliers per review (#124).
- A finding's outcome is read from the marker that ends its thread's first comment, as everything else reads it, so a one-click suggestion whose code holds a marker-like line can't change which finding the outcome records (#95).
- The
api-compatfloor reads a C# verbatim string whose text starts with an escaped quote, such as@"""Hi"" there", as a verbatim string, instead of as a raw string that ran to the end of the file and made every change in it count as public. A measurement across NCodeGroup's C# found no file where conditional compilation unbalances the braces, so the scanner stays (#122). - A finding's outcome follows a writer's latest
Autopilot: fixed in <sha>reply when the pull request gained that commit after the review, even when the fix is on other lines, and a writer'sAutopilot: not changedreply, even when the line changed for another reason; the monthly report's hand count found both counted the other way (#97).
Security
- A step that always runs revokes the acknowledgement's app token before the review step starts, even when the acknowledgement script never ran or stopped before revoking it (#92).
- Changed file names are logged without control characters, so a file name can't break a log line or start a workflow command (#92).
- One-click suggestion blocks are posted only on pull requests whose author can write to the repository. On anyone else's, the fix is in prose only and the check run says how many suggestions were held back, so a model-written change prompted by an outsider's code is never one click from being committed (#88).
- Every line the engine writes to the workflow log is made one printable line in one place: a line break becomes a space and other control characters are dropped, so a file name or other pull request text in any log line can't break it, start a workflow command or forge another line (#110).
- Model-written text in the bot's comments (the summary, finding threads,
/review explainanswers, the outcomes table and still-open replies) can no longer mention anyone or post a link: outside code spans and fenced code blocks, a zero-width space follows each@that starts a mention and breaks](,][,]:,http://,https://andwww., and text whose code GitHub might read differently is escaped throughout. Code such as@Overridein backticks stays as written (#95). - Model-written text in the bot's comments can no longer cross-reference an issue or pull request: outside code spans and fenced code blocks, a zero-width space follows each
#that starts a reference such as#12orowner/repo#12, and breaksGH-12, including when the#, the hyphen or a digit is written as a character reference such as#,#or-. The engine's own references, such as the issue that tracks a finding, stay live (#132). - The action turns off setup-node's automatic npm cache, so the job that holds the model credentials never restores a cache, and the README says why the action doesn't cache its dependencies: the review's triggers can only read the default branch's cache, and a cache any trusted-trigger workflow in a consumer saved would be restored as-is into that job (#134).