Skip to content

v0.4.0

Choose a tag to compare

@github-actions github-actions released this 04 Nov 09:14
· 1582 commits to main since this release
72e7d30

Minor Changes

  • Measured review rates instead of impressions. A marked corpus-* bundle kind
    carries a checked-in case manifest with ground-truth bugs and provenance;
    calibrate grades a model's findings against that ground truth offline and
    reports precision, recall, and false-positive rate, and trend aggregates the
    same numbers per model and panel. Ordinary local-* reviews keep refusing the
    live join. Ships a seeded starter corpus under bench/corpus/ and
    docs/benchmark.md, an operator runbook for real PRs, real agents, cost
    estimates, and publishing measured rates.

  • Experimental MCP server. verdict mcp serves the review verbs to an MCP host
    over stdio (newline-delimited JSON-RPC 2.0): no daemon, no HTTP, no new
    dependency, sharing the host's credentials and exiting when stdin closes.
    Tool schemas reflect the CLI request types, results are the --json
    envelope's data, and typed refusals return isError with code, message, and
    hint intact; --read-only exposes only the non-writing verbs, while post is
    available in normal mode and in --operator mode. verdict mcp config --agent claude|cursor|codex prints the exact host entry, and --write merges it
    idempotently into the documented config (.mcp.json, .cursor/mcp.json, or
    $CODEX_HOME/config.toml), backing the old file up to <path>.verdict.bak
    and refusing unparseable or foreign entries instead of rewriting them. --json
    prints one envelope, never prompts, and never starts the server; verdict --help now lists the MCP entry so the surface is discoverable.

  • verdict is one static binary — no checkout, no jq, no shell script. Install
    it with the verified install.sh (macOS/Linux) or install.ps1 (Windows) into
    ~/.verdict/bin, the Homebrew tap, or the npm wrapper; both installers verify
    the cosign signature and the archive SHA-256, smoke-test the installed binary
    from its final path, and add PATH with one marked line
    (VERDICT_NO_MODIFY_PATH=1 opts out). In CI the new setup-verdict action
    installs a pinned, verified copy on Ubuntu, macOS, and Windows runners without
    touching shell rc files. verdict upgrade updates whichever channel installed
    it — a verified self-replace for installer installs, the owning package
    manager's global update for npm and Homebrew — and verdict uninstall removes
    only installer-owned files; both refuse, naming the exact command, when the
    install belongs to something else. verdict skill prints or installs the
    payload embedded in the binary (SKILL.md, prompts/, schemas/, and the
    scripts/verdict shim), so the served skill can never skew from the binary.

    Upgrading from 0.3.x: the old script install is replaced, not patched.
    scripts/review.sh is gone — every verb is verdict <verb> with the same
    flags and --json envelopes. The old symlink-only install.sh became
    install-skill.sh (checkout-only), and the copy-based install verb is gone,
    so reinstall the skill with verdict skill install or npx skills add. Docs
    and scripts that call the old script path must be updated. Start at the
    rewritten quickstart, the new glossary of review terms, the CI recipe in
    docs/ci-recipe.md, and verdict --help, which now names each verb's
    provider scope.

  • Posting is harder to get wrong and harder to do twice. post refuses a second
    review on the same reviewed SHA before any network read, naming the exact
    fetch --base-sha advance; it also refuses duplicates found in the bundle's
    own fetched review list, so a fresh checkout, a CI re-run, or another machine
    cannot repost what the PR already has — --force still overrides, and the
    refusal names the re-fetch needed when the earlier review was deleted. Points
    outside the diff hunks post under "Unanchored observations" instead of as
    inline anchors GitHub would reject. A body past the 65,536-character hosting
    limit splits into follow-up COMMENT reviews — first review keeps the footer,
    later parts carry a continuation header, and a failed follow-up refuses typed
    naming the review that already landed — instead of being trimmed. A 403/429
    carrying a rate-limit signal retries within bounds, honoring retry-after or
    the reset marker and never publishing twice; when the last permitted attempt
    is still retryable, the refusal carries retry_at in RFC 3339 so an agent can
    wait for that instant.

    A stale-fetch downgrade note is now followed by a blank line, like the drift note, instead of a stray literal \n in the posted review body.

  • The PR conversation is a local, schema'd record instead of something each verb
    re-reads from the host. Every fetch writes .review/<pr>/threads.json (schema
    published) holding each thread's host id, diff anchor, resolved and outdated
    state read from the host's own fields, root author kind, and the comments with
    authors, timestamps, and capped bodies; GitHub comes from GraphQL
    reviewThreads (REST carries no resolution), and GitLab, GitCafe, Forgejo,
    Bitbucket, and Azure fill the same shape, paging rather than truncating.
    verdict threads <owner/repo> <pr> [--provider …] [--json] re-reads just that
    surface and rewrites only that file, so re-review, address-reviews, readiness,
    and calibration can join prior feedback on any of the six hosts. Posted inline
    comments now carry a hidden final-line <!-- verdict-finding sig=… --> marker,
    so a new comment links back to the finding that produced it while older
    comments keep reading as unlinked.

  • Published machine contracts beside the CLI. schemas/ ships JSON Schemas for
    the review input, findings store, cost and usage records, PR threads, and the
    --json envelope with per-verb data profiles, included in the npm tarball;
    schemas/validate.sh checks documents against them offline (no jq; exit 0
    valid, 1 rejected, 2 bad usage), and the drift rule is additive within schema
    version 1. An experimental TypeScript client
    (@stainless-code/verdict/client) wraps each verb with envelope types
    generated from those schemas and refusal codes generated from the CLI, with no
    runtime dependencies. The experimental Go package pkg/verdict embeds the
    same operations in-process through a context-first client with typed requests
    and responses and a *RefusalError for errors.As; its API may still change
    before 1.0. A new "Build a client" guide documents the --json protocol —
    envelope keys, exit and refusal codes, versioning, binary resolution,
    timeouts, output caps, and when MCP beats spawning a process — with a
    dependency-free Python client generated from the schemas and an offline smoke
    test.

  • The readiness gate can say what it knows. report --check and
    report --completeness gain a live-thread dimension: open threads the local
    store does not know about fail the gate, and unreachable lookups warn and stay
    neutral. New --live auto|strict|skip picks the posture — auto keeps that
    fail-open default, strict fails closed when the host cannot be reached, and
    skip pays zero network and gates on the local dimensions only, saying so in
    the output and the --json warnings. post and the gate now name the same
    advance for head drift, and a binding APPROVE on a known-stale fetch
    downgrades to COMMENT instead of approving past the moved head.

  • Chunked split-range reviews. Very large PRs are reviewable again: with
    --chunk (or .review/chunk) an over-cap diff is sliced on file boundaries
    into numbered chunks with a versioned manifest, the agent reviews one chunk at
    a time, and the results synthesize into one verdict with a summed Basis line.
    post and record refuse coverage that claims to be complete but is attested
    partial, and warn loudly when a chunked review is unattested. fetch-local
    supports the same flow.

  • verify refuses before it runs. Destructive, networked, credential-reading,
    and repo-escaping commands are rejected with a typed error before execution
    (over 40 regression shapes), so a hostile or careless verifyCmd cannot
    delete, exfiltrate, or escape the repository. The recorded tail also drops
    credentials no environment value names — vendor-prefixed tokens,
    Authorization and Proxy-Authorization headers, bare Bearer values, and
    URL userinfo — and strips ANSI escapes and other control characters, so a
    command cannot rewrite the proof it lands in; .verify.out keeps the raw
    bytes. doctor warns on excess token scope, and the manual documents an
    opt-in hostile-PR sandbox recipe plus the residual risk a string scan cannot
    remove.

Patch Changes

  • Cost accounting reads what the operator actually knows. cost.json entries
    gain optional cache_read_input_tokens and cache_creation_input_tokens
    (subsets of input_tokens; absent or null reads unknown), and
    report --cost rolls them into a top-level cached_input_tokens scalar with
    totals unchanged. The spend gate can now price comparable runs from usage
    records the operator hands over with VERDICT_USAGE=<file|dir>, in a published
    schema (schema, model, tokens, optional USD, source, window), with the
    self-reported cost.json as the fallback; a handed record joins by its window
    bracketing the run, and a named path that is missing or malformed refuses typed
    before the bundle is written. Automatic discovery of other tools' transcripts
    is gone — the optional converters live outside the binary in
    examples/usage-adapters/. With nothing handed over, output is byte-identical.

  • Isolated local reviews. fetch-local --worktree checks the review target into
    a temporary detached worktree and builds the bundle from that checkout instead
    of the live tree, so caller tracing and verification run against real, stable
    code without disturbing the working tree. The checkout is always removed
    before the verb exits (success, refusal, or cancel); when no checkout can be
    created the verb refuses typed worktree-failed and writes nothing. Without
    the flag every byte behaves as before.

  • Small additive reads. post.json and record.json now record tool_version,
    prompts_hash, and model, and calibrate and trend surface them so a
    mixed-version history reads as mixed. show --json gains a data.fix block
    beside the unchanged data.item: full sig, status, severity, deduped files,
    locator, suggestion, a verify_hint read from the bundle, and a fixed
    untrusted-evidence caution — a pure read with no flags, no execution, and no
    store writes, and missing history reads as null rather than invented text.
    doctor human output opens with a run stamp and closes with one line,
    Summary: healthy|unhealthy — N ok, M warn, K fail, while --json envelopes
    and exits stay byte-identical. Builds with no release metadata now say
    0.0.0-unknown everywhere — --version, the skill payload and receipt, and
    the posted or reported footer — instead of the footer saying unknown.

  • Failures a 0.3.1 user could hit are gone. fetch on GitLab looks up the
    project id with the plain owner%2Frepo path — the old request carried a
    trailing encoded newline, which could fail the lookup at the first step.
    Provider lists are read completely: Forgejo files and reviews, GitLab MR notes,
    and Azure change entries and threads page instead of stopping at the first
    page, and a page cap, a failed page, or a failed thread read lands an envelope
    warning instead of silently under-covering. fetch retries GitHub's transient
    "internal error, please try again" on the same three-attempt budget post
    already had, while auth and validation failures still fail fast. Machine mode
    never prompts, even on a TTY: init --json and install --json without
    --yes/--force refuse typed needs-confirm, write nothing, and print the
    hint instead of waiting on a Proceed? prompt. Malformed agent input fails
    closed with bad-review-input — unparsable review.json, non-object
    comments/triage/sections entries, non-string scalars, misshapen
    verification claims — and a corrupt findings store now refuses typed
    corrupt-store across report, show, calibrate, and trend instead of
    crashing mid-render with no envelope, never starting fresh over history.

    A value flag given without its value now fails as a usage error naming what
    it wanted (--provider needs a value (want <name>), exit 2) before any
    network call, instead of falling back to a default target. A bare --stash
    still means stash@{0}.

    Cancellation is clean. SIGTERM now cancels a running verb the way Ctrl-C
    does: exit 3 with a canceled envelope, and child process groups are killed
    instead of orphaned. A cancelled verb returns promptly even when a grandchild
    process still holds its output pipe. Interrupting verify refuses
    canceled and records nothing, where it used to record a false
    timeout after 300s proof and exit 0. The mcp stdio server takes the same
    signal: SIGTERM cancels an in-flight tool call and kills its process tree
    instead of orphaning a running verify for the rest of the verify budget.
    On Windows, verdict upgrade stages its
    helper beside the installed binary, so an install directory on another drive
    no longer makes the final move fail after the upgrade reported success.
    On Windows, verify starts its command suspended and puts it in its
    kill-on-close job object before it runs, so a grandchild spawned in the first
    instant can no longer escape a timeout or cancel kill.
    threads.json always labels a thread's author kind, including a thread with
    no readable comments, as the published schema requires. A cancel that lands
    inside a read verb's service (ready, trend, threads, calibrate,
    check-tree, doctor, report, next) exits 3 with the canceled envelope
    like every other verb, where it used to exit 1 as an ordinary refusal.

    Provider page walks finish what they start. A GitLab merge request whose diff
    list spans host pages (glab api --paginate concatenates one JSON array per
    page) now yields every changed file instead of a PR.json with no files at
    all, and a merged Bitbucket pull request (state: MERGED) folds to merged
    rather than unknown. A GitCafe review/comment cursor walk that stops early — a
    failed page, an unreadable page, or the 20-page cap — keeps the pages it already
    read, warns, and marks threads.json truncated: true instead of quietly
    under-covering the conversation; Forgejo's review list does the same at the
    provider cap. Hosts with no failures read exactly as before.
    A failed or unreadable Bitbucket diffstat/comments page, a GitLab changed-file
    payload that is not a diff list, and a failed Azure iteration read no longer
    pass as an empty list: each warns, and the Bitbucket threads feed is marked
    truncated: true.

    Report artifacts survive a kill. report --output, report --sarif, and the
    MCP report artifact are staged beside the target and renamed into place, so a
    crash mid-write can no longer leave a truncated report behind. A symlinked
    --output path keeps the link and replaces only the file it names, an existing
    file keeps its own mode (a new one is still owner-only), and a device like
    /dev/stdout still receives the bytes directly. A failed write refuses
    report-failed and leaves the previous report intact.

    Numbers in machine output render the way jq 1.8 renders them. A literal such as
    9.223372036854775808e18 now prints as 9223372036854775808 on every
    architecture, where arm64 used to saturate at 9223372036854775807 and amd64
    wrap to -9223372036854775808.

    verdict mcp install no longer corrupts a Codex config.toml that uses a TOML
    multi-line string: a [mcp_servers.verdict] header spelled inside a ''' or
    """ literal used to be merged into as if it were real, and the merge now
    refuses mcp-config-unmergeable (file untouched, add the table by hand)
    instead of guessing.

  • Reviews of other repositories stop carrying this repository's own conventions.
    The shared system prompt now defers branch, commit-message, formatting, and
    style rules to the target repo's AGENTS.md/CLAUDE.md/CONTRIBUTING, with
    verdict's own rules staying in its AGENTS.md, so reviews of this repo keep
    them. The fix flow opens with the untrusted-data preamble, the abort-if-not-
    introduced pin is stated where the flow can follow it, and the manual records
    the machine-modes-never-prompt doctrine and a combined
    report --check --completeness pre-push probe. --show-prompts wraps every
    untrusted bundle section in a per-run nonce fence named in a trusted prologue,
    so a forged close smuggled in a diff stays evidence instead of ending the
    section; thread files stay off the initial-review reading list by design.
    Missing trusted prompt files no longer vanish silently: a missing
    prompts/system.md refuses typed, while a missing SKILL.md,
    prompts/review.md, or prompts/diagrams.md warns and proceeds; missing
    bundle files still skip silently.

  • Credential and untrusted-input hardening. Every credential-bearing provider
    read is bound to the configured origin: fetch refuses a Bitbucket pagination
    next, a Forgejo metadata diff_url, or an Azure metadata remoteUrl whose
    scheme and host differ from the API base, instead of sending the token to that
    host, and Bitbucket paging stops at a page cap or a repeated cursor with a
    warning that the bundle may be incomplete. Command-stderr sidecars written by
    fetch and post (.diff.err, .post.err, .recheck.err, provider .err)
    are now owner-only 0600, since they can carry token-adjacent forge
    diagnostics. Untrusted input is capped: record and post refuse an oversized
    agent file while reading it instead of after allocating the whole document,
    verify and the fetch runners cap captured child output (a capped proof
    records a truncation marker instead of growing the process), and every store
    reader rejects oversized documents while keeping its own refusal code. A review
    anchor, knowledge-base counter, override threshold, adopt anchor, verify
    timeout, or read number that does not fit in an integer now takes its typed
    refusal instead of wrapping into a nonsense negative span or count.

    The verify gate now refuses a command it cannot fully scan. A destructive
    tool name hidden inside $(…) substitutions nested deeper than the gate's
    scan bound, or inside a substitution body glued directly to a word (for
    example $( (echo r))m -rf dir), was previously treated as clean. Both now
    refuse before anything runs. A pull request file name containing a tab or
    newline can no longer forge or shift rows in hunks.tsv, and the ordered
    JSON reader rejects trailing bytes after the document ({"a":1}}) instead of
    silently ignoring them.