Skip to content

assertrust v0.2.0

Latest

Choose a tag to compare

@github-actions github-actions released this 18 Nov 21:56
v0.2.0
c4a4d88

assertrust 0.2.0 adds an optional AI judge. The rules only read test code, so a test deleted along with the feature it tested looks just like a check quietly dropped to get CI green. With your own key, the judge reads each finding with the change around it and answers one question: does the test still check what it promises?

build/validate-exports.test.ts  HIGH  test-disabled
  file deleted: its test no longer runs · fcd351b
  judge: coverage lost: The validateExports logic was moved verbatim into a vite.config.ts build plugin that still runs in the build and throws on mismatch; the old unit test was deleted with its module and not replaced, though the behavior stays.

Turn it on

      - uses: BeyondFolder/assertrust@v0
        with:
          judge: true
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Or assertrust check --judge in a terminal. Findings it calls fine are folded away, and those it says lost coverage or look suspicious fail the check. Each change is judged once and the verdict cached, at about 2 cents a finding with the default, Sonnet 5.5. OpenAI, Gemini and local Ollama models work too. Without a key, assertrust works exactly as before. The README has the details.

Status: early. On the labelled history of 11 repos, 88% of the findings the judge fails are right, and it folds away 80 of the 97 false alarms.

Release Notes

The optional AI judge: with your own key, a model reads each finding with the change around it and says whether the test still checks what it promises.

  • assertrust check --judge gives each shown finding a verdict: fine, coverage-lost, suspicious or unsure. Findings judged fine are folded away, and coverage-lost and suspicious fail the check ([judge] fail-on). Changed expected values, hidden until now, show up when the judge doesn't call them fine.
  • Sonnet 5.5 at medium effort by default. --judge-model and --judge-effort, or [judge] in assertrust.toml, pick another: models of Anthropic, OpenAI, Google Gemini and Ollama work, named provider/model.
  • Verdicts are cached in .git/assertrust/judge/, so a change is judged once and CI never flips. With no key, no network or a bad answer, findings show without verdicts and the check doesn't fail.
  • --judge-dry-run prints exactly what a model would be sent, and needs no key.
  • Verdicts show in the terminal, in JSON ("judge", still "schema": 1), and in the GitHub annotations and job summary.
  • The Action takes judge, judge-model and judge-effort, and keeps verdicts between runs with actions/cache.
  • assertrust backtest --judge measures the judge against labels. On the labelled history of 11 repos, with the default model, 88% of the findings it fails are right, and it folds away 80 of the 97 false alarms.
  • Building from source needs Rust 1.97.1.

Install assertrust 0.2.0

Install prebuilt binaries via shell script

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/BeyondFolder/assertrust/releases/download/v0.2.0/assertrust-installer.sh | sh

Download assertrust 0.2.0

File Platform Checksum
assertrust-aarch64-apple-darwin.tar.xz Apple Silicon macOS checksum
assertrust-x86_64-apple-darwin.tar.xz Intel macOS checksum
assertrust-aarch64-unknown-linux-gnu.tar.xz ARM64 Linux checksum
assertrust-x86_64-unknown-linux-gnu.tar.xz x64 Linux checksum