Skip to content

v1.0.0 — Catches nobbled tests

Choose a tag to compare

View all tags
@TheSoloHacker47 TheSoloHacker47 released this 01 Sep 14:41
· 3 commits to main since this release
v1.0.0
069f82f

Nobble reads a pull request diff and flags changes where the tests were weakened to make the code pass, rather than the code being fixed to make the tests pass.

Static, deterministic, offline. No LLM in the detection path, no network calls, no data leaves your runner.

Quickstart

- uses: actions/checkout@v4
  with: { fetch-depth: 0 }
- uses: TheSoloHacker47/nobble@v1

Or locally: npx nobble --base main

What's in it

  • 16 rules across assertion tampering (NOB-101…105), security and boundary mocking (NOB-201…203), escape hatches (NOB-301…303), config and CI tampering (NOB-401…404), and suppression hygiene (NOB-001).
  • Three language adapters — TypeScript/JavaScript/TSX (Jest, Vitest, Mocha, Chai), Ruby (RSpec, Minitest), Python (pytest, unittest) — plus a regex fallback so skip, suppression, coverage-config and CI rules run on every language.
  • Four reporters: terminal, markdown, JSON, and SARIF for inline annotations via code scanning.
  • Non-blocking by default. The action comments and exits 0 unless you opt into fail-on: block.
  • Inline suppression via nobble-ignore <RULE>: <reason>, with the reason mandatory — a suppression with no reason doesn't suppress, it reports itself.

Measured, not claimed

Run against the last 50 merged pull requests of vitejs/vite, pallets/flask and sinatra/sinatra — 150 ordinary human PRs:
PRs flagged 5.3% (8 of 150)
PRs that touched a test file and were flagged 16.7% (7 of 42)
PRs reaching verdict block 2.0% (3 of 150)

The three that would block are sinatra/sinatra#2115 (titled "Skip broken tests."), its revert, and one deleting two test cases — all worth a reviewer's attention.

Performance: 1002 changed files across three languages in 700ms, against a 5s budget.

Reproduce the rate yourself with npm run smoke -- --limit 50 --verbose.

See it working

Pull request #1 weakens a payment suite on purpose — assertions removed, a test deleted, a test skipped, current_user mocked, and an unconditional return true added to an auth middleware. Nobble's comment on it is the screenshot in the README.

Notes

The Action runs on node24. Node 20 was removed from GitHub-hosted runners on 2026-09-16, so an action declaring node20 no longer runs.

Every ambiguity resolved while building this — including the two places the original build spec turned out to be wrong about the world — is recorded in DECISIONS.md.

Assets 2
Loading