Skip to content

Releases

Honest Tests Audit v0.2.0

Choose a tag to compare

@ranjan-mohanty ranjan-mohanty released this 15 Nov 01:26
af5918e

Honest Tests Audit v0.2.0

Find the tests a pull request adds that can't fail, and say so in a PR comment.

honest-tests is an agent skill plus a CLI that finds tests which pass whether or not the code works:
expect(x).toBe(x), tests with no assertions, assertions on a mock's configured return value,
try/catch blocks that swallow the failure, and more. Built-in mutation testing then proves which
tests are weak. This release adds a GitHub Action, Python support and four more JS test runners.

demo

New: the GitHub Action

# .github/workflows/honest-tests.yml
name: Honest tests
on: pull_request
permissions:
  contents: read
  pull-requests: write # create/update the PR comment
jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0 # the audit diffs against the merge base
      - uses: mintwork/honest-tests@v0

It audits only the test files the PR changes and keeps one comment up to date:

Honest Tests Audit

This PR adds 12 tests — 4 can't fail.

Severity Change Test Finding Location
error added applies a discount Assertion is inside a try/catch that swallows errors … swallowed-assertion test/pricing.test.ts:39
  • It fails the step only for findings in tests the PR adds or modifies (fail-on: error | warning | never). Findings in tests the PR didn't touch are listed, but they don't fail the step.
  • It writes the same report to the job summary. It can also write SARIF (sarif-file) for GitHub code
    scanning.
  • It only parses test files. It never installs your dependencies or runs your code, so it's safe on
    pull requests from forks. Fork PRs get a read-only token, so the comment is skipped there and the
    job summary still has the report.
  • Inputs, outputs and permissions are in the
    README. Code scanning and
    monorepo setups are in examples/.

CLI: SARIF and --changed

npx @mintwork/honest-tests audit . --changed origin/main  # only test files this branch changed
npx @mintwork/honest-tests audit . --sarif > ht.sarif     # SARIF 2.1.0 for code scanning
  • --changed <ref> audits the test files changed since the merge base, including uncommitted and
    untracked files. It reports how many tests the change adds or modifies, and how many of those can't
    fail.
  • --sarif reports findings, and surviving mutants when mutation runs, as SARIF results.

3 new rules (16 in total)

These rules are tuned to miss cases rather than raise false alarms.

  • reimplemented-algorithm (warning): the expected value repeats the arithmetic on the call's own
    inputs, e.g. expect(total(p, q)).toBe(p * q * 1.2).
  • oracle-copied-from-output (warning): the expected value is a float with rounding noise, e.g.
    2.9000000000000004, so it was copied from the code's output.
  • weak-called-assertion (info): a bare toHaveBeenCalled() on a shared mock whose arguments other
    tests check. It's a warning when that's the only kind of check in the test.

Faster mutation testing

  • Parallel mutants: each worker runs in its own project copy. The default is half the CPU cores,
    at most 4 (--concurrency). The fixture's 26 mutants take about 4 s instead of about 10 s. If the
    suite fails when it runs in several copies at once (fixed ports, a shared database), the run falls
    back to one worker, so false kills can't inflate the score.
  • Coverage map: with 2 or more related test files, each mutant runs only the test files that
    execute its line. Mutants that no test executes are reported without a run (--coverage-map auto|on|off).

More runners: Mocha, node:test, Bun and Deno

  • audit understands their test and assertion styles, e.g. t.test subtests, t.assert.*,
    this.skip(), bun:test mocks, Deno.test, t.step and @std/assert.
  • mutate runs the related test files with mocha, node --test, bun test or deno test.

Python: the pytest adapter

audit now checks pytest and unittest files (test_*.py, *_test.py). It understands assert,
pytest.raises, self.assert*, Mock / patch / mocker / monkeypatch and skip markers. It also
catches Python-only bugs:

  • assert (x == 1, "msg"): a tuple is always truthy.
  • assert m.called_once_with(1): this isn't a Mock assertion. It returns a truthy child mock.

Python's own ast module parses the files, and the audited code is never imported or run.

Limitations:

  • The adapter needs python3 3.9 or later on PATH. Without it, Python files are listed as
    unsupported.
  • The 3 new rules are JS/TS-only for now.
  • Mutation testing is JS/TS-only.

Install

npx @mintwork/honest-tests audit .               # CLI, Node 22+
npx @mintwork/honest-tests init claude           # agent skill into .claude/skills/

Full changelog: CHANGELOG.md

honest-tests v0.1.2

Choose a tag to compare

@ranjan-mohanty ranjan-mohanty released this 14 Nov 21:55
57caea5

honest-tests v0.1.2

A maintenance release: no changes to what honest-tests detects or reports.

  • The README on npm now shows the same badges as GitHub (the downloads badge, which showed
    "package not found or too new" for the brand-new package, is gone for now).
  • Published from GitHub Actions with npm trusted publishing (OIDC) and signed provenance; no npm
    token exists for this package.
npx @mintwork/honest-tests audit .

Full feature list: v0.1.0 ·
Changelog

honest-tests v0.1.1

Choose a tag to compare

@ranjan-mohanty ranjan-mohanty released this 14 Nov 20:32
ad48563

honest-tests v0.1.1

New home: @mintwork/honest-tests. This release is the same tool as v0.1.0, published under the
mintwork organization.

npx @mintwork/honest-tests
  • npm: @mintwork/honest-tests. The unscoped honest-tests
    package is deprecated in favour of this one.
  • GitHub: mintwork/honest-tests. Old ranjan-mohanty/honest-tests links redirect.
  • Skill, plugin and editor-integration instructions now use the new package name.

See v0.1.0 for the full feature list.

honest-tests v0.1.0

Choose a tag to compare

@ranjan-mohanty ranjan-mohanty released this 14 Nov 19:04
f023d4f

honest-tests v0.1.0

Your AI wrote 200 tests. Many can't fail.

honest-tests is an agent skill plus a CLI for JS/TS test suites. It finds tests that can't fail, and
then proves which tests are weak by mutating your code and watching the suite stay green. It works
with Claude Code, Codex, Cursor or no agent at all.

demo

npx honest-tests audit .                 # find tests that can't fail
npx honest-tests mutate src/pricing.ts   # break the code on purpose, see which tests notice

Highlights

  • 13 rules for fake and weak tests. The rules catch things like expect(x).toBe(x), tests with
    no assertions, asserting on a mock's configured return value, mocking the module under test,
    try/catch blocks that swallow the AssertionError, assertions that only run inside a catch,
    un-awaited .resolves, toBeTruthy without the (), and leftover .only and .skip.
  • New: asserts-on-own-literal. This rule catches
    const p = { service: 's3' }; expect(p.service).toBe('s3'). Nothing from the code under test is
    involved, so the test can never fail. The pattern is common in runtime "type tests", because
    TypeScript types don't exist at runtime. The rule only fires when the asserted value traces back
    purely to literals declared in the same test, and that binding is never passed to a function,
    mutated or reassigned.
  • Built-in mutation testing. mutate flips operators, booleans, constants, conditions and return
    values. It runs only the tests that reach the file, and lists every change nobody noticed. It works
    on a temporary copy of the project, so the working tree is never touched, even on Ctrl-C.
  • Agent-native.
    • An Agent Skill teaches the rule a test is honest only if it fails when the behavior it names is
      broken
      . It covers a writing workflow (self-audit before saying "done") and a phased audit
      workflow (detect, verify, report, fix).
    • A Claude Code plugin adds /honest-tests:audit and /honest-tests:mutate.
    • honest-tests init codex|cursor writes an AGENTS.md section or a Cursor rule.
  • Fits into CI. It provides --json output (schema v1), --fail-on, --min-score and
    suppression comments that take a reason.

What it found in the wild

  • Open-source projects: across 5 popular OSS JS/TS projects (11,178 tests, all 13
    current rules), 2.6% of tests got an error-level finding, and 0.6% had a high-confidence can't-fail
    bug (never-failing-assertion, catch-only-assertion, tautological-assertion,
    asserts-on-mock or asserts-on-own-literal).
  • 100% line coverage: in one app that enforces 100% line coverage, mutation testing still found
    that 15% of the changes to a formatting module are caught by no test.
  • Precision of the new rule: asserts-on-own-literal was field-checked on 13 OSS JS/TS projects
    (about 27,000 tests). It made 14 hits, and a manual review found every one was an assertion that
    can't fail. There were no false positives. A prototype pass also found 15 hits in one real app.

Install

npx honest-tests audit .                          # CLI, Node 22+
npx honest-tests init claude                      # skill into .claude/skills/
/plugin marketplace add ranjan-mohanty/honest-tests
/plugin install honest-tests@honest-tests

Known limitations

  • JS/TS only (vitest, jest, node:assert, chai). The pytest adapter is next.
  • When mutate picks related tests, it doesn't follow path-alias imports (@/lib/x). Pass
    --test-cmd in that case.
  • Mutants run one at a time. The bundled fixture takes about 12 s for 26 mutants.

Full changelog: CHANGELOG.md