Repository navigation
Releases
Honest Tests Audit v0.2.0
Honest Tests Audit v0.2.0
Find the tests a pull request adds that can't fail, and say so in a PR comment.
honest-tests is an agent skill plus a CLI that finds tests which pass whether or not the code works:
expect(x).toBe(x), tests with no assertions, assertions on a mock's configured return value,
try/catch blocks that swallow the failure, and more. Built-in mutation testing then proves which
tests are weak. This release adds a GitHub Action, Python support and four more JS test runners.
New: the GitHub Action
# .github/workflows/honest-tests.yml
name: Honest tests
on: pull_request
permissions:
contents: read
pull-requests: write # create/update the PR comment
jobs:
audit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0 # the audit diffs against the merge base
- uses: mintwork/honest-tests@v0It audits only the test files the PR changes and keeps one comment up to date:
Honest Tests Audit
This PR adds 12 tests — 4 can't fail.
Severity Change Test Finding Location error added applies a discount Assertion is inside a try/catch that swallows errors … swallowed-assertiontest/pricing.test.ts:39
- It fails the step only for findings in tests the PR adds or modifies (
fail-on: error | warning | never). Findings in tests the PR didn't touch are listed, but they don't fail the step. - It writes the same report to the job summary. It can also write SARIF (
sarif-file) for GitHub code
scanning. - It only parses test files. It never installs your dependencies or runs your code, so it's safe on
pull requests from forks. Fork PRs get a read-only token, so the comment is skipped there and the
job summary still has the report. - Inputs, outputs and permissions are in the
README. Code scanning and
monorepo setups are in examples/.
CLI: SARIF and --changed
npx @mintwork/honest-tests audit . --changed origin/main # only test files this branch changed
npx @mintwork/honest-tests audit . --sarif > ht.sarif # SARIF 2.1.0 for code scanning--changed <ref>audits the test files changed since the merge base, including uncommitted and
untracked files. It reports how many tests the change adds or modifies, and how many of those can't
fail.--sarifreports findings, and surviving mutants when mutation runs, as SARIF results.
3 new rules (16 in total)
These rules are tuned to miss cases rather than raise false alarms.
reimplemented-algorithm(warning): the expected value repeats the arithmetic on the call's own
inputs, e.g.expect(total(p, q)).toBe(p * q * 1.2).oracle-copied-from-output(warning): the expected value is a float with rounding noise, e.g.
2.9000000000000004, so it was copied from the code's output.weak-called-assertion(info): a baretoHaveBeenCalled()on a shared mock whose arguments other
tests check. It's a warning when that's the only kind of check in the test.
Faster mutation testing
- Parallel mutants: each worker runs in its own project copy. The default is half the CPU cores,
at most 4 (--concurrency). The fixture's 26 mutants take about 4 s instead of about 10 s. If the
suite fails when it runs in several copies at once (fixed ports, a shared database), the run falls
back to one worker, so false kills can't inflate the score. - Coverage map: with 2 or more related test files, each mutant runs only the test files that
execute its line. Mutants that no test executes are reported without a run (--coverage-map auto|on|off).
More runners: Mocha, node:test, Bun and Deno
auditunderstands their test and assertion styles, e.g.t.testsubtests,t.assert.*,
this.skip(),bun:testmocks,Deno.test,t.stepand@std/assert.mutateruns the related test files withmocha,node --test,bun testordeno test.
Python: the pytest adapter
audit now checks pytest and unittest files (test_*.py, *_test.py). It understands assert,
pytest.raises, self.assert*, Mock / patch / mocker / monkeypatch and skip markers. It also
catches Python-only bugs:
assert (x == 1, "msg"): a tuple is always truthy.assert m.called_once_with(1): this isn't a Mock assertion. It returns a truthy child mock.
Python's own ast module parses the files, and the audited code is never imported or run.
Limitations:
- The adapter needs
python33.9 or later onPATH. Without it, Python files are listed as
unsupported. - The 3 new rules are JS/TS-only for now.
- Mutation testing is JS/TS-only.
Install
npx @mintwork/honest-tests audit . # CLI, Node 22+
npx @mintwork/honest-tests init claude # agent skill into .claude/skills/Full changelog: CHANGELOG.md
honest-tests v0.1.2
honest-tests v0.1.2
A maintenance release: no changes to what honest-tests detects or reports.
- The README on npm now shows the same badges as GitHub (the downloads badge, which showed
"package not found or too new" for the brand-new package, is gone for now). - Published from GitHub Actions with npm trusted publishing (OIDC) and signed provenance; no npm
token exists for this package.
npx @mintwork/honest-tests audit .honest-tests v0.1.1
honest-tests v0.1.1
New home: @mintwork/honest-tests. This release is the same tool as v0.1.0, published under the
mintwork organization.
npx @mintwork/honest-tests- npm:
@mintwork/honest-tests. The unscopedhonest-tests
package is deprecated in favour of this one. - GitHub: mintwork/honest-tests. Old
ranjan-mohanty/honest-testslinks redirect. - Skill, plugin and editor-integration instructions now use the new package name.
See v0.1.0 for the full feature list.
honest-tests v0.1.0
honest-tests v0.1.0
Your AI wrote 200 tests. Many can't fail.
honest-tests is an agent skill plus a CLI for JS/TS test suites. It finds tests that can't fail, and
then proves which tests are weak by mutating your code and watching the suite stay green. It works
with Claude Code, Codex, Cursor or no agent at all.
npx honest-tests audit . # find tests that can't fail
npx honest-tests mutate src/pricing.ts # break the code on purpose, see which tests noticeHighlights
- 13 rules for fake and weak tests. The rules catch things like
expect(x).toBe(x), tests with
no assertions, asserting on a mock's configured return value, mocking the module under test,
try/catchblocks that swallow theAssertionError, assertions that only run inside acatch,
un-awaited.resolves,toBeTruthywithout the(), and leftover.onlyand.skip. - New:
asserts-on-own-literal. This rule catches
const p = { service: 's3' }; expect(p.service).toBe('s3'). Nothing from the code under test is
involved, so the test can never fail. The pattern is common in runtime "type tests", because
TypeScript types don't exist at runtime. The rule only fires when the asserted value traces back
purely to literals declared in the same test, and that binding is never passed to a function,
mutated or reassigned. - Built-in mutation testing.
mutateflips operators, booleans, constants, conditions and return
values. It runs only the tests that reach the file, and lists every change nobody noticed. It works
on a temporary copy of the project, so the working tree is never touched, even on Ctrl-C. - Agent-native.
- An Agent Skill teaches the rule a test is honest only if it fails when the behavior it names is
broken. It covers a writing workflow (self-audit before saying "done") and a phased audit
workflow (detect, verify, report, fix). - A Claude Code plugin adds
/honest-tests:auditand/honest-tests:mutate. honest-tests init codex|cursorwrites anAGENTS.mdsection or a Cursor rule.
- An Agent Skill teaches the rule a test is honest only if it fails when the behavior it names is
- Fits into CI. It provides
--jsonoutput (schema v1),--fail-on,--min-scoreand
suppression comments that take a reason.
What it found in the wild
- Open-source projects: across 5 popular OSS JS/TS projects (11,178 tests, all 13
current rules), 2.6% of tests got an error-level finding, and 0.6% had a high-confidence can't-fail
bug (never-failing-assertion,catch-only-assertion,tautological-assertion,
asserts-on-mockorasserts-on-own-literal). - 100% line coverage: in one app that enforces 100% line coverage, mutation testing still found
that 15% of the changes to a formatting module are caught by no test. - Precision of the new rule:
asserts-on-own-literalwas field-checked on 13 OSS JS/TS projects
(about 27,000 tests). It made 14 hits, and a manual review found every one was an assertion that
can't fail. There were no false positives. A prototype pass also found 15 hits in one real app.
Install
npx honest-tests audit . # CLI, Node 22+
npx honest-tests init claude # skill into .claude/skills//plugin marketplace add ranjan-mohanty/honest-tests
/plugin install honest-tests@honest-tests
Known limitations
- JS/TS only (vitest, jest, node:assert, chai). The pytest adapter is next.
- When
mutatepicks related tests, it doesn't follow path-alias imports (@/lib/x). Pass
--test-cmdin that case. - Mutants run one at a time. The bundled fixture takes about 12 s for 26 mutants.
Full changelog: CHANGELOG.md

