Repository navigation
Releases
v0.5.0
Added
Run mutation tests with Bun
The experimental Bun adapter runs ESM JavaScript and TypeScript tests with bun test --isolate. The CLI runs on Node.js 24 or newer and needs Bun 1.4 or newer on PATH. To mutate TypeScript under src/ and discover tests under test/, add these two files at the project root. PR #45
// mutator.config.ts
import { defineConfig } from '@mizchi/mutator';
export default defineConfig({
runner: 'bun',
include: ['src/**/*.ts'],
bunConfig: 'bunfig.toml',
});# bunfig.toml
[test]
root = "./test"Run pnpm exec mutator. For example, tests that call isAdult(age) with 3 and 30 both pass when age >= 18 becomes age > 18. The report marks that mutant Survived. Adding expect(isAdult(18)).toBe(true) makes it Killed.
Select affected tests before the first mutation run
@mizchi/test-impact provides a standalone CLI and Node API. If test/checkout.test.ts imports an edited src/price.ts, the following command lists the checkout test without running it. Use --format json to see why each file was selected.
pnpm add -D @mizchi/test-impact
pnpm exec test-impact --since HEADmutator select-tests --since HEAD now follows static imports when no snapshot exists. With a snapshot, it parses changed scopes and ranks tests using past kills and coverage. The standalone API also accepts per-test Chromium coverage to narrow the selection. Test-impact guide
Fixed
- Diff reports agree on the score. If both mutants in the diff are killed, the terminal and HTML report both show 100%, even when an unchanged file has a cached survivor. JSON marks out-of-selection mutants as
Pending, and the kill matrix counts only selected mutants. PR #29 - Reordering module initialization invalidates cached results. Swapping two route registrations can change which handler runs first. An incremental run now rechecks the tests that load the module, including when the registrations differ only inside inline callbacks. Upgrading rebuilds the old snapshot on the next mutation run. PR #30
- Deleted functions and renamed files keep their test history. If a change removes
discount()and editstax(),select-testscan select tests of both functions. For a renamed module it also looks up tests importing the old path. PR #31 - Broad include globs select source files.
mutator --include 'src/domain/**'accepts nested directories and skips files such asREADME.md. PR #41 - The time budget covers mutant type checking. With
--time-budget 30, the checker consults the deadline between batches. Unchecked mutants remainPendingfor the next run. PR #42 - CLI failure output is easier to use. A failing baseline names the test and shows its first error. When a Vitest test logs to stdout,
mutator --reporter agent > report.jsonstill writes a single JSON document and sends the test output to stderr. PR #43 · PR #39 - Plugin examples import
definePlugin,defineMutator, anddefineIgnorerfrom the public@mizchi/mutatorentry point so they resolve in pnpm projects. PR #38
v0.4.5
Added
- Set
--max-workersormaxWorkersin the config to cap active test workers across a mutator invocation. The baseline can use the full budget. During mutation testing,-j 2 --max-workers 8gives two concurrent Vitest or Playwright sessions up to four workers each. Jest runs each mutant in band.
Changed
- Watch mode now waits one second before its initial run and groups edits into one-second windows. Use
--watch-throttle 5000for five seconds or--watch-throttle 0for immediate scheduling. Continued edits keep the original deadline, and edits during a run queue one follow-up.
v0.4.4
Added
- Use
--watchto rerun mutation tests on edits. In Git it defaults to the working-tree diff againstHEADand includes mutants covered by changed tests.
Fixed
v0.4.3
Added
- Progress for long runs. After the baseline and type check, the CLI shows a progress bar when mutant execution is estimated to take at least 10 seconds or there are at least 100 mutants to run. The bar shows completed mutants, elapsed time and ETA on stderr. Redirected output keeps line-by-line progress, so
--reporter agentstill writes a single JSON document to stdout. - Resume after interruption. Run the same command again after Ctrl-C or a terminated process. The CLI saves each completed verdict to a SQLite checkpoint, rechecks the baseline and type information on restart, and reports how many verdicts it restored. Unfinished mutants run again. Changes to sources, tracked tests or execution settings can invalidate saved verdicts.
v0.4.2
Fixed
- Vitest/Vite+ runtimes installed next to the Vitest config file (
vitestConfig/--config) resolve when the project root has no runner, as in a toolchain isolated in a sibling directory. Since 0.3.0 such projects failed withneither vite-plus/test/node nor vitest/node resolves from <root>. The root still wins when it has a runner, and the setup file'svitestimport resolves from the same place as the runtime.
v0.4.1
Patch fixes for Vitest/Vite+ project coverage, cold include performance and Windows execution. All six npm packages share version 0.4.1.
Fixed
- Windows source include and test-file matching accept both native and Vite path separators (#24, PR #26).
- Vitest inline
test.projectsreceive instrumentation and keep their own setup, test identities and cached project locations (#19). Snapshot compatibility is bumped to discard older project/path verdicts. - Cold explicit
includescopes in Vitest and Vite+ collect only related tests. Computed imports, CommonJS, child processes, workers and unsupported transforms retain a full dry run;--full-dry-runalways collects all tests (#20). - Filesystem transform caching stays disabled in every Vitest project, with regression coverage for alternating include sets (#22).
- Relative test IDs, mutant identities, cache keys and reports consistently use
/on Windows. TypeScript 6/7 virtual file overlays normalize compiler paths; Jest and Playwright terminate server/process trees with Windows APIs (#25). - Installed Vitest/Vite+ runtime paths resolve filesystem aliases and Windows 8.3 paths consistently. Playwright coverage accepts Windows source-map paths and native test locations.
- OXC uses the regular parser on Windows to avoid large raw-transfer reservations exhausting the system commit limit. On other platforms it falls back when the buffer cannot be reserved.
CI
- npm publishing waits for complete Linux and Windows verification, shared with pull requests. Both OSes install the six packed packages outside the workspace in paths containing spaces and exercise Vitest, Vite+, Jest, Playwright, shipped skills and Playwright source coverage. POSIX-only signal/orphan tests remain on Linux.
v0.4.0 — Vite+ and Playwright E2E source coverage
This release brings Vite+ (vp test) compatibility and opt-in Playwright source coverage, alongside runtime estimates and CI sharding. All six npm packages share version 0.4.0; these are the changes since 0.3.0.
Added
Vite+ (vp test)
Projects using vp test can use the existing Vitest adapter. Runner detection recognizes the project's Vite+ installation or Vite config; configuration stays in vite.config.ts and tests import their API from vite-plus/test:
// vite.config.ts
import { defineConfig } from 'vite-plus';
export default defineConfig({
test: { include: ['test/**/*.test.ts'] },
});// test/example.test.ts
import { expect, test } from 'vite-plus/test';Install @mizchi/mutator@0.4.0 in the existing Vite+ project and run pnpm exec mutator. The adapter resolves the project's vite-plus/test/node API before standalone Vitest and injects setup through the matching vite-plus/test API, so another installed Vitest version does not split the runtimes. Integration tests cover Vite+ 1.7.0 alongside standalone Vitest 5.1.0. See the Vite+ setup notes.
Playwright E2E: explicit opt-in and original-source coverage
Browser mutation testing through a Vite dev server is experimental and must now be enabled explicitly. Installing Playwright or adding its config no longer selects it automatically; runner: 'auto' selects Vitest or Jest.
For an existing Playwright + Vite app, install the CLI and adapter:
pnpm add -D @mizchi/mutator@0.4.0 @mizchi/mutator-playwright@0.4.0
pnpm exec playwright install chromiumThe app needs mutatorVite() from @mizchi/mutator-playwright/vite, and the Playwright config must start Vite through webServer. Tests use the adapter's fixture:
import { expect, test } from '@mizchi/mutator-playwright';The setup guide covers the complete wiring. Enable the runner in mutator.config.ts:
import { defineConfig } from '@mizchi/mutator';
export default defineConfig({
runner: 'playwright',
playwrightConfig: 'playwright.config.ts',
include: ['src/**/*.ts', 'src/**/*.tsx'],
concurrency: 2,
timeBudget: 300,
playwrightSourceCoverage: true, // optional; Chromium only, default false
});The optional playwrightSourceCoverage collector maps Chromium V8 execution ranges through inline source maps back to original function scopes. It records which E2E executed each function, including functions with no mutation candidates. The snapshot stores this correspondence for incremental dry runs and mutator select-tests --since main; source-only edits also invalidate mutation verdicts from affected tests. Missing/stale maps, navigations, frames and workers retain conservative candidates. New sources without historical dependencies fall back to all E2Es.
Runtime cost: browser mutation testing can take many minutes or hours on a large app because each mutant starts a Playwright process. The additional V8 collector runs in dry runs only and is disabled by default. In the synthetic large benchmark (16,392 mutants, 32 E2Es; median of three runs), enabling it increased dry-run time from 5.00s to 8.82s with one worker and 3.32s to 4.88s with four workers. The function correspondence index is about 244kB. See the measurements and reproduction commands.
The runnable cart/profile sample includes all configs and a verifier. It checks ordinary E2Es, source correspondence, unchanged cache reuse, source-only edits and agreement between incremental and cold full runs. From a repository checkout:
pnpm install --frozen-lockfile
pnpm --dir examples/playwright-source-coverage exec playwright install chromium
pnpm --dir examples/playwright-source-coverage run verifyThe verified sample kills all 9 mutants initially, reuses all 9 unchanged, then re-tests 5 and reuses 4 after editing a helper without mutants. README and packaged skills link to the sample.
Large runs and CI
- Runtime estimates after the baseline and type checking. Large mutation runs show an ETA calibrated from observed timings and a progress heartbeat every 15 seconds.
--shard INDEX/COUNTpartitions mutant keys deterministically across CI jobs. Each shard keeps independent snapshots and default report directories; scores and quality gates cover that shard.- The GitHub Action accepts a
shardinput that also separates its cache keys.
Changed
- Upgrade note: Playwright users must set
runner: 'playwright'orplaywrightConfiginmutator.config.ts, or pass an explicit--runner playwright/--playwright-configflag. Keep browser specs outside the unit runner's include patterns. Chromium is required only for the optional V8 source collector; the regular Playwright adapter remains available without it. - The snapshot version is bumped: the first run after upgrading rebuilds incompatible cached coverage and verdicts.
- The repository's mutation workflow runs four shards and uploads a report artifact for each, even when a quality gate fails.
- Update Vitest to 5.1.0, Playwright to 1.65.1, Vite to 8.3.8, oxc-parser to 0.159.0, magic-string to 1.4.5, Secretlint to 14.0.1 and Node type definitions to 26.6.6. Secretlint's recommended preset is built in, so the separate deprecated package is removed. TypeScript 6 compatibility tests are retained.
- Releases use newly created full-version tags only. Existing tags remain immutable.
Fixed
- Default runner detection no longer starts Playwright from its config files or installed package. Browser mutation testing is opt-in through an explicit runner or Playwright config option, with setup and runtime-cost guidance.
- Vite+ sessions resolve the project's
vite-plus/test/nodeandvite-plus/testAPIs together, preventing another installed Vitest or a package exposed only throughNODE_PATHfrom being selected. - Mutation sessions disable the persistent module cache so every session populates its own mutant registry, including repeated and parallel runs.
v0.3.0
Added
--include-test-changesextends--sinceto code covered or imported by added, edited or removed tests. Reports include the tests that selected each mutant, and unresolved findings remain visible when the same diff is checked again.--shared-cachereuses validated snapshots across Git worktrees, separated by package, source scope and environment.- JSON and agent reports expose severity-tagged
findings,selectionReasons,testChangesandcomparison. Detection-regression findings identify unchanged code that tests detected before but no longer detect; an unavailable previous snapshot is reported explicitly. --kill-matrixincludes anobservationsrecord for every collected test:detected,noKills,notReachedorincomplete. New tests with no kills and tests whose detection power was not measured are reported separately.- The opt-in
numericMutationsplugin addsMathRoundingandNumericConstantmutants for rounding methods and explicitly configured constants, including negative values. Custom mutator factories can setcacheKeyto invalidate results when captured configuration changes.
Changed
- Type-check verdicts, including
CompileErrorand equivalence decisions, are cached against the source and compiler inputs. Unchanged reruns skip the checker; declaration, configuration, resolution and file-discovery changes invalidate the cache. The exportedTypeCheckercontract now exposes these inputs. - Each include/exclude scope keeps its own snapshot under
.mutator/scopes/, so alternating between scopes preserves their results..mutator/snapshot.jsoncontinues to contain the latest run for existing scripts. - The snapshot version is bumped: the first run after upgrading rebuilds incompatible cached results.
Fixed
- Directory-existence probes track existence separately from entry listings, so parallel work in sibling projects no longer invalidates an unchanged project's type-check results.
- Changes to enclosing workspace manifests and lockfiles invalidate package caches in pnpm/npm monorepos. Workspace discovery stops at the repository boundary.
- Unresolved regressions caused by removed killing tests remain reported on subsequent
--sinceruns instead of disappearing after the first recheck. - Incomplete kill matrices no longer establish that a test detects nothing or is subsumed by another test. Early exits, timeouts and pending work are distinguished from complete measurements.
- Snapshot writes and updates to the latest-run snapshot are atomic, preventing readers from observing partially written JSON.
These changes incorporate feedback from mizchi/mutator を入れてみた (#18).
v0.2.0
- The
mutation-test-loopagent skill ships with@mizchi/mutator, andmutator init-skill [--dir <dir>] [--force]copies it into the project (default.claude/skills). It refuses to overwrite an edited copy unless--force(#12). - The child-process warning lists one line per test file, with the number of its spawning tests and the project files they run, instead of every command (#10).
- Jest runner: a mutant whose failing tests span several test files is now
Killedwith its killing test recorded, instead ofRuntimeErrorwith none. Jest's--bailexited before writing its JSON report; each finished test file is now recorded as it completes. A test file that fails to load is still aRuntimeError(#13). - A mutant run that runs out of time is re-run once with 3x the budget before it is reported as
Timeout, so a machine that is busier than during the dry run no longer turns slow kills and survivors into timeouts (or drops them from--kill-matrix). Runs stopped by the hit limit are not re-run. The summary reports how many runs were re-run (#14). - A GitHub Action (
uses: mizchi/mutator@v0) runs mutator on the pull request's diff and keeps.mutator/in the Actions cache (#11). - A changed test file that the runner's config excludes on purpose (Vitest
exclude) is no longer reported as not collected (#10). - Playwright runner (experimental):
--runner playwrightmutates browser code that Playwright Test drives through a Vite dev server, with@mizchi/mutator-playwright(a Vite plugin and a test fixture). Playwright's artifacts go to mutator's temporary directory, not the project (#15). - Playwright: mutant runs bound
expecttimeouts to at least 1 s (was 2 s), and results come from mutator's own reporter: the fixture app runs in 39.5 s instead of 59.4 s with the same verdicts (#16). - The CLI flushes stdout before it exits: a report larger than a pipe's buffer (64 KB), such as
--reporter agentpiped tojq, was cut off. - Playwright: a
webServerwith neitherurlnorportis reported as such (it was taken for a server already answering); Playwright's artifacts stay out of the project even when its config fails to load. - Agent skills for using mutator, shipped under
skills/(skills-npm convention):mutator(setup, config, reports, CI) in@mizchi/mutatornext tomutation-test-loop, andmutator-playwrightin@mizchi/mutator-playwright.mutator init-skillnow copies every skill the CLI ships, with its reference files. - Critical regions: a critical mutant left unrun (
--time-budget,--mutants-per-line) fails the gate, and on GitHub Actions undetected critical mutants are::errorannotations titledCritical mutant ...(also unreached ones in full runs). - A runner that cannot start (a Playwright config it cannot list, a dev server that fails or a port in use) exits 1 with its message instead of a stack trace.
- Playwright: a test file depends on the local files it imports (helpers, fixtures), so editing one re-collects its coverage.
- A run whose tests reach no mutant at all warns that the setup (include, instrumentation) is likely wrong (
noCoveragein the agent report). It is decided from coverage, so reached mutants left pending by--time-budgetdo not trigger it. - A plugin module that exports nothing usable (a misspelt
visit, a missing name) is a config error saying what is missing. - A mutant reached while a test file loads (or in its
beforeAll) ran only that file's tests, not the tests of other files that reach it while running, so it could be reported Survived although one of those kills it. Both run now; the snapshot version is bumped so such cached verdicts are not reused. - A
RuntimeErroris re-run once and counts only if it fails the same way again: under load, an unhandled error from a flaky test file was credited as a detection of whatever mutant was running (seen as bursts of RuntimeError that the same mutants did not reproduce alone, where they survive). - Ctrl-C and SIGTERM kill every process the run started before exiting (130 / 143). A worker stuck in a mutant's loop used to keep spinning at full CPU after mutator was stopped.
- Playwright tests of this repository no longer depend on the machine's load (the timeout test hangs on purpose; the expect-bound test compares with a passing run).
- After a run killed with SIGKILL, the next run stops the workers it left spinning (recorded under
.mutator/processes/with their start time; only those still orphaned and started when recorded). - Vite+ projects run as Vitest projects: an installed
vite-plusselects Vitest when there is no config file (it worked only through the NODE_PATH that pnpm's bin shims set), and tests may import fromvite-plus/test. - Playwright: a mutant run stopped by the hit limit is a
Timeout(hitLimit); it was reportedKilledby the test that hit it. - Jest:
weakMutation: falsereaches Jest's transformer; the workers still compiled weak-mutation probes. - One classification of mutant runs for every runner. Vitest: a test file whose only load errors come from the hit limit is a
Timeout, not aRuntimeError. Jest: a test failing on the hit limit and on something else isKilled, notTimeout.killedBylists the tests, then the files, without duplicates. - Plugin modules loaded by the Jest transformer and the Playwright Vite plugin get the CLI's checks and messages. The plugin fingerprint changes, so cached results of runs with plugins are run again once.
ConditionalExpressionlooks through<T>xassertions:if (<boolean>(a > b))no longer gets the sametrue/falsemutants twice.rangeskeeps an insertion mutant on a range edge, as--sincedoes.--time-budgetalso stops the remaining parts of a long run split by test file; a run stopped between parts staysPending. Every started run used to finish all its parts, which overran a 600 s budget by more than 30 minutes.