Testenix

Fast tests. Clear results.

Testenix is an experimental native Python testing framework built around five guarantees:

1. tests and fixtures are ordinary typed Python functions;
2. synchronous and asynchronous code use the same execution model;
3. local parallel execution is a core feature rather than a plugin;
4. retries never erase previous failures;
5. every report is derived from one versioned, lossless result model.

The project is currently an alpha. Its native runtime has no third-party dependencies and does not depend on pytest. An optional compatibility bridge can delegate unchanged pytest suites to the pytest installation already present in a project.

Installation

Install the published package with:

python -m pip install testenix

For an existing pytest project, install the optional convenience extra:

python -m pip install "testenix[pytest]"

If a supported pytest (>=8.3,<10) is already installed in the same environment, the base testenix package is sufficient.

Testenix requires Python 3.11 or newer. The project is currently an alpha; pin the version before using it in CI. Use the GitHub installation below only when you intentionally want unreleased changes from the current development branch.

To try the current checkout before publication:

python -m pip install .
# or, for an isolated CLI installation
uv tool install .

You can also install the latest source directly from GitHub:

python -m pip install "testenix @ git+https://github.com/polishdataengineer/testenix.git@main"
# include pytest when the project environment does not already provide it
python -m pip install "testenix[pytest] @ git+https://github.com/polishdataengineer/testenix.git@main"

Run an existing pytest suite

Yes. Testenix provides a transparent compatibility bridge for existing pytest projects:

testenix pytest -q --tb=short tests

Everything after testenix pytest is forwarded unchanged to the same interpreter as python -m pytest. This preserves pytest collection, conftest.py, fixtures, parametrization, markers, assertion rewriting, plugins, configuration, output, node IDs, and exit codes. Consequently, output produced by uv run pytest tests or testenix pytest ... is pytest's UI, not the native Testenix console reporter.

testenix pytest tests/test_api.py::test_health -k smoke --maxfail=1
testenix pytest -n auto tests  # requires pytest-xdist in the same environment
testenix pytest --junitxml=reports/pytest.xml tests

This command is a compatibility and migration bridge. It does not use the native Testenix collector, scheduler, worker pool, retries, history, result model, or reporters. Its performance is therefore pytest performance plus launcher and adapter overhead; the native benchmark results do not apply to it. See the pytest compatibility guide for the complete capability matrix and migration choices.

Convert pytest or unittest tests to native Testenix

The safe migrator creates a new native suite while leaving every original test untouched:

# inspect static support only
testenix migrate auto tests --dry-run

# run the source baseline plus serial and parallel candidates, but publish nothing
testenix migrate auto tests --check --report-json reports/migration-check.json

# validate and atomically create a new directory
testenix migrate pytest tests --output tests_testenix \
  --report-json reports/migration.json
testenix run tests_testenix

For pytest migration, install testenix[pytest]; unittest migration uses the standard library. auto supports pytest and unittest in separate modules within one selection.

Large unsupported suites get a grouped console summary instead of one wall of repeated lines. Use --report-json FILE or --report-json - to retain every individual diagnostic with its source and line. A blocked run labels safe in-memory mappings as a statically convertible subset; only a report with status: published means that Testenix created the requested output.

This is a copy-and-validate transaction, not an in-place rewrite. Testenix fingerprints the sources, generates into private staging, runs the green original suite in a disposable project copy, runs the native candidate with one worker and again in parallel, compares inventory and outcome totals plus every mapped test outcome, rechecks every source hash, and only then performs an atomic no-overwrite publish. Any validation or publication failure before that rename leaves the requested output absent. If only the optional audit-report write fails after a successful rename, Testenix warns without pretending the already published output was rolled back. Report paths must be new, inside the project, and disjoint from both source and generated suites. There is no --force option, and old tests are never deleted or renamed.

The converter stops on semantics it cannot preserve. The pytest subset introduced in v0.2 covers module functions, one static parametrization, simple local/adjacent-conftest fixtures, statically declared autouse fixtures, bare @pytest.mark.asyncio coroutine tests through fresh function-scoped loop wrappers, and simple pytest classes. Native tmp_path and a dependency-free monkeypatch implementation cover the common setattr and setenv forms with automatic per-test rollback, including calls through statically provable module-local helpers. Complex class lifecycle, async fixtures, unmarked async tests, configured async loop scopes or debug mode, custom event_loop_policy, and the rest of pytest's built-in fixtures remain blocked. The unittest adapter preserves per-test lifecycle and assertions by generating native wrappers around the original TestCase.run() protocol; those wrappers locate originals independently of cwd and verify the complete selected-Python-source SHA-256 manifest, so the old unittest files must remain present. Keep the generated unittest directory at its published path as well; rerun migration after moving either side.

See the full safe migration guide for the support matrix, rollback contract, CI rollout, audit-report schema, and performance interpretation for suites with thousands of tests.

Native quick start

Create tests/test_multiplication.py with this complete example:

from collections.abc import AsyncIterator

from testenix import case, cases, fixture, test


@fixture(scope="module")
async def multiplier() -> AsyncIterator[int]:
    yield 2


@test("multiplication uses an async fixture", tags={"unit"})
@cases(
    case(id="positive", value=3, expected=6),
    case(id="zero", value=0, expected=0),
)
async def multiplication(multiplier: int, value: int, expected: int) -> None:
    assert multiplier * value == expected

Run it with:

testenix run tests

The native command uses a compact, file-level report by default and still prints complete failure details and a final summary. A typical failing run looks like this (the run ID and timings vary):

$ testenix run tests
Testenix  |  4 tests  |  2 files  |  2 workers

PASS  tests/test_multiplication.py                  2 passed           [8ms]
FAIL  tests/test_checkout.py                        1 passed, 1 failed [12ms]

Problems (1)
FAIL      tests/test_checkout.py::test_rejects_expired_card
          attempt 1, call: expected status 402, got 200

4 tests, 3 passed, 1 failed in 0.084s

Use -q to hide the header and file table while retaining collection errors, failure details, and the final summary. -v prints one result row per test; -vv also exposes worker, attempt, and phase metadata. --show-skips includes skip and expected-failure reasons, --durations N lists the N slowest tests (--durations 0 lists all), and --color auto|always|never controls ANSI styling.

Plain test_* functions are collected without @test; the decorator is useful for descriptions, tags, and per-test timeouts.

Configuration lives in pyproject.toml:

[tool.testenix]
workers = "auto"
retries = 0
paths = ["tests"]
history = ".testenix/history.sqlite3"
# shard_modules = true
# manifest = ".testenix/collection.json"
# json = "reports/testenix.json"
# junit = "reports/junit.xml"

workers = "auto" is adaptive: after selection, Testenix caps concurrency by the number of independently schedulable units and the machine capacity. With reliable duration history it models worker startup cost and predicted makespan; on a cold run it uses a conservative cap instead of starting one worker for every logical CPU. Use an integer when CI must have a fixed resource limit, or measure the project directly:

# benchmark native candidates; "benchmark" is an alias for "tune"
testenix tune tests --warmups 1 --repeats 5
testenix benchmark tests --candidates 1,2,4,8 --json reports/tuning.json

# write the measured recommendation to [tool.testenix].workers
testenix tune --write

Tuning disables Testenix history for its samples, gives every complete suite run a 300-second deadline by default, and validates that every native candidate keeps the same inventory and outcomes. On timeout it terminates the coordinator boundary and tracked workers; override the deadline with --run-timeout SECONDS. Windows uses a Job Object before process resume. POSIX always signals the new root session and identity-checks observed detached descendants; a process that calls setsid() and exits between creation and the first OS snapshot cannot be given an absolute kernel-containment guarantee, so benchmark hostile suites inside a container. The automatic sweep is resource-aware and conservatively stays within 1/2/4 workers; larger experiments require explicit --candidates. --write refuses to persist a workers-only recommendation if transient --shard-modules, --no-shard-modules, or --manifest settings differ from the loaded project profile, or if the configuration file changes while measurement is running. The writer compares the original bytes again immediately before its atomic replacement; an immutable checkout is still required to exclude an external writer racing that final filesystem operation. Tuning fingerprints project Python/TOML sources (including linked source directories), explicit suite files, and the trusted manifest before the probe and rechecks content plus file identity after every sample; any observed drift discards the result. Installed packages, non-source data, and other runtime dependencies remain external inputs. Its main result is the project-local worker recommendation. An optional --pytest-source PATH comparison is orientation for that exact source/native pair, not a publishable speed claim by itself; use the full benchmark contract below for public comparisons.

Command-line options override this table:

testenix run [PATH ...] [--workers auto|N] [--retries N] [--timeout SECONDS]
                    [--tag TAG ...] [--json FILE] [--junit FILE]
                    [--history FILE | --no-history] [-q | -v | -vv]
                    [--shard-modules] [--manifest FILE]
                    [--color auto|always|never | --no-color]
                    [--show-skips] [--durations N]

Repeated --tag options use AND semantics: a selected test must contain every requested tag. The console report is always printed after execution; it is deterministic output, not a live progress display. JSON preserves the complete run/test/attempt/phase model, JUnit targets CI systems, and SQLite history supplies duration estimates to later runs. History is enabled at .testenix/history.sqlite3 by default; use --no-history for a side-effect-free run.

The same runner is available as a typed library API:

from testenix import TestenixConfig, Status, run


def main() -> None:
    result = run("tests", TestenixConfig(workers="auto", history_path=None))
    failed_ids = [test.test.id for test in result.tests if test.status is not Status.PASS]
    print(failed_ids)


if __name__ == "__main__":
    main()

Async applications can await testenix.run_async(...); cancellation terminates active collection and execution process trees before returning control to the caller.

Optional sharding and trusted collection manifests

Module affinity remains the safe default: ordinary tests from one module stay in one worker, so module fixtures and observable module state are not split across processes. Projects with a large module may explicitly request finer parallelism with --shard-modules or shard_modules = true. Testenix statically rejects splitting when it finds module/session fixtures, module-global or mutable class state, nested mutable containers, imported fixture providers, or import-time lifecycle hazards such as eager calls in assignments, decorators, default arguments, or class bases. Function-scoped fixtures defined in the collected module, including autouse fixtures, can be recreated per test. Imported providers keep module affinity because their source is outside the manifest fingerprint boundary. Static analysis cannot prove the absence of every dynamic side effect, so this mode is opt-in and should first be exercised in CI.

Normal execution imports each selected module during supervised collection and again in its execution worker. A trusted collection manifest removes the collection-side import from subsequent unchanged runs:

testenix manifest tests --output .testenix/collection.json
testenix run tests --manifest .testenix/collection.json

The manifest contains every selected test file plus statically discoverable project-local Python import dependencies, SHA-256 fingerprints, collected tests, collection issues, and sharding decisions. Parameter names are retained, but their values are redacted because collection-time case data can contain credentials or other environment-derived secrets. Before trusting it, Testenix compares the requested roots, exact file inventory, and every source digest. Malformed manifest JSON is rejected; a missing, added, deleted, unreadable, or changed source marks a valid manifest stale and makes the runner fall back to normal isolated collection. It never executes a stale inventory. This is an explicit trust optimization, not an automatic cache: collection affected by environment variables or external state must be regenerated when those inputs change.

Exit codes

Code	Meaning
0	No gating failure. Passing, skipped, xfailed, and cached tests are non-gating.
1	A test failed, errored, timed out, crashed, unexpectedly passed, or was finalized as flaky.
2	Collection, command/configuration error, or an explicit tag filter selected no tests.
3	Internal runner, report, or history error.
4	Migration found an unsupported construct and published nothing.
130	Interrupted by the user.

Result semantics

Testenix preserves the complete hierarchy run -> test -> attempt -> phase. The phases are setup, call, and teardown. A failed attempt followed by a successful retry is reported as FLAKY, not PASS. Setup errors, teardown errors, timeouts, process crashes, expected failures, unexpected passes, and infrastructure failures remain distinct outcomes.

A process crash gets one automatic recovery attempt, but CRASH -> PASS is still FLAKY and gates CI; Testenix cannot prove that an abrupt process exit was unrelated to the test. An unfinished asyncio task is cancelled at the test boundary and fails its owner instead of leaking into the next test.

Where Testenix is deliberately different

The testenix run engine is not a native drop-in reimplementation of pytest. Its v0.3 value is a smaller, coherent native stack: async tests and async fixtures need no plugin, parallel execution and duration-aware scheduling need no xdist, every retry remains visible, and a worker crash cannot silently erase tests that completed before it. The native runtime has no third-party dependencies.

For unchanged pytest semantics, use testenix pytest. That bridge intentionally delegates to pytest instead of silently approximating fixtures, markers, plugins, or hook behavior.

For supported pytest and unittest modules, testenix migrate provides a conservative path to the native engine. Differential validation is part of that command; a syntactically successful rewrite alone is never treated as a completed migration.

The trade-off is ecosystem maturity. pytest currently has much broader plugin, IDE, assertion rewriting, and migration support. Testenix should only claim to be better for the guarantees above until the adoption work in the roadmap is complete and measured on real projects.

Documentation and LLM reference

The complete documentation is published at polishdataengineer.github.io/testenix. Every page can copy its own text or the complete project reference for an LLM.

• llms.txt provides a compact index.
• llms-full.txt contains the guides, public API, architecture, roadmap, and benchmark context in one file.
• Documentation for LLMs explains the source-of-truth and copying workflow.

Benchmarks

The checked-in 3.15x result is a historical Testenix 0.1.0 synthetic baseline, not a Testenix 0.3.0 measurement. On one M4 Pro/CPython 3.11 machine, native testenix run completed 100,000 generated no-op tests across 16 modules in a median 8.04 seconds, compared with 25.33 seconds for pytest and 21.30 seconds for pytest-xdist. The run used four workers, --no-history, pytest-xdist 3.8's default load scheduler, one warm-up, and five counterbalanced measured rounds. It does not describe testenix pytest, which executes through pytest, or promise the same ratio for a real project.

No Testenix 0.3.0 scaling result is checked in yet. The provenance-gated matrix harness covers 100/500/1,000/3,000 tests, balanced/dominant/single-module layouts, 1/2/4/auto workers, and both default history and --no-history, plus explicit safe-module sharding. Until that clean five-round matrix is published, the project does not claim a 0.3.0 speedup. A separate manifest harness can measure a private real project without copying its code, stdout, environment values, or absolute paths into the result. It marks a result publishable only when a current successful migration report proves exact per-test inventory/outcome parity, complete source/generated Python-file inventories and hashes, and binds canonical pytest and Testenix commands to the migrated source/output roots. Without that report, the result is explicitly diagnostic-only. Publishable source roots must be directories so support files such as conftest.py are covered. Manifest commands put all options before -- and their exact suite roots after it, preventing an option value from being mistaken for the measured target. Duplicate performance flags are rejected, imported module files must belong to their claimed distributions, and every timeout performs bounded worker-tree cleanup before the next sample. The same Windows Job Object/POSIX identity-tracking boundary and POSIX setsid() caveat described for testenix tune apply to these harnesses.

The separate safe-migration benchmark used 3,000 tests across 64 modules and four native workers. After conversion, pytest no-op tests ran in 0.521 seconds versus 1.539 seconds through sequential pytest (2.96x faster). The result reverses for empty unittest methods: native wrappers took 1.192 seconds versus 0.161 seconds through a sequential stdlib-based outcome probe (7.40x slower). Adding 1 ms of synthetic work per unittest method changed the medians to 2.577 versus 4.066 seconds (1.58x faster). Migration itself was a one-time 5.94–17.25-second validation-and-publication transaction and is not included in those recurring-run medians. These synthetic comparisons depend on module layout, test duration, and worker count; they do not establish a universal advantage over pytest, pytest-xdist, unittest, or real project suites.

# current-version synthetic matrix; refuses dirty/version-mismatched publication input
uv run --no-editable python benchmarks/run_scaling_matrix.py \
  --output benchmarks/scaling_matrix_0_3_0.json

# redaction-safe real-project comparison driven by an argument-array manifest
cp benchmarks/real_project_manifest.example.json /tmp/testenix-project-benchmark.json
# edit the copied manifest for the target project
uv run --no-editable python benchmarks/run_project_benchmark.py \
  --project /path/to/project \
  --manifest /tmp/testenix-project-benchmark.json \
  --output /tmp/testenix-project-result.json

See the generated results and chart, raw JSON, methodology, and performance analysis.

Current limitations

• Parallel workers are isolated processes. Normal tests from one module stay together, so a module-scoped fixture is not duplicated merely because --workers is greater than one. Optional --shard-modules splitting is fail-closed for statically visible hazards but remains an explicit trust decision for dynamic module behavior.
• Reproducible 1k/10k/100k comparisons, profiler findings, memory measurements, and the Rust/PyO3 decision are documented in the performance analysis. Session-scoped fixtures remain per worker process, not run-global.
• Every test with an explicit or global timeout is hard-isolated in its own process. This makes a blocking synchronous call killable on every supported platform, but module/session fixtures used by timed tests cannot be shared with neighbouring tests.
• Collection imports user modules in a supervised process. A crash becomes a collection error and a hung import has a bounded 30-second deadline instead of taking down the coordinator. An explicitly generated, hash-verified --manifest can bypass that collection import on unchanged sources; stale manifests safely fall back to isolated collection.
• Case values are reconstructed by rediscovering the module in the worker and do not need to be pickle-serializable. They must still be reproducible during module import; reports store a JSON-safe representation when a value itself is not serializable.
• Synchronous test and fixture bodies run outside Testenix's internal asyncio loop. APIs restricted to Python's main thread, such as installing signal handlers, are not supported inside those bodies in v0.3. Migrated pytest-asyncio wrappers are synchronous from Testenix's perspective and therefore share this restriction while creating a fresh event loop for each test or case.
• On every supported platform, an executable script that calls the programmatic run()/run_async() API must use the standard if __name__ == "__main__": multiprocessing guard because Testenix deliberately uses supervised spawn workers. The testenix CLI handles process startup itself.
• The pytest bridge does not translate delegated outcomes into Testenix RunResult, JSON, history, retry, timeout, or scheduling semantics. Use pytest's own flags and installed plugins in that mode.
• Native migration requires a green source baseline and a new output directory. Filesystem changes inside the project are isolated by disposable copies during validation, but network, database, cloud, and other external test side effects are not sandboxed.
• A normal module is one scheduler-affinity unit. Converting 3,000 tests in one module does not create 3,000 parallel units unless the project opts into --shard-modules and its static safety checks pass. Spread independent tests across modules or validate opt-in sharding, then run testenix tune before making a project-specific speed claim.
• Test impact analysis, result caching, remote workers, and deep pytest-result aggregation are not part of version 0.3.

Project status

Testenix 0.3.0 is pre-1.0 software. The distribution, import package, CLI, configuration namespace, and state directory consistently use testenix. The project is licensed under MIT and releases are published to PyPI through Trusted Publishing.

Development

The project targets Python 3.11 and newer.

uv sync --no-editable
uv run --no-editable pytest
uv run --no-editable ruff check .

See the architecture, the roadmap, and the benchmarking contract.