TESTING

Testing

Analysis Date: 2026-08-15

Framework

• pytest >=8, configured in pyproject.toml [tool.pytest.ini_options]
• All dev dependencies declared under [project.optional-dependencies].dev — never rely on a developer's PATH
• pytest-randomly randomises test order every run; every count the suite reports rests on it being deterministic, and order-dependence stays visible this way
• hypothesis >=6 for property-based tests
• pytest-cov for coverage; diff-cover for branch-restricted coverage
• mutmut for mutation testing

Test Structure

tests/
├── conftest.py       # shared fixtures, live-verification accounting, FORCE_COLOR pop
├── support.py        # shared helpers: podman gate decorator, patch_all()
├── fixtures/         # static fixture files used by multiple test modules
├── test_schema.py    # schema validators, parse logic
├── test_launcher_*.py  # launcher sub-systems (build, install, init, scan, …)
├── test_emit.py      # profile assembly output
├── test_paths.py     # path resolution
└── …                 # ~100 test files total, >2200 tests

tests/ is added to pythonpath in pyproject.toml, making support and conftest importable as top-level modules by the suite. Pyright also receives this path via extraPaths = ["tests"].

Running Tests

Always use the wrapper script. Never hand-compose mise/uv/pytest.

tools/run-tests.sh                          # whole suite, quiet
tools/run-tests.sh tests/test_schema.py     # one file
tools/run-tests.sh -k install -x           # filter, stop on first failure

The script exists because three independent traps make the suite fail locally while CI stays green:

1. Per-branch venvs. mise.toml sets UV_PROJECT_ENVIRONMENT to ~/.local/share/harnessed/venvs/<branch>/.venv — outside the repo, unreachable from a fresh worktree. run-tests.sh runs uv sync --extra dev first, which is a no-op when already in sync.
2. --extra dev is mandatory. A plain uv sync omits pytest. uv run pytest then silently falls back to a system pytest on a different Python, where every test errors ModuleNotFoundError: No module named 'harnessed'.
3. mise trust in fresh worktrees. The script runs mise trust (a no-op once trusted) before any command.

Record the baseline test count before making changes. A count drop is a regression even if new tests pass.

A green run is not end-to-end proof. The suite runs no podman build and no harnessed container-run; those require the live layer.

Live / Podman Gate

Tests that exercise real podman or external binaries are gated behind HARNESSED_PODMAN=1. Use the support.podman decorator — do not write the skipif inline:

# tests/support.py
def podman(func):
    """Gate a test on `HARNESSED_PODMAN=1` and mark it as one the gate governs."""
    gated = pytest.mark.skipif(
        not PODMAN_REQUESTED, reason="set HARNESSED_PODMAN=1 for live podman tests"
    )(func)
    return pytest.mark.live_podman(gated)

The live_podman marker is what allows the run's accounting in conftest.py to be honest. When HARNESSED_PODMAN=1 is set but the gate-governed tests still skip (e.g., image not built), conftest.pytest_sessionfinish sets exitstatus = 1 — asking for live verification and silently delivering none is treated as a failure.

HARNESSED_PODMAN=1 tools/run-tests.sh   # run with live podman layer

CI runs the two layers in separate workflows — do not read the hermetic one as "CI never runs podman":

• .github/workflows/test.yml is hermetic by design. No podman on the runner, so every HARNESSED_PODMAN-gated test skips.
• .github/workflows/live.yml is the live layer and does set HARNESSED_PODMAN: "1" (live.yml:154). It runs on push to main, on a nightly 0 4 * * * cron, and on manual dispatch — but deliberately not on pull_request, because podman build minutes on every PR would get the job muted, and a disabled check verifies as little as a skipped one.

The terminal summary always reports how many live tests did NOT run.

Fixtures and Shared Helpers

tests/conftest.py

Provides:

• pytest_configure: registers live and live_podman markers
• pytest_terminal_summary: prints which live tests were skipped and why
• pytest_sessionfinish: fail-closed guard when HARNESSED_PODMAN=1 was set
• catalog_local_restored(checkout): context manager that snapshots and restores catalog-local/ symlinks after a test; lives here (not support.py) because test_live_gate_accounting copies conftest verbatim into a pytester sandbox where support is not importable
• Module-level os.environ.pop("FORCE_COLOR", None): must run before any module that constructs a rich.Console; a fixture runs too late

tests/support.py

# Shared helpers
def podman(func) -> func:          # gate + mark decorator for live tests
def patch_all(monkeypatch, name, value) -> None:  # patch a name in ALL loaded harnessed.* modules

patch_all is the correct way to monkeypatch any helper called from more than one harnessed module. A from .x import y binds y into the importing module's globals; patching only one module leaves the real binding live in every other. patch_all raises if no loaded module binds the name, so a typo or rename fails loudly.

# tests/test_launcher_*.py — typical usage
support.patch_all(monkeypatch, "_host_os", lambda: "darwin")

tests/fixtures/

Static files (YAML manifests, Dockerfiles, config seeds) consumed by multiple test modules.

Test Organisation Pattern

Tests are grouped into classes when they share a subject or setup:

# tests/test_schema.py
class TestValidateNoRawNpm:
    def test_clean_recipe_passes(self):
        r = _make_recipe()
        validate_no_raw_npm(r)  # must not raise

    def test_npm_command_raises(self):
        r = _make_recipe(servers=[McpServer(name="s", command="npm", args=["install"])])
        with pytest.raises(RecipeLintError, match="pnpm"):
            validate_no_raw_npm(r)

    def test_npx_command_raises(self):
        r = _make_recipe(servers=[McpServer(name="s", command="npx", args=["some-pkg"])])
        with pytest.raises(RecipeLintError, match="pnpm dlx"):
            validate_no_raw_npm(r)

Standalone tests (not in a class) are used for single-concern checks that do not benefit from grouping.

Mocking Approach

• monkeypatch (pytest built-in) for attribute and env-var patching
• support.patch_all(monkeypatch, name, value) when a helper is imported across multiple modules
• No dedicated mock library by convention; prefer direct substitution via monkeypatch. One deliberate exception: tests/test_persist_mounts.py uses unittest.mock

tmp_path for Filesystem Isolation

Use the pytest tmp_path fixture to build minimal catalog trees for schema and path tests. The catalog lookup functions accept an explicit root argument precisely to enable this:

# tests/test_schema.py
def test_minimal_stack_loads(self, tmp_path):
    (tmp_path / "stacks" / "mystack").mkdir(parents=True)
    (tmp_path / "stacks" / "mystack" / "stack.yaml").write_text("name: mystack\n")
    stack = load_stack("mystack", root=tmp_path)
    assert stack.name == "mystack"

Mutation Testing

mutmut is configured in pyproject.toml [tool.mutmut]:

paths_to_mutate = ["src/harnessed/"]
tests_dir = ["tests/"]
also_copy = ["tests/", ".github/", "catalog/", "tools/", "schemas/", "mise.toml", "README.md"]

also_copy is wide because many tests assert against repo assets (workflow YAML, catalog files, shell scripts under tools/). Two known limitations:

1. The src/harnessed/catalog symlink does not exist in mutmut's tree — export HARNESSED_DIR=$PWD when running mutmut.
2. mutmut dereferences symlinks, so tests asserting a path IS a symlink fail in the mutants tree — narrow the run to the relevant test files.

Targeted invocation:

HARNESSED_DIR=$PWD mise exec -- uv run --extra dev mutmut run "*<function>*"

Coverage

• pytest-cov collects coverage during the normal test run
• diff-cover restricts the coverage report to lines changed on the current branch — the relevant number for a PR, not the whole-codebase percentage

FORCE_COLOR / ANSI in Assertions

tests/conftest.py pops FORCE_COLOR at module import time (before any rich.Console is constructed). Never "fix" a plain-text-vs-ANSI assertion failure by editing the assertion — it means the environment is wrong. The test must run in an environment where FORCE_COLOR is unset, which run-tests.sh guarantees.

What Is Not Tested by the Suite

• podman build — no container images are built
• harnessed container-run — no end-to-end launches
• These require HARNESSED_PODMAN=1 and a usable podman installation. The hermetic workflow (test.yml) deliberately does not set it; live.yml does, post-merge and nightly — so they are unexercised per-PR, not unexercised outright