TESTING

Testing

Analysis Date: 2026-07-31

Test framework, structure, and patterns for tests/. Follow these exactly when writing new tests.

---

Framework

• pytest >= 8 ([project.optional-dependencies].dev in pyproject.toml)
• pytest-cov for coverage (no coverage config beyond that)
• jsonschema >= 4 for catalog-schema validation tests
• Config: [tool.pytest.ini_options] → testpaths = ["tests"] in pyproject.toml

---

Running Tests

Setup (once per worktree)

mise trust && mise exec -- uv sync --extra dev

The --extra dev flag is required. Without it, pytest is not installed and uv run pytest silently falls through to a system pytest on a different Python, producing ModuleNotFoundError: No module named 'harnessed' on every test — which looks like a broken checkout rather than a missing extra.

The venv is per-branch: mise.toml sets UV_PROJECT_ENVIRONMENT to ~/.local/share/harnessed/venvs/<branch>/.venv. A fresh worktree starts with no venv; a plain uv sync without --extra dev silently installs the project without pytest.

Fast unit + assembly tests (default; no containers)

mise exec -- uv run pytest -q

This covers all tests that do not require podman. Run before every PR.

Live container tests (podman-gated)

HARNESSED_PODMAN=1 mise exec -- uv run pytest tests/test_recipes_integration.py

Builds each catalog stack and asserts every declared skill/command/plugin/MCP is present in the running container. Gated behind HARNESSED_PODMAN=1 so the default suite remains fast.

---

Two-Tier Test Architecture

Every test file falls into one of two tiers:

1. Fast unit / assembly tests — no containers, no network, no podman. The default pytest run. These cover: schema parsing, emit logic, path resolution, assembly oracle (does the stack assemble without error?), capability oracle (do declared capabilities match what's shipped?), host-launch logic, pin and lint validators, wheel packaging.

2. Live container tests (HARNESSED_PODMAN=1) — build real images and assert capabilities exist in running containers. Only tests/test_recipes_integration.py is in this tier.

---

Test Structure

Tests live exclusively under tests/. There is no src/-adjacent test layout.

tests/
├── conftest.py                         # shared autouse fixtures
├── fixtures/                           # on-disk YAML fixture trees
│   ├── recipes/                        # minimal recipe dirs (low-recipe, npm-recipe, svc-recipe)
│   ├── services/                       # minimal service dirs
│   └── stacks/                         # minimal stack dirs (low-stack, npm-stack, svc-stack)
└── test_*.py                           # one file per module/feature (70+ files)

Use class TestXxx: to group related tests. Standalone tests that do not fit a group are bare def test_…() functions. There is no hard rule; choose whichever is clearer:

# src: tests/test_schema.py
class TestValidateNoRawNpm:
    def test_clean_recipe_passes(self): ...
    def test_npm_command_raises(self): ...
    def test_pnpm_command_passes(self): ...

# standalone — tests/test_capability_tests.py
def test_discover_finds_only_sh_files_sorted(tmp_path): ...
def test_fold_pass(): ...

---

conftest.py: Autouse Fixtures and Module-Level Traps

tests/conftest.py contains two autouse fixtures and one module-level side-effect that must not be changed without understanding the trap it resolves.

FORCE_COLOR must be popped at module level

# tests/conftest.py
# MODULE LEVEL, NOT A FIXTURE — and that is the whole point.
os.environ.pop("FORCE_COLOR", None)

rich reads FORCE_COLOR when a Console is constructed. launcher.py builds _out/_err at module import. An autouse fixture runs AFTER the import and is therefore too late. Terminal shell integration (Ghostty exports FORCE_COLOR=3) sets it without the developer opting in, so ANSI escapes appear inside CliRunner-captured output and any plain-string assertion fails: "no such stack 'x'" is not in "\x1b[1;31merror:\x1b[0m no such stack \x1b[32m'x'\x1b[0m".

Do not move this pop into a fixture. Do not add a parallel pop in a fixture.

_isolated_user_catalog (autouse)

Points $XDG_CONFIG_HOME at an empty tmp_path_factory directory for every test. Without this, the user overlay at ~/.config/harnessed/catalog shadows repo stacks by name, producing machine-dependent results in CI vs. local.

The fixture also symlinks containers/ back into the fake XDG root so rootless podman finds its storage.conf (which declares a non-default graphroot on some machines). Without the symlink, HARNESSED_PODMAN=1 tests would get dial tcp [::1]:443: connect: connection refused instead of using the real image store.

_git_identity (autouse)

Sets GIT_AUTHOR_* and GIT_COMMITTER_* environment variables so that tests creating throwaway git repos and committing (git commit) succeed in CI, where no global ~/.gitconfig identity exists.

---

Fixtures and Test Data

tmp_path and monkeypatch

Use tmp_path (per-test isolated directory) for all filesystem operations. Use monkeypatch for environment variables and function/method patching:

# tests/test_launch_host.py
def test_host_home_is_keyed_by_stack_and_harness_only(self, monkeypatch, tmp_path):
    monkeypatch.setenv("XDG_DATA_HOME", str(tmp_path))
    assert paths.host_home("s", "claude") == tmp_path / "harnessed" / "home" / "s" / "claude"

On-disk fixture trees

tests/fixtures/ holds minimal catalog trees for tests that need real YAML on disk:

tests/fixtures/recipes/low-recipe/recipe.yaml
tests/fixtures/stacks/low-stack/stack.yaml

Use these for tests that load catalog objects via load_recipe() or load_stack() with an explicit root= argument.

Private factory functions

Tests that need multiple object variants use private factory functions (leading underscore) rather than parameterized fixtures. Keep them local to the test module:

# tests/test_schema.py
def _make_recipe(name: str = "test", servers: list | None = None) -> Recipe:
    return Recipe(name=name, servers=servers or [], root=Path("/tmp/fake-recipe"))

# tests/test_install_script.py
def _recipe(tmp_path, name="r", *, install: str | None = None, script_body: str = "true\n"):
    d = tmp_path / name
    d.mkdir(parents=True, exist_ok=True)
    (d / "recipe.yaml").write_text(f"name: {name}\n")
    (d / "install.sh").write_text(script_body)
    return load_recipe(d, strict=True)

---

Mocking

Use monkeypatch.setattr for function and attribute patching. Never import unittest.mock — the test suite does not use MagicMock or @patch.

Patching subprocess/exec calls

When testing code that calls os.execvpe or subprocess, monkeypatch the attribute on the imported module, not on the stdlib directly:

# tests/test_launch_host.py
def test_launch_host_flag_assembles_and_execs_claude(self, monkeypatch, tmp_path):
    captured: dict = {}

    def fake_execvpe(file, argv, env):
        captured.update(file=file, argv=argv, ccd=env.get("CLAUDE_CONFIG_DIR"))
        raise SystemExit(0)  # execvpe replaces the process; halt cleanly instead

    monkeypatch.setattr(launcher.os, "execvpe", fake_execvpe)
    monkeypatch.setattr(launcher.os, "chdir", lambda *_a: None)

Patching launcher internals

Patch module-level functions as attributes of the module object:

monkeypatch.setattr(launcher, "_service_refs", lambda _s: ["beads-server"])
monkeypatch.setattr(launcher, "_runtime", lambda: "podman")
monkeypatch.setattr(launcher, "_ensure_services", lambda rt, stack, **kw: ensured.append((rt, stack)))

---

CLI Tests

Use typer.testing.CliRunner to invoke CLI commands without spawning a subprocess:

# tests/test_launch_host.py
from typer.testing import CliRunner
from harnessed import launcher

runner = CliRunner()

result = runner.invoke(launcher.app, ["host-run", "hostspike", "claude", str(tmp_path)])
assert result.exit_code == 0, result.output

The CliRunner does not allocate a TTY. Combined with the module-level FORCE_COLOR pop in conftest.py, this ensures rich emits plain text and assertions on output substrings work.

If an ANSI assertion fails ("error:" not in result.output when there are escape codes), the environment is wrong — do not strip ANSI in the test. See the conftest.py note above.

---

Exception Testing

with pytest.raises(RecipeLintError, match="pnpm"):
    validate_no_raw_npm(r)

with pytest.raises(SchemaError, match="stack manifest not found"):
    load_stack(tmp_path / "nonexistent")

Always supply match= with a string that identifies the specific error, not just the exception class. This prevents a test from passing on the wrong failure path.

---

Parametrize

Use @pytest.mark.parametrize directly — no alias. Use it at the class method level when the parametrized dimension is a variant of one scenario:

# tests/test_paths.py
@pytest.mark.parametrize("bad", ["..", "beads/..", "../beads", "beads//stealth", "/beads", "beads/", ""])
def test_rejects_path_traversal(self, bad): ...

# tests/test_catalog_json_schemas.py
@pytest.mark.parametrize("kind", sorted(_KINDS))
def test_schema_is_well_formed(kind): ...

Use pytest.param(…, id=…) to give human-readable IDs when parametrizing with complex objects:

pytest.param(kind, f, id=f"{kind}:{f.parent.name}")

---

Source Inspection Tests

Some tests assert on the source code of a function using inspect.getsource(). Use this pattern when the behavior is architectural (e.g., "the scan must mount the volume args") and cannot be tested by calling the function:

# tests/test_emit.py
def test_the_credentialed_post_build_scan_covers_the_volumes_instead(self):
    import inspect
    from harnessed import launcher

    src = inspect.getsource(launcher._build_stack)
    assert "_scan_image_in_container" in src, "build no longer scans at all"
    assert "extra_args=vol_args" in src

Use sparingly — prefer a behavioral test when one is practical.

---

Anchoring to the Repo Root

Tests that need to locate catalog entries or fixture files anchor to the repo root relative to __file__, not to os.getcwd():

ROOT = Path(__file__).resolve().parents[1]   # repo root (tests/ → root)
CATALOG = Path(__file__).resolve().parents[1] / "catalog" / "recipes"

---

Optional Test Dependencies

Use pytest.importorskip when a test depends on an optional package that may not be installed:

# tests/test_catalog_json_schemas.py
jsonschema = pytest.importorskip("jsonschema")
from jsonschema import Draft202012Validator  # noqa: E402

---

The Capability-Test Oracle

tests/test_recipes_integration.py runs two layers:

1. Assembly oracle (fast, no podman): every catalog stack resolves and assembles, and expected_capabilities(stk, recipes) returns a non-empty set. Pin validation is exercised via the floating-recipe fixture Dockerfile.

2. Live container check (HARNESSED_PODMAN=1): harnessed build + harnessed test per stack; asserts every declared capability is present in the right location in the container.

Stacks in NO_CAPABILITY_ORACLE (CLI-only, no skill/command/mcp surface) are excluded from the oracle sweep and covered manually. Stacks in NO_LIVE_CONNECT assemble but point at placeholder URLs and skip the live connect test.

When you add a new recipe that delivers capabilities via its Dockerfile, declare them in expect: in recipe.yaml so the oracle knows what to probe. Adding the stack to the catalog means it is covered automatically by the parametrized sweep.

---

Do Not Fix Color Assertion Failures by Patching Assertions

If a test fails with:

assert "no such stack 'x'" in "\x1b[1;31merror:\x1b[0m no such stack \x1b[32m'x'\x1b[0m"

The environment is wrong — FORCE_COLOR was set before the module-level pop in conftest.py had a chance to run, or a new conftest was added that constructs a Console early. Do not strip ANSI in the assertion. Fix the environment setup so plain text is emitted under CliRunner.