Skip to content

tia v1.0.0 — Test Impact Analysis for pytest

Choose a tag to compare

@breadMSA breadMSA released this 19 Jun 12:46
· 8 commits to main since this release

tia runs only the tests your change actually affects. Record a per-test
map once; then a git diff selects the minimal set to run.

Read this first: honest disclosure

While benchmarking on Flask, an early run reported a 73% median skip
rate
. It looked great. It was a lie — coverage.py's default core on
Python 3.12+ (sysmon) records only the first test to hit each line and
silently drops the rest, so any test reusing a shared helper went unmapped
and would have been skipped. We forced the C tracer, every test mapped,
and the number fell to the truth: ~21% median skip on Flask (≈40%
mean; cosmetic commits 99%+). Flask is small and tightly coupled — close
to the worst case — and we publish that floor on purpose rather than a
cherry-picked ceiling. Full per-commit table: benchmark/RESULTS.md.

The cardinal sin of test selection is skipping a test that would have
failed. Every design choice here is bent toward never doing that.

What's in 1.0

  • Method-level analysis — maps changes to functions, immune to line shifts.
  • Silent dependencies — tracks non-.py files each test opens (config,
    fixtures, templates) via an open audit hook, so a config.yaml change
    selects exactly its readers — not a silent miss.
  • CI mode — maps bake in their own line→function tables (shallow-clone
    safe) and publish to a shared store; tia serve is a zero-dependency
    HTTP backend. GitHub Actions template included.
  • Dynamic-dispatch safety net — detects reflection (getattr by name,
    eval, importlib, __getattr__) and degrades to file-level there.
  • Cosmetic-change filter — ignores comment/docstring/format-only edits
    via AST comparison, so "fix typo" / "run black" selects nothing.

Honest limitations

Subprocess/other-thread code isn't traced; cross-file dynamic dispatch is
beyond it (undecidable in general — mitigation, not a solution). Run the
full suite on a cadence as a safety net. See the README for the full list.