diffcone 0.4.0

Upgrading from 0.3.0: make your execution-evidence recordings again. The
store format is now 6 (0.3.0 wrote 1): --evidence auto skips an older
recording, naming one is an error, and the run action plans from the
code until record makes a new one. Expect plans to select more tests
than 0.3.0 did: most of the changes below close cases where diffcone
missed an affected test. The JSON report keeps schema version 3 and gains
fields: always_run and analysis.counts.always_run;
environment_checked and preloaded_modules in analysis.evidence;
rootdir, configfile, junit_prefix and run_arguments in pytest's
discovery[].config; and evidence_not_used in the plan run -o writes.
check --format json gains mapping.rootdir and mapping.junit_prefix.
New discovery notes: unparsable_config (discovery incomplete),
unreadable_plugins and plugin_outside_roots.

Planning

• diffcone.toml takes [[always_run]] entries: the tests (or
  benchmarks) whose id matches a pattern are selected in every plan, with
  or without a recording, and the report gives the entry as the reason and
  counts the targets each entry matched; the text report flags an entry
  that matches nothing. A malformed entry, or one naming an unknown
  runner, is an analysis error, so the plan selects everything rather than
  quietly leaving those tests out. diffcone report computes a job's
  selected share without them.
• A test is selected when the fixtures, hooks or plugins around it change
  though its own code did not (rule lifecycle_changed), for example when
  another test module adds a plugin to pytest_plugins; with a recording,
  compared with what ran around it when it was recorded. pytest plugin
  modules are dependencies of every test, since their import-time code
  runs for the whole session.
• diffcone now assumes a test does not depend on what an earlier test left
  behind. A change to module state made by code that runs only inside one
  test counts for that test only, not for the other tests and benchmarks
  reading the same state: the test itself, a function-scoped fixture,
  unittest's setUp/tearDown, and a helper in a test module or
  conftest that only such code calls. A patch undone when the test ends
  counts the same way, also for lookups on a library module it patches:
  monkeypatch and mocker methods on pytest's fixtures, a with mock.patch(...) block, a @mock.patch(...) decorator,
  MonkeyPatch.context(), and a test or fixture that saves a module
  attribute and puts it back. Still counted for every reader: a test
  module's top-level code, wider-scoped fixtures, hooks, all project code
  (a cache a test fills through it), a doctest's function, a manifest
  entry on project code, a test or setUp that module-level code, a
  wider-scoped fixture or a hook calls, and a store through a
  monkeypatch that a wider-scoped fixture requests or the test module
  overrides (one overridden in a parent conftest is still taken as
  undone). If a test relies on an earlier test's state,
  declare the dependency in diffcone.toml.
• A fixture or function that only empties or fills a module-level dict,
  list or set (REG.clear()), or resets another module's variable
  (mod.X = None), does not count as reading it: an autouse reset fixture
  no longer selects every test when the code filling the registry
  changes.
• An attribute read on an object of unknown type is matched to
  module-level functions, classes and variables only in modules such an
  object can be: modules passed around or bound to another name, obtained
  by name or named in a string, and test modules and conftests, which
  pytest hands out. A name bound only to modules (m = ops, then
  m.value()) is followed like the import itself. Methods still match
  whatever the receiver.
• A text a test writes or runs (pytester.makepyfile, exec, -c code)
  whose template field the test's own parameter fills, when
  @pytest.mark.parametrize on the test gives that parameter literal
  values, is read once per value instead of counting as text built at run
  time, which any change reached. A value that is not a literal, an
  indirect parameter, another decorator on the test, or a parametrize
  on its class or in pytestmark keeps the old rule.
• Indexing a very long function is no longer quadratic in its calls.
• Editing the docstring of a method under @classmethod, @staticmethod
  or @property no longer counts as code run at import (which selected
  every test importing the module): these builtins only copy the
  docstring. Code that reads the docstring is still selected, and so is a
  project decorator with one of those names.

Fixed

Static analysis

• A change to a dependency, build or CI file outside the source roots
  selected nothing when the roots don't hold the repository root. These
  now select every target: lock files (uv.lock, poetry.lock, any
  *.lock, pylock.toml, conda-lock.yml), requirements* files (also
  dev-requirements.txt, Requirements.txt, requirements.pip),
  environment* files, a .txt/.yml/.yaml file in a directory named
  deps or dependencies, a workspace member's pyproject.toml,
  uv.toml, hatch.toml, poetry.toml, pdm.toml, pytest.toml,
  .env and .pth files, sitecustomize.py, noxfile.py,
  .github/workflows/, checker configurations that pytest plugins run as
  tests (mypy.ini, ruff.toml), compiled sources, anything under a
  *.dist-info, *.egg-info or *.egg directory, a build script
  anywhere under a root (a nested setup.py), and a lock-file change
  between INDEX and WORKTREE. Evidence mode treats them the same way.
• A name taken from a dict, list or set written in the code
  (getattr(handlers, NAMES[key]), a lazy-export table) is bounded by it
  only while every use of the table is a read: a table changed through an
  alias, a helper it is passed to, its module (core.TABLE[k] = v),
  globals(), vars(), sys.modules, monkeypatch or mock.patch (an
  object or a dotted string) is no longer read as its literal. A
  parameter or class-body name that shares its name with such a table is
  no longer bounded by it.
• A lookup on a standard-library or third-party module
  (getattr(logging, name)) counts writes that reach the module
  indirectly: an alias, a helper it is passed to, a loop, __setattr__,
  vars(), a module found at run time, a submodule handed on
  (register(os.path)). Handing a library object such as
  datetime.timezone.utc to a call does not make every lookup on that
  library depend on the caller.
• Modules whose name is known only at run time: once code can reach one,
  no literal table is bounded. A name that starts with fixed text
  (import_module(f"plugins.{name}")) stands for the modules with that
  prefix; a new types.ModuleType is no project module unless it is
  installed in sys.modules under one's name; exec(code, namespace)
  does not reach the calling module; m.__dict__["X"] = v changes only
  X. runpy.run_module(name), runpy.run_path(path) and python -m <name> with a name nothing bounds count as an import of any module, as
  importlib.import_module(name) does.
• An object put in sys.modules (assignment, setdefault, update,
  monkeypatch.setitem, mock.patch.dict) was ignored. It is now the
  module of that name: lookups on that module see it, and while it stays
  installed, the module's importers depend on the code that installed it.
  Without a recording, code that installs an object under a name computed
  at run time makes every test importing a project module depend on it.
• A change to a module-level variable made through a module object the
  analysis did not resolve to the variable's module is a dependency of
  the variable's readers: importlib.import_module("pkg.reg").TABLE["x"] = f, sys.modules[...], a function-local import, a star import, or a
  module passed to a helper that writes into it. A function's
  __globals__, a frame's f_globals, pickle.loads,
  pkgutil.resolve_name, pydoc.locate, gc.get_objects() and
  importlib.util.module_from_spec copies are recognised as ways to
  change a module's tables, and a lookup by a computed name on a module
  sees objects other code puts on that module. Star-imported names
  (from sys import *, then modules[...]) are seen too.
• State changed through an argument or a receiver (register(REGISTRY),
  registry.add(x)) did not reach its readers, in either mode; nor did a
  change to code that may call such a writer differently, or a write
  through a module attribute and an item (store._CACHE["k"].append(x),
  del store._CACHE[k]). A method called on an instance attribute
  (self._client.get()) counts as that class's method only when every
  assignment of the attribute creates an instance of one class (or of a
  third-party class), and a module-level object made by a third-party
  call is changed only by methods of classes deriving from a third-party
  class; calls on such attributes are still matched by name. A function
  a module-level variable's initialiser calls runs at import, so a change
  to it reaches the module's importers.
• Changing the arguments of a call made at import time (X = set_mode("slow"), a class attribute, a decorator's or default's
  argument) did not select the tests that read the state the callee
  changes. An import added to or removed from a module that starts or
  stops running another module's import-time code reaches the readers of
  what that code writes (a registry filled at import), in both modes;
  added imports stay non-structural otherwise.
• Moving a module's or a class's statements around (a constant moved
  below the decorator that reads it, two @register functions swapped, a
  def moved below a try/except import of the same name) was no
  change at all: the order of statements both sides have is now...

diffcone 0.3.0

CI

• diffcone report --dir DIR reports on many CI runs of the actions: per
  job, the share pull requests selected and why some were planned from the
  code, and for pushes whether each was checked, its new failures, flaky
  re-runs and misses. It exits 1 on a miss or a check that failed.
• A new report action runs it on a schedule over the recent runs'
  artifacts, comments the report on an issue, and opens an issue for each
  miss.
• run and record write context.json (what they did, and why a plan
  was made from the code or a push was not checked) into the results they
  upload; run also writes ran.json, the plan that ran.
• The verdict record keeps for a re-run of the same commit
  (.diffcone/check.json) also says whether the push was checked and
  whether the check itself failed, so a re-run reports a failed check as
  one rather than as a miss.

Execution evidence

• On Windows, uv python list run through the path shutil.which gives
  (...\uv.EXE) is recognised as an inert query, as it is elsewhere; it
  made a third of a Windows suite always selected.

diffcone 0.2.0

Platforms

• Windows is tested in CI (Python 3.11-3.14): commands are split without
  POSIX escapes, the recorder plugin is copied where a symlink is not
  allowed, the recorder reads Windows command lines and existence checks and
  matches short (8.3) path names, and the actions find Python and the
  venv's Scripts directory.

CI

• record takes check: true for a default branch whose every push runs
  the full suite: it plans the push's own change from the previous
  recording, records, and fails the job on a new failure the plan did not
  select that fails again when re-run (a flaky one is only reported).
  fail-on-test-failure: true fails the job on failing tests, as a plain
  test run would. Pull requests then run only the selection.

Planning

• A dynamic import through a literal table of tuples is bounded by the
  table: a PEP 562 lazy-export __getattr__ (target = _LAZY.get(name), then import_module(target[0])) no longer reaches every
  module. D.get(key) reads a literal dict like D[key].
• A lookup by a run-time name on a third-party or standard-library module
  (getattr(logging, level), dir(builtins)) no longer selects every test
  that reaches it whenever anything in its import closure changes, unless
  project code stores something on that module.

Execution evidence

• A test that runs a python -c snippet that can't run project code (an
  interpreter version check importing only built-in modules, with nothing
  after the code and no PYTHONPATH-like environment) or uv python list
  is no longer always selected.
• Editing a module that looks at its own file at import
  (Path(__file__).resolve()) no longer replans all its importers from
  the code: that stat records only that the file exists. Any other stat of
  a source file during a test counts as reading it.
• A test that leaves a thread running is no longer always selected;
  instead every test is credited with the code background threads are in
  the middle of while it runs, which also covers a later test running
  beside a thread that loops in one function.

Discovery

• Test files and directories named after -- replace testpaths, as in
  pytest; a node id, or a path after an option that may take it as its
  value, is reported instead of guessed.
• --ignore and --ignore-glob (in addopts or after --) are honoured,
  below the paths pytest starts from, as pytest applies them:
  ignored tests are no longer targets, which run reported as selected
  but not collected.
• Importing a library's test helper whose name looks like a test class
  (from fastapi.testclient import TestClient, aiohttp's TestServer) no
  longer reports an incomplete target list.
• A conftest pytest does not load (a sibling package's tests, outside
  testpaths) is no longer read or reported.

diffcone 0.1.0

The first release.

Planning

• diffcone plan compares two snapshots (git revisions, the staged
  INDEX, or the WORKTREE) and reports which tests and benchmarks a
  change can affect, each with the dependency path or the fallback rule
  that selected it. Both revisions are analysed; analysis never runs
  project code. JSON (schema_version 3) and text reports; exit codes
  distinguish a complete plan (0), a degraded one that selects everything
  (1), no plan (2) and a target list that may be short of what the runner
  collects (3).
• Static discovery of pytest tests (configuration, collection rules,
  inheritance, star-imported suites, the fixture chain with fixtures bound
  by alias or import, fixtures and marks inherited from base classes in
  other modules, marks stored in variables, parametrized names supplied to
  fixtures, plugins declared by plugins, and overrides of fixtures that
  pytest and installed plugins request, tests bound by assignment,
  runTest, pytest 9's pytest.toml, doctests with their fixtures,
  session-wide hooks from any conftest, the usefixtures option, plugins
  of sibling packages) and ASV benchmarks (benchmark_name, imported
  setup, the code a timeraw_ benchmark runs), without importing
  project code; what it cannot see it reports. A conftest outside the
  source roots keeps its tests always selected, and a changed runner
  configuration or build file outside them selects everything.
  diffcone discover writes the targets as a manifest.
• Dependencies the analysis cannot see can be declared in diffcone.toml.
• A target depends on everything it depended on at the base too, and on
  its module's import even when a hand-written manifest does not list the
  module; an added import selects the code that now resolves through it;
  a docstring edit selects only where code runs or reads the docstring.
  Unknown manifest keys are errors.
• Indexes, discovery and per-module results are cached under
  .diffcone/cache/; a cached plan is identical to an uncached one.
  diffcone prune --keep REV shrinks the cache to what planning at given
  commits reads.

Running and checking

• diffcone run plans and runs only the selected targets with pytest or
  ASV; diffcone validate runs the whole suite at both snapshots and
  checks the plan against outcome changes (and, with --coverage, against
  what each test executed); diffcone corpus does so over a commit range.
• diffcone check compares a plan with the JUnit XML of a full run, and of
  other selective runs (pytest-testmon's, say), reporting every failure the
  plan missed; Markdown output for CI job summaries.

Execution evidence (opt-in, Python 3.12+ in the project)

• diffcone collect records what each test executed, at a commit;
  plan --evidence selects the tests whose record meets the change, and
  plans the rest statically or selects everything, saying which.
  run --collect advances the record to the new commit. --env-var
  records project variables that change what tests do. Tests that start a
  subprocess or subinterpreter, or leave a thread running, are always
  selected; a shared fixture computed once by one xdist worker is credited
  to every test using it.
• Cython: with a profile=True build recorded on Python 3.13+, edits to
  function bodies and to the names a file binds outside functions select
  the tests that executed the code concerned.
• A recording settles what static discovery cannot know (whether a plugin
  collects a class pytest's rules skip), so such plans no longer stop at
  exit 3.

CI

• Composite GitHub Actions (actions/record, actions/run,
  actions/check) to record nightly on the default branch and plan, run
  and check each pull request against it; see docs/ci.md.