Skip to content

CodeWitness 1.0.18

Choose a tag to compare

@github-actions github-actions released this 28 Oct 16:27
· 11 commits to main since this release

CodeWitness 1.0.18 exercises the one phase it could report without ever running: the Python coverage
measurement, which every machine this project has run on has honestly declined to perform.

What changed

  • The Python coverage phase is now exercised end to end. PythonCoverageProvider reports coverage as
    not measured when the project's interpreter cannot import coverage.py — the right answer, since
    coverage.py is not part of the Python distribution and CodeWitness installs nothing. The consequence went
    unnoticed: this machine's interpreter cannot import it (ModuleNotFoundError: No module named 'coverage'),
    CI installed no Python package beyond the interpreter itself, and fixtures/python-withdrawal-ledger
    deliberately declares no dependencies, so the branch that actually runs coverage run and coverage json
    had never executed anywhere. Its only tests substituted their inputs: PythonCoverageProviderTest fed the
    parser a JSON file and PythonTimeoutTest fed the killed-run path a sandbox double. A regression that made
    the provider always decline to measure would have passed the entire suite, which is exactly the failure the
    CI assertion step exists to prevent for every other language.

  • CI installs coverage.py and asserts it is importable. The new step runs
    python3 -m pip install coverage, and the "Assert every toolchain is present" step now runs
    python3 -c 'import coverage; print("coverage.py", coverage.__version__)' beside the assertions for Xdebug,
    GHDL, fpc and the rest. pytest is deliberately not installed, so the fixture keeps running on the
    standard-library unittest runner the adapter falls back to and that fallback stays what the end-to-end
    suite exercises. This is the arrangement the PHP adapter has had since Xdebug was pinned; Python was the
    one language whose coverage phase was reported but never measured.

  • A new end-to-end test measures a real project through coverage.py. PythonCoverageEndToEndTest
    (@Tag("e2e")) copies fixtures/python-withdrawal-ledger — a copy, because coverage.py writes a data file
    into the directory it runs in and an end-to-end test must not leave .coverage in the fixture it tests —
    inspects it with PythonLanguageAdapter so the interpreter probe is the adapter's own, runs the provider,
    and asserts a real measurement: measured, tool coverage.py, a non-zero statement total, a non-zero
    covered count, covered ≤ total, a non-empty per-file view whose keys are all Python source files, and the
    JSON report the numbers were read from still on disk. On the fixture that is 36 of 37 statements, 97.3%,
    per file ledger.py and tests/test_ledger.py.

  • The absent-driver branch is pinned by a unit test, on both of its claims. PythonCoverageProviderTest
    now asserts that a project which recorded hasCoverage=false is reported as not measured, with tool
    coverage.py and a note naming the reason, and that the numbers stay at zero rather than being invented —
    and that nothing is run: the context carries a sandbox whose runner throws, so a provider that started an
    interpreter anyway would fail the test rather than cost one.

  • Both directions were proved before the change was recorded. Forcing the provider to always decline
    made the new end-to-end test fail with
    coverage.py is importable, so the phase must measure rather than decline: coverage not measured (the project interpreter cannot import coverage); and with coverage.py absent the test skipped itself
    (Tests run: 1, Failures: 0, Errors: 0, Skipped: 1) instead of passing as if it had measured.

  • The documentation says which arrangement makes the phase real. docs/status.md and docs/languages.md
    now state that CI installs coverage.py and asserts it, that a machine without it skips the end-to-end test
    rather than failing, and that Python branch coverage is not measured — the run does not enable --branch,
    so the report carries an empty 0/0 rather than a figure nobody collected. README.md records the same
    arrangement next to the per-fixture toolchain notes.

What this does not fix

  • CodeWitness still installs nothing. coverage.py stays the environment's responsibility, exactly as
    coverage.py's absence is a reported condition rather than a failure. Bundling it would not even work: the
    interpreter that is probed is the project's own — .venv and VIRTUAL_ENV are preferred over the ambient
    one — and the workspace copy skips any directory named coverage, so a vendored copy could neither be
    imported by a project's virtual environment nor reach the copy the run works in.
  • The pytest branch of the coverage run is still unexercised in CI. When the interpreter can import
    pytest the suite is re-run under coverage run -m pytest; CI has no pytest, deliberately, so the exercised
    branch is the unittest one. On a machine with coverage.py and pytest both importable that branch runs,
    and it is the same three commands with a different module argument, but it has no test of its own.
  • A developer machine without coverage.py skips the end-to-end test. That is the documented contract for
    every toolchain-dependent test in this repository, and it is why CI asserts presence: the skip cannot hide
    a regression there. python3 -m pip install --user coverage enables it locally.

Verified for this release

  • Full reactor test suite: 1,760 tests, 0 failures, 0 errors, 0 skipped across the 22 test modules (up
    from 1,759; the added unit test is the only fast-suite addition, and the end-to-end test is excluded from
    the default suite by its e2e tag).
  • The new end-to-end test with coverage.py importable: Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
    for PythonCoverage* — the end-to-end test measured the fixture and did not skip.
  • The same test with the driver absent: Skipped: 1, not a silent pass.
  • The self-check reports all seven requirements VERIFIED against the built 1.0.18 jar.

Compatibility

  • No report format, verdict, exit code, CLI, configuration-key or adapter-SPI change. The change is a CI
    step, two tests and documentation; a project verified by 1.0.17 is verified identically by 1.0.18.

Process note

The inconsistency was visible in the repository for several releases before it was fixed: the PHP adapter's
Xdebug assertion has a comment saying in as many words that the assertion exists so the coverage phase is
"exercised rather than silently skipped in the report", and the Python adapter had no counterpart. What made
it invisible is that the honest behaviour — reporting "not measured" with a reason — is indistinguishable in
a green build from the phase never being run, so the only signal was that no machine ever had the driver.
Reading the tests rather than the reports is what showed it: both of the provider's tests substituted
their inputs, and both passed for years' worth of runs without the provider executing once.