Repository navigation
CodeWitness 1.0.18
CodeWitness 1.0.18 exercises the one phase it could report without ever running: the Python coverage
measurement, which every machine this project has run on has honestly declined to perform.
What changed
-
The Python coverage phase is now exercised end to end.
PythonCoverageProviderreports coverage as
not measured when the project's interpreter cannot import coverage.py — the right answer, since
coverage.py is not part of the Python distribution and CodeWitness installs nothing. The consequence went
unnoticed: this machine's interpreter cannot import it (ModuleNotFoundError: No module named 'coverage'),
CI installed no Python package beyond the interpreter itself, andfixtures/python-withdrawal-ledger
deliberately declares no dependencies, so the branch that actually runscoverage runandcoverage json
had never executed anywhere. Its only tests substituted their inputs:PythonCoverageProviderTestfed the
parser a JSON file andPythonTimeoutTestfed the killed-run path a sandbox double. A regression that made
the provider always decline to measure would have passed the entire suite, which is exactly the failure the
CI assertion step exists to prevent for every other language. -
CI installs coverage.py and asserts it is importable. The new step runs
python3 -m pip install coverage, and the "Assert every toolchain is present" step now runs
python3 -c 'import coverage; print("coverage.py", coverage.__version__)'beside the assertions for Xdebug,
GHDL,fpcand the rest. pytest is deliberately not installed, so the fixture keeps running on the
standard-libraryunittestrunner the adapter falls back to and that fallback stays what the end-to-end
suite exercises. This is the arrangement the PHP adapter has had since Xdebug was pinned; Python was the
one language whose coverage phase was reported but never measured. -
A new end-to-end test measures a real project through coverage.py.
PythonCoverageEndToEndTest
(@Tag("e2e")) copiesfixtures/python-withdrawal-ledger— a copy, because coverage.py writes a data file
into the directory it runs in and an end-to-end test must not leave.coveragein the fixture it tests —
inspects it withPythonLanguageAdapterso the interpreter probe is the adapter's own, runs the provider,
and asserts a real measurement:measured, toolcoverage.py, a non-zero statement total, a non-zero
covered count, covered ≤ total, a non-empty per-file view whose keys are all Python source files, and the
JSON report the numbers were read from still on disk. On the fixture that is 36 of 37 statements, 97.3%,
per fileledger.pyandtests/test_ledger.py. -
The absent-driver branch is pinned by a unit test, on both of its claims.
PythonCoverageProviderTest
now asserts that a project which recordedhasCoverage=falseis reported as not measured, with tool
coverage.pyand a note naming the reason, and that the numbers stay at zero rather than being invented —
and that nothing is run: the context carries a sandbox whose runner throws, so a provider that started an
interpreter anyway would fail the test rather than cost one. -
Both directions were proved before the change was recorded. Forcing the provider to always decline
made the new end-to-end test fail with
coverage.py is importable, so the phase must measure rather than decline: coverage not measured (the project interpreter cannot import coverage);and with coverage.py absent the test skipped itself
(Tests run: 1, Failures: 0, Errors: 0, Skipped: 1) instead of passing as if it had measured. -
The documentation says which arrangement makes the phase real.
docs/status.mdanddocs/languages.md
now state that CI installs coverage.py and asserts it, that a machine without it skips the end-to-end test
rather than failing, and that Python branch coverage is not measured — the run does not enable--branch,
so the report carries an empty0/0rather than a figure nobody collected.README.mdrecords the same
arrangement next to the per-fixture toolchain notes.
What this does not fix
- CodeWitness still installs nothing. coverage.py stays the environment's responsibility, exactly as
coverage.py's absence is a reported condition rather than a failure. Bundling it would not even work: the
interpreter that is probed is the project's own —.venvandVIRTUAL_ENVare preferred over the ambient
one — and the workspace copy skips any directory namedcoverage, so a vendored copy could neither be
imported by a project's virtual environment nor reach the copy the run works in. - The pytest branch of the coverage run is still unexercised in CI. When the interpreter can import
pytest the suite is re-run undercoverage run -m pytest; CI has no pytest, deliberately, so the exercised
branch is theunittestone. On a machine with coverage.py and pytest both importable that branch runs,
and it is the same three commands with a different module argument, but it has no test of its own. - A developer machine without coverage.py skips the end-to-end test. That is the documented contract for
every toolchain-dependent test in this repository, and it is why CI asserts presence: the skip cannot hide
a regression there.python3 -m pip install --user coverageenables it locally.
Verified for this release
- Full reactor test suite: 1,760 tests, 0 failures, 0 errors, 0 skipped across the 22 test modules (up
from 1,759; the added unit test is the only fast-suite addition, and the end-to-end test is excluded from
the default suite by itse2etag). - The new end-to-end test with coverage.py importable:
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
forPythonCoverage*— the end-to-end test measured the fixture and did not skip. - The same test with the driver absent:
Skipped: 1, not a silent pass. - The self-check reports all seven requirements
VERIFIEDagainst the built 1.0.18 jar.
Compatibility
- No report format, verdict, exit code, CLI, configuration-key or adapter-SPI change. The change is a CI
step, two tests and documentation; a project verified by 1.0.17 is verified identically by 1.0.18.
Process note
The inconsistency was visible in the repository for several releases before it was fixed: the PHP adapter's
Xdebug assertion has a comment saying in as many words that the assertion exists so the coverage phase is
"exercised rather than silently skipped in the report", and the Python adapter had no counterpart. What made
it invisible is that the honest behaviour — reporting "not measured" with a reason — is indistinguishable in
a green build from the phase never being run, so the only signal was that no machine ever had the driver.
Reading the tests rather than the reports is what showed it: both of the provider's tests substituted
their inputs, and both passed for years' worth of runs without the provider executing once.