test: six suites could exit 0 having tested nothing

[JC-000]

Step 1 of the test-suite migration: the six suites that could exit 0 having executed none of the code they test. No new coverage — this changes only what today's runs mean.

Rule applied, from audit commit 7497e48: an involuntary skip is a failure; an explicit skip is allowed but must never be silent. Accounting copied from test_x509.py:219-224,773, test_ecdsa_kat_oracle.py:459-462, test_finished_verify.py:322-326.

Expect currently-green runs to turn red. That is the point — the A/B evidence for every one of the six is below, so newly-red runs are not a regression.

Toolchain for all runs: BACKEND=uci PRG, sha256 7461ab0eb1bc488659fecefe1f828290b7fcfaa55057fc8f90d4b5fbae39946e, built in this worktree after make clean. Harness python /Users/someone/Documents/c64-ChaCha20-Poly1305/.venv/bin/python3.

---

1. test_x509_name.py — missing label returned 0

The whole 23-vector hostname suite — the only coverage x509_verify_hostname has — exited green on any build not exporting the label, i.e. every BACKEND=ip65 build.

How the vacuity was reproduced: not with an ip65 PRG. .incbin resolves against the CWD and this is a worktree three levels down, so an ip65 build here would silently assemble the parent checkout's blob (CLAUDE.md, "Do not build ip65 in a nested git worktree"). Instead the exact trigger condition was reproduced directly: the x509_verify_hostname line was removed from build/labels.txt of the UCI build, which is precisely the state an ip65 build presents to Labels.address().

before

$ grep -v x509_verify_hostname labels-uci.txt > build/labels.txt
$ C64_SKIP_BUILD=1 python3 tools/test_x509_name.py
SKIP: label x509_verify_hostname missing — this is a BACKEND=uci-only feature (see #135); nothing to test on this build.
EXIT=0

after

$ C64_SKIP_BUILD=1 python3 tools/test_x509_name.py
CANNOT RUN: label x509_verify_hostname missing — server name validation is a BACKEND=uci-only feature (see #135). None of the vectors executed; this run certifies nothing.
EXIT=2

Green path unchanged — full VICE run against the unmodified UCI labels:

  17 vectors (8 accept / 9 reject), cert_buf capacity 2048 B
  ... 17 x [PASS] ...
RESULTS: 17/17 passed
EXIT=0

2. test_x509_name.py — oversized vectors left the denominator

A vector whose DER exceeded cert_buf was dropped with a bare continue: declared, no verdict, invisible in RESULTS: n/n passed. Now counted as a failure and listed under "VECTORS THAT COULD NOT RUN", same rule as the KAT oracle's "every declared vector must produce a verdict".

Not triggered on today's vectors (largest live leaf is en.wikipedia.org at 1,636 B against a 2,048 B cert_buf, see the 17/17 run above) — so this is a latent-vacuity fix, and the vectors most likely to trip it are exactly the real CA-issued leaves. Diff:

-            if len(der) > cert_cap:
-                print(f"  SKIP {name}: {len(der)} B exceeds cert_buf")
-                continue
+            if len(der) > cert_cap:
+                print(f"  [-] CANNOT RUN {name}: {len(der)} B exceeds "
+                      f"cert_buf ({cert_cap} B) — counted as a failure")
+                oversize.append((name, len(der)))
+                failed += 1
+                continue

3 & 4. test_tls_record.py / test_tls_handshake.py — "No tests ran", exit 0

Both printed [?] No tests ran (routines not yet implemented?) and then sys.exit(0 if failed == 0 else 1) — zero checks, zero failures, exit 0.

Reaching total == 0 needs every test group to return (0, 0), so both were driven through their own real main() with run_tests returning (0, 0) and only the environment stubbed (make, VICE, wait_for_text). Identical driver before and after; the summary/exit logic exercised is the module's own.

before (test_tls_record)

  Passed: 0/0
  Failed: 0/0

  [?] No tests ran (routines not yet implemented?)
============================================================
DRIVER: test_tls_record.main() exited with 0

after

  Passed: 0/0
  Failed: 0/0

  [-] TLS RECORD LAYER: NO TESTS RAN — every group was skipped
      (routines missing or all groups errored). This run certifies nothing.
============================================================
DRIVER: test_tls_record.main() exited with 1

before (test_tls_handshake)

  [?] No tests ran (routines not yet implemented?)
DRIVER: test_tls_handshake.main() exited with 0

after

  [-] TLS HANDSHAKE: NO TESTS RAN — every group was skipped
      (routines missing or all groups errored). This run certifies nothing.
DRIVER: test_tls_handshake.main() exited with 1

Non-empty runs are untouched, checked both ways through the same driver:

FAKE_PF=17,0  test_tls_record     -> [+] ALL 17 TESTS PASSED        exit 0
FAKE_PF=16,1  test_tls_handshake  -> [-] 1 TEST(S) FAILED           exit 1

tools/run_all_tests.py imports run_tests from these two modules, not main(), so the aggregate is unaffected.

5. test_ecdsa_p384_kat.py --sha-only — fabricated the expected answer

_run_one_vector returned "valid": vec["expected_valid"]. Every vector therefore compared equal to its own expectation, _run_backend tallied it as a pass, and OVERALL: PASS was printed for a run in which ecdsa_verify_384 was never executed.

The P-384 build does not exist (CLAUDE.md, "Known issues"), so the flow was driven against a fake device that answers every DMA and every run_subroutine(), returns the correct host-computed SHA-384 for the splice cross-check (so steps 1-8 succeed exactly as on a working overlay), and records which addresses were executed. _run_one_vector and _run_backend — the code under test — run unmodified.

before

  [vice] rfc6979_p384_sample: running (msg=6 B, expect=VALID)...
      [--sha-only] dual-overlay structural PASS (skipped ecdsa_verify_384)
    PASS [vice] rfc6979_p384_sample: valid=True (expected True) verify=0.000s overall=0.000s carry=$FE

DRIVER: --sha-only tally      : passed=1 failed=0
DRIVER: verify stub executed? : False
DRIVER: OVERALL would be      : PASS
DRIVER: main() would exit     : 0

after

  [vice] rfc6979_p384_sample: running (msg=6 B, expect=VALID)...
      [--sha-only] swap + SHA-384 + splice OK; ecdsa_verify_384 NOT run — no verdict for this vector
    FAIL [vice] rfc6979_p384_sample: --sha-only: ecdsa_verify_384 was never executed, so this vector has no verdict (counted as a failure). The swap/SHA-384/splice steps up to it did pass.

DRIVER: --sha-only tally      : passed=0 failed=1
DRIVER: verify stub executed? : False
DRIVER: OVERALL would be      : FAIL
DRIVER: main() would exit     : 1

--sha-only keeps its diagnostic value (it still reports which of swap / SHA-384 / splice broke, at step granularity) but can no longer report PASS; main() says so explicitly and the usage text now documents it. The full (non---sha-only) path is untouched.

6. test_dns.py / test_http_integration.py — prereq check exited 0

A host with no TAP rig — every macOS dev machine, including this one — printed one SKIP line and exited 0, reporting a green DNS path (4 assertions) and a green end-to-end HTTP path (5 assertions) that had not run.

before (real runs, no stubbing — the prerequisite genuinely is absent here)

$ python3 tools/test_dns.py
SKIP: tap-c64 interface not found
EXIT=0

$ python3 tools/test_http_integration.py
SKIP: tap-c64 interface not found
EXIT=0

after

$ python3 tools/test_dns.py
CANNOT RUN: test_dns.py needs the TAP network rig (tools/rig-up-macos.sh or the Linux tap-c64 setup).
  0 of 4 DNS assertions executed — this run certifies nothing.
  Set C64_NET_TESTS_OPTIONAL=1 to make skipping it a deliberate, exit-0 choice.
EXIT=2

$ python3 tools/test_http_integration.py
CANNOT RUN: test_http_integration.py needs the TAP network rig (tap-c64 + x64sc + dnsmasq).
  0 of 5 HTTP assertions executed — this run certifies nothing.
  Set C64_NET_TESTS_OPTIONAL=1 to make skipping it a deliberate, exit-0 choice.
EXIT=2

Exit 2 is "could not run", deliberately distinct from 1 "tests failed" (the convention test_x509_name.py already uses for its FATALs). The explicit half of the rule is the opt-in:

$ C64_NET_TESTS_OPTIONAL=1 python3 tools/test_dns.py
EXPLICIT SKIP (C64_NET_TESTS_OPTIONAL=1): test_dns.py did NOT run.
  0 of 4 DNS assertions executed; this exit 0 certifies nothing about DNS.
EXIT=0

Neither suite is dispatched by tools/run_all_tests.py.

---

Not fixed, and why (adjacent, out of this step's scope)

test_x509_name.py's real-certificate block prints NOTE: openssl unavailable — real-certificate vectors skipped and silently drops six vectors. On this machine openssl is on PATH: the actual cause is that tools/https_e2e/certs/server.pem does not exist (it is generated by the listener, and only the README is in tree). So the suite quietly runs 17 vectors, not the 23 it is documented as running, and blames the wrong prerequisite. Same class as the six above, but it is a seventh defect and this PR was scoped to six — filing separately rather than widening the diff.

Verification

• pytest at repo root: 31 passed (unchanged).
• py_compile clean on all six files.
• No src/, Makefile, CLAUDE.md or doc changes; tools/test_*.py only.

🤖 Generated with Claude Code