Skip to content

test: split the monolithic modules and measure what mutation testing actually says - #102

Merged
DLANSAMA merged 3 commits into
mainfrom
refactor/test-decomposition
Aug 5, 2026
Merged

test: split the monolithic modules and measure what mutation testing actually says#102
DLANSAMA merged 3 commits into
mainfrom
refactor/test-decomposition

Conversation

@DLANSAMA

@DLANSAMA DLANSAMA commented Aug 5, 2026

Copy link
Copy Markdown
Owner

Step 4 of 4. Stacked on #101.

The splits

Three modules held a third of the suite:

Was Now
test_printer_commands.py (1333) 7 files by command surface
test_json_contracts.py (1031) 6 files + a shared json_contract_base.py
test_camera_cmd.py (927) 3 files (capture / command / output)

Verbatim moves. Each verified by diffing the collected test-id set before and after — identical in all three cases.

That check turned out to be necessary but not sufficient: it passed while two module-level helpers had been swept into the wrong file, because collection does not run anything. The full suite caught it. Both checks are worth keeping, and the PR keeps both.

Mutation testing: measured, not asserted

You asked me to fix a "high mutation survivor rate". I re-ran the baseline on a clean tree before changing anything, because the floor is CI-enforced and narrowing scope moves the score.

50.7% (killed 1061 / 2091), up from the documented 41.2%. Per-module numbers derived from the mutant sources and survivor list — they reconcile to the 50.7% total, so they are measured, not estimated:

Module Score vs documented
job/payload.py 69.4% flat
netsafety.py 67.3% +9
slicer/options.py 64.2% +2
download/naming.py 61.7% +6
job/predict.py 49.1% +16
download/validation.py 40.3% +10
slicer/output.py 21.8% +0.8

Two findings worth your attention

1. The C.4 prediction did not pan out. docs/mutation-baseline.md claimed the hermetic Orca stub "should score materially higher" for slicer/output.py and asked for a re-run. Re-run: 21.8%, up 0.8 points. Line coverage went 79.8% → 92.7% while the mutation score did not move — the textbook signal that those tests execute _finalize_slice without constraining it. That module now holds 269 of the 1027 survivors (26%). The doc now says this instead of the optimistic prediction.

I did not try to fix it here. The honest options are to extract the pure decision logic out of _finalize_slice, or to stop counting it — both are production refactors, not test changes.

2. A real bug in the mutation runner. scripts/run_mutation_baseline.sh assigned MUTATION_SCORE_FLOOR without exporting it, so the score check — a child python process — never saw it and fell back to its own hardcoded default. A local run printed floor: 40% in its header and enforced a different number a few lines later. CI was unaffected only because the workflow sets the variable at job level.

Now exported, one value, and the child errors out rather than inventing a default.

Floor raised 40 → 48, just under the measured 50.7% — the same discipline already used for the coverage floor. The old 40 was set against 41.2% and had ~10 points of silent-drift room.

New boundary tests

tests/test_download_validation_boundary.py (32 tests) covers the _reject_* functions in download/validation.py — in the mutation scope, but with no direct tests. They were only reached incidentally through whole-command tests, which is exactly why mutants there survived: nothing asserted what they do.

It asserts observable outcomes (exit code, failed_step, machine-readable fields, credential redaction) and deliberately not error-message prose. Pinning message text couples tests to copy-editing; those mutants stay recorded as equivalent.

While writing them I found .exe is not on the refuse-list and text/html deliberately passes (it is the HTML-scrape path). Both are correct — the list names archive/document/image types, not a security allowlist — and are now documented in the tests.

Also

scripts/syntax_smoke.py now discovers scripts/ and tests/ instead of carrying a curated list that named two files this PR deletes — the same stale-parallel-list failure CLAUDE.md warns about. 87 → 157 files compiled.

Gates

1247 passed, 1 deselected     (was 1214 on #101)
coverage 87.1%
mutation 50.7% against the new floor of 48
ruff / ruff format / mypy / bandit / layers / drift-check                green
syntax / help / workflow / package-contents smokes                       green

@DLANSAMA
DLANSAMA force-pushed the refactor/generated-schemas branch from 95f985c to a040d25 Compare August 5, 2026 12:52
@DLANSAMA
DLANSAMA force-pushed the refactor/test-decomposition branch from d050ff6 to 8db48a7 Compare August 5, 2026 13:06
DLANSAMA added a commit that referenced this pull request Aug 5, 2026
1374 collected / 1373 passing, 89.0% Linux — measured on this branch after the
rebase onto main-with-TUI, not inherited from either side of the doc conflict.

These were placeholders until measured: each stacked PR has to stand up green on
its own, and test_docs_consistency rejects a placeholder exactly as it rejects a
stale number. PR #102 re-measures on top.
@DLANSAMA
DLANSAMA force-pushed the refactor/test-decomposition branch from 8db48a7 to efa86ec Compare August 5, 2026 13:27
DLANSAMA added a commit that referenced this pull request Aug 5, 2026
1374 collected / 1373 passing, 89.0% Linux — measured on this branch after the
rebase onto main-with-TUI, not inherited from either side of the doc conflict.

These were placeholders until measured: each stacked PR has to stand up green on
its own, and test_docs_consistency rejects a placeholder exactly as it rejects a
stale number. PR #102 re-measures on top.
@DLANSAMA
DLANSAMA force-pushed the refactor/generated-schemas branch from ca24471 to a6a9bf9 Compare August 5, 2026 14:31
DLANSAMA added a commit that referenced this pull request Aug 5, 2026
…writing them (#101)

* refactor: generate docs/schemas from typed contracts instead of hand-writing them

The 25 files in docs/schemas were hand-maintained, which meant nothing stopped
them drifting from what the commands actually emit. They are now generated from
frozen dataclasses in bambu_cli/contracts/, and CI regenerates and diffs, so
drift is a build failure.

  bambu_cli/contracts/base.py     Contract base + spec() field constraints
  bambu_cli/contracts/models.py   25 published contracts + 8 nested structures
  scripts/gen_schemas.py          the generator (--check is the CI gate)

Fidelity was verified before overwriting anything: the generator was diffed
against the committed schemas until it produced ZERO losses — every constraint
the hand-written files expressed (minLength, minimum, nested required, field
descriptions, the whole status.printer shape) is reproduced. The 78 differences
that remain are all strictly more precise: types alongside consts, item types
on arrays, and explicit nullability.

Two real defects surfaced while doing it, both previously invisible:

  - download.json required 7 fields; a naive model made 4 of them optional.
    Caught by the loss diff, fixed with spec(required=True).
  - error_envelope typed next_command as `{}` (anything), which hid that
    job/send emits it as null. The generated schema is explicit, and the
    contract test failed until the model said so.

Pydantic is a DEV dependency only. It derives JSON Schema from the dataclass
annotations at build time and is never imported at runtime — verified by a test
that subprocess-imports the package and asserts pydantic is absent from
sys.modules, and visible in uv.lock: runtime deps stay at 3, with pydantic under
the test extra behind a python_version >= '3.10' marker. Serialization stays in
emit_json because that pass applies the credential redaction a model_dump_json()
would bypass; emit_json/emit_json_line now accept a contract or a plain dict.

The contracts annotate optionals as `X | None`, which only evaluates on 3.10+.
Nothing at runtime resolves them (dataclasses keeps annotations as strings), so
the 3.9 floor is unaffected — asserted by a test, and the generator refuses to
run below 3.10 with an explanatory message rather than failing obscurely.

Wired so far: version, light, pause, resume, stop. The remaining commands still
emit dicts; every one of them is validated against its generated schema by
tests/contracts/, so the contract holds either way. Converting the rest --
especially job's incrementally-built summary -- is follow-up work.

Docs: schema counts and coverage refreshed to measured values (1215 collected /
1214 passing, 87.0% Linux) rather than left stale.

Gates: 1214 passed (was 1177), coverage 87.0% (was 86.5%), ruff/ruff-format/
mypy/bandit/layers/drift-check/syntax/help/workflow/compat/package smokes green.
Verified on a real CPython 3.9.25 that contracts import, payloads render, and
pydantic stays absent.

* feat: add the Tui output contract now that #97 has landed

docs/schemas/tui.json arrived on main with the TUI. The drift gate did exactly
what it was built to do and failed with "no contract generates: tui.json" —
the check works in both directions, so a published schema with no model behind
it is a failure, not a silent pass.

Same shape as Go: both are human-only front-ends with no machine contract, so
the only payload either emits is the --json refusal. Verified the generated
schema loses nothing from the hand-written one.

* docs: record the measured numbers at this point in the stack

1374 collected / 1373 passing, 89.0% Linux — measured on this branch after the
rebase onto main-with-TUI, not inherited from either side of the doc conflict.

These were placeholders until measured: each stacked PR has to stand up green on
its own, and test_docs_consistency rejects a placeholder exactly as it rejects a
stale number. PR #102 re-measures on top.

---------

Co-authored-by: DLANSAMA <258674612+DLANSAMA@users.noreply.github.com>
Base automatically changed from refactor/generated-schemas to main August 5, 2026 14:40
…actually says

Three modules held a third of the suite and had to be opened to find anything:

  test_printer_commands.py  1333 -> 7 files by command surface
  test_json_contracts.py    1031 -> 6 files + a shared json_contract_base.py
  test_camera_cmd.py         927 -> 3 files (capture / command / output)

Splits are verbatim moves. Each was verified by diffing the collected test-id
set before and after -- identical in all three cases. That check is necessary
but not sufficient: it passed while two module-level helpers had been swept into
the wrong file, because collection does not run anything. The full suite caught
it. Both checks are worth keeping.

Mutation testing: measured rather than asserted.

Re-ran the baseline on a clean tree. Score is 50.7% (killed 1061 / 2091), up
from the documented 41.2%. Per-module numbers derived from the mutant sources
and the survivor list -- they reconcile to the 50.7% total, so they are measured:

  job/payload      69.4%   netsafety        67.3%   slicer/options   64.2%
  download/naming  61.7%   job/predict      49.1%   download/validation 40.3%
  slicer/output    21.8%

Two findings worth the reader's attention:

1. The C.4 prediction in docs/mutation-baseline.md did not pan out, and the file
   now says so. It claimed the hermetic Orca stub "should score materially
   higher" for slicer/output.py and asked for a re-run. Re-run: 21.8%, up 0.8
   points. Line coverage went 79.8% -> 92.7% while the mutation score did not
   move, which is the signal that those tests execute _finalize_slice without
   constraining it. That module now holds 269 of the 1027 survivors.

2. scripts/run_mutation_baseline.sh assigned MUTATION_SCORE_FLOOR without
   exporting it, so the score check -- a child python process -- never saw it and
   used its own hardcoded default. A local run printed "floor: 40%" in its header
   and enforced a different number a few lines below. CI was unaffected only
   because the workflow sets the variable at job level. Now exported, one value,
   and the child errors out rather than inventing a default.

Floor raised 40 -> 48, just under the measured 50.7%, matching the discipline
already used for the coverage floor. The old 40 was set against 41.2% and had
~10 points of silent-drift room.

New tests/test_download_validation_boundary.py (32 tests) covers the _reject_*
functions in download/validation.py, which are in the mutation scope but had no
direct tests -- they were only reached incidentally through whole-command tests,
which is why mutants there survived. It asserts observable outcomes (exit code,
failed_step, machine-readable fields, credential redaction) and deliberately not
error-message prose. That module went 30-31% -> 40.3%.

scripts/syntax_smoke.py now discovers scripts/ and tests/ instead of carrying a
curated list that named two files this commit deletes -- the same stale-parallel-
list failure CLAUDE.md warns about. 87 -> 157 files compiled.

Gates: 1247 passed (was 1214), coverage 87.1%, ruff/ruff-format/mypy/bandit/
layers/drift-check/syntax/help/workflow/package-contents green; mutation 50.7%
against the new floor of 48.
The redaction test wrote a literal user:pass@host URL, which tests/privacy_smoke.py
flags as an email address and a credential-bearing URL — so the test proving
credentials get redacted was itself committing one. All five test legs red;
lint was green, which is why it was not caught by the lint-only checks.

Switch to username-only + IP host, the convention the sibling tests in
test_job.py and test_mqtt_print_and_setup.py already use with the same
explanatory comment. Still exercises the userinfo-stripping path.

My miss: privacy_smoke was run before this file existed and not again after.
1406 collected / 1405 passing, 89.1% Linux coverage — measured after rebasing
onto main with the TUI merged, not carried over from either side of the
conflict. The rebase left placeholders precisely so these would be measured
rather than guessed; test_docs_consistency rejected them until they were.
@DLANSAMA
DLANSAMA force-pushed the refactor/test-decomposition branch from efa86ec to c6e6b56 Compare August 5, 2026 14:41
@DLANSAMA
DLANSAMA marked this pull request as ready for review August 5, 2026 14:50
@DLANSAMA
DLANSAMA merged commit a1f695f into main Aug 5, 2026
6 checks passed
@DLANSAMA
DLANSAMA deleted the refactor/test-decomposition branch August 5, 2026 14:50
DLANSAMA added a commit that referenced this pull request Aug 5, 2026
…#107)

* docs: 0.5.0 truth pass — correct what the docs now get wrong

Audits every tracked doc against the 0.5.0 code and the release commit's own
CI run (31044588411 on 5b08720), and fixes what had gone false or stale.

Actively false, now corrected:
- CONTRIBUTING claimed the four 2026-07 audit gaps were open. All four had
  closed: B.4 cli helper extraction, B.5 single TLS pin helper, the remaining
  JSON schemas (generated now), and the camera bind/pin hardenings. It also
  cited ~82% coverage against a measured 89.
- quality-roadmap said "Phase D schemas largely landed but not complete for
  every command" while docs/api.md and a contract test say the opposite and
  enforce it. Schemas are complete and generated.
- quality-roadmap still framed the TUI as unmerged work on feat/tui that "has
  not reached CI yet". It shipped in 0.5.0 (#97 + #104).
- test-backlog put the mutation floor at 40 (it is 48) and asked for a mutmut
  re-run on slicer/output.py that already happened on 2026-08-04 and disproved
  the prediction it was based on.
- manual's OrcaSlicer detection table listed only the legacy Flatpak app id
  and no Flatpak profile paths at all, contradicting the prose two paragraphs
  above and the real candidate list in config.py.
- docs/plans/interactive-mode-plan.md still read "Draft for implementation"
  and told the reader not to start until 0.4.0 was tagged; plate go shipped in
  0.4.0. Marked implemented, historical text kept as written.

Stale or incomplete:
- Measured numbers refreshed with their source and date: 1419 passing / 1420
  collected, 89.1% branch coverage over 8120 statements on local Linux, and
  the full CI matrix (Windows 88.8% binding, macOS 89.1%, Linux 89.2-89.3%).
  Local and CI figures are cited separately, not reconciled.
- tui/ coverage row was one module behind (#104 added widgets/summary.py):
  now 14 of 18 at 100%, package minimum 95.8%.
- AGENTS module table was missing bambu.py and printer.py; its repo-only list
  was missing docs/releasing.md, docs/README.md and docs/plans/*.
- AGENTS and CONTRIBUTING gate lists omitted pip-audit, gen_schemas --check
  and check_layers, all blocking in the same CI job.
- releasing.md's dev-bump example still said 0.5.0.dev0.
- README told pipx / uv tool users to `pip install 'platecli[tui]'`, which
  installs into the wrong environment; and its agent section did not mention
  that go and tui refuse --json.

No grade was inflated: the scoreboard stays A- / A, coverage 89 against a
target of 92, per-module floors still unenforced, camera residuals still open.

* fix: plate tui — stop the scrollbar clipping which material was detected

On a terminal short enough that the prepare form overflows, #prepare-inputs
grows a vertical scrollbar, and a Textual scrollbar takes real cells rather
than overlaying. The longest radio label is the AMS-detected material —
"PLA — easy, rigid, most models  (detected in AMS)", 49 cells, plus 4 for the
toggle and its padding = 53 — against a column that offered 54. One cell of
slack, which the scrollbar consumed: the label rendered as "(detected in AMS"
with the closing paren shaved off, so the screen silently truncated the one
thing that tells you which filament it found.

Widening the column to 62 leaves 54 cells even while scrolling. Found by
recording the TUI and looking at the frames — every existing assertion passed,
because the label *string* was intact and only its rendering was cut. The new
test asserts the RadioSet has room for its widest button while the scrollbar is
up, and fails when the column goes back to 60.

* chore: untrack the raw hero capture, and ignore raw captures going forward

docs/job-hero.mp4 is the full-length VHS capture of the hero run — 2.3 MB, most
of it the dead air while OrcaSlicer works. The postable cut that scripts/
trim_hero.sh produces from it (docs/job-hero-post.mp4, 427 KB) is already
committed and is what the launch material uses, so the raw file was pure repo
weight and no doc referenced it.

It was never meant to be tracked. It was untracked AND unignored, which the
post-audit gameplan explicitly called out as a trap, and it duly got swept into
#102 — a test-decomposition PR — by a `git add`. Ignoring it closes the trap.

The file stays on disk for re-cutting the video without re-running a real print;
it stays in history, which is fine: frames were extracted and checked and it
leaks no IP, serial, or home path. This only removes it from HEAD.

* docs: record the TUI against a real print, and put each GIF under what it shows

The 0.5.0 headline feature had no imagery at all, and the picture below its
README section was the `plate doctor` GIF — health-check output captioned as if
it illustrated the live dashboard. Meanwhile `plate doctor` is introduced 40
lines earlier with nothing beside it. Both are now under what they actually
show: doctor beside the `plate doctor` line, the new capture under "Watch the
printer while it works".

docs/tui.gif is a real printer mid-print, not a mockup and not --sim: the
dashboard at 13% with nozzle 219.9->220C, bed 60->60C and layer 4/240, then the
job monitor (including its "the print keeps going; nothing here can stop it"
line), then the two-column prepare screen. Frames were extracted and read
before committing — no IP, serial, access code or home path in any of them,
and the shell prompt is bare.

docs/tui.tape records how, and carries the two things that cost time here:
vhs Width/Height are PIXELS, not cells, so the obvious 1100px window is only
~93 columns and silently captures the NARROW single-column prepare layout
rather than the two-column one; and an idle printer records as a static 0%,
so the print has to be running before the tape is rolled.

---------

Co-authored-by: DLANSAMA <258674612+DLANSAMA@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant