test: split the monolithic modules and measure what mutation testing actually says - #102
Merged
Conversation
DLANSAMA
force-pushed
the
refactor/generated-schemas
branch
from
August 5, 2026 12:52
95f985c to
a040d25
Compare
DLANSAMA
force-pushed
the
refactor/test-decomposition
branch
from
August 5, 2026 13:06
d050ff6 to
8db48a7
Compare
DLANSAMA
added a commit
that referenced
this pull request
Aug 5, 2026
1374 collected / 1373 passing, 89.0% Linux — measured on this branch after the rebase onto main-with-TUI, not inherited from either side of the doc conflict. These were placeholders until measured: each stacked PR has to stand up green on its own, and test_docs_consistency rejects a placeholder exactly as it rejects a stale number. PR #102 re-measures on top.
DLANSAMA
force-pushed
the
refactor/test-decomposition
branch
from
August 5, 2026 13:27
8db48a7 to
efa86ec
Compare
DLANSAMA
added a commit
that referenced
this pull request
Aug 5, 2026
1374 collected / 1373 passing, 89.0% Linux — measured on this branch after the rebase onto main-with-TUI, not inherited from either side of the doc conflict. These were placeholders until measured: each stacked PR has to stand up green on its own, and test_docs_consistency rejects a placeholder exactly as it rejects a stale number. PR #102 re-measures on top.
DLANSAMA
force-pushed
the
refactor/generated-schemas
branch
from
August 5, 2026 14:31
ca24471 to
a6a9bf9
Compare
DLANSAMA
added a commit
that referenced
this pull request
Aug 5, 2026
…writing them (#101) * refactor: generate docs/schemas from typed contracts instead of hand-writing them The 25 files in docs/schemas were hand-maintained, which meant nothing stopped them drifting from what the commands actually emit. They are now generated from frozen dataclasses in bambu_cli/contracts/, and CI regenerates and diffs, so drift is a build failure. bambu_cli/contracts/base.py Contract base + spec() field constraints bambu_cli/contracts/models.py 25 published contracts + 8 nested structures scripts/gen_schemas.py the generator (--check is the CI gate) Fidelity was verified before overwriting anything: the generator was diffed against the committed schemas until it produced ZERO losses — every constraint the hand-written files expressed (minLength, minimum, nested required, field descriptions, the whole status.printer shape) is reproduced. The 78 differences that remain are all strictly more precise: types alongside consts, item types on arrays, and explicit nullability. Two real defects surfaced while doing it, both previously invisible: - download.json required 7 fields; a naive model made 4 of them optional. Caught by the loss diff, fixed with spec(required=True). - error_envelope typed next_command as `{}` (anything), which hid that job/send emits it as null. The generated schema is explicit, and the contract test failed until the model said so. Pydantic is a DEV dependency only. It derives JSON Schema from the dataclass annotations at build time and is never imported at runtime — verified by a test that subprocess-imports the package and asserts pydantic is absent from sys.modules, and visible in uv.lock: runtime deps stay at 3, with pydantic under the test extra behind a python_version >= '3.10' marker. Serialization stays in emit_json because that pass applies the credential redaction a model_dump_json() would bypass; emit_json/emit_json_line now accept a contract or a plain dict. The contracts annotate optionals as `X | None`, which only evaluates on 3.10+. Nothing at runtime resolves them (dataclasses keeps annotations as strings), so the 3.9 floor is unaffected — asserted by a test, and the generator refuses to run below 3.10 with an explanatory message rather than failing obscurely. Wired so far: version, light, pause, resume, stop. The remaining commands still emit dicts; every one of them is validated against its generated schema by tests/contracts/, so the contract holds either way. Converting the rest -- especially job's incrementally-built summary -- is follow-up work. Docs: schema counts and coverage refreshed to measured values (1215 collected / 1214 passing, 87.0% Linux) rather than left stale. Gates: 1214 passed (was 1177), coverage 87.0% (was 86.5%), ruff/ruff-format/ mypy/bandit/layers/drift-check/syntax/help/workflow/compat/package smokes green. Verified on a real CPython 3.9.25 that contracts import, payloads render, and pydantic stays absent. * feat: add the Tui output contract now that #97 has landed docs/schemas/tui.json arrived on main with the TUI. The drift gate did exactly what it was built to do and failed with "no contract generates: tui.json" — the check works in both directions, so a published schema with no model behind it is a failure, not a silent pass. Same shape as Go: both are human-only front-ends with no machine contract, so the only payload either emits is the --json refusal. Verified the generated schema loses nothing from the hand-written one. * docs: record the measured numbers at this point in the stack 1374 collected / 1373 passing, 89.0% Linux — measured on this branch after the rebase onto main-with-TUI, not inherited from either side of the doc conflict. These were placeholders until measured: each stacked PR has to stand up green on its own, and test_docs_consistency rejects a placeholder exactly as it rejects a stale number. PR #102 re-measures on top. --------- Co-authored-by: DLANSAMA <258674612+DLANSAMA@users.noreply.github.com>
…actually says Three modules held a third of the suite and had to be opened to find anything: test_printer_commands.py 1333 -> 7 files by command surface test_json_contracts.py 1031 -> 6 files + a shared json_contract_base.py test_camera_cmd.py 927 -> 3 files (capture / command / output) Splits are verbatim moves. Each was verified by diffing the collected test-id set before and after -- identical in all three cases. That check is necessary but not sufficient: it passed while two module-level helpers had been swept into the wrong file, because collection does not run anything. The full suite caught it. Both checks are worth keeping. Mutation testing: measured rather than asserted. Re-ran the baseline on a clean tree. Score is 50.7% (killed 1061 / 2091), up from the documented 41.2%. Per-module numbers derived from the mutant sources and the survivor list -- they reconcile to the 50.7% total, so they are measured: job/payload 69.4% netsafety 67.3% slicer/options 64.2% download/naming 61.7% job/predict 49.1% download/validation 40.3% slicer/output 21.8% Two findings worth the reader's attention: 1. The C.4 prediction in docs/mutation-baseline.md did not pan out, and the file now says so. It claimed the hermetic Orca stub "should score materially higher" for slicer/output.py and asked for a re-run. Re-run: 21.8%, up 0.8 points. Line coverage went 79.8% -> 92.7% while the mutation score did not move, which is the signal that those tests execute _finalize_slice without constraining it. That module now holds 269 of the 1027 survivors. 2. scripts/run_mutation_baseline.sh assigned MUTATION_SCORE_FLOOR without exporting it, so the score check -- a child python process -- never saw it and used its own hardcoded default. A local run printed "floor: 40%" in its header and enforced a different number a few lines below. CI was unaffected only because the workflow sets the variable at job level. Now exported, one value, and the child errors out rather than inventing a default. Floor raised 40 -> 48, just under the measured 50.7%, matching the discipline already used for the coverage floor. The old 40 was set against 41.2% and had ~10 points of silent-drift room. New tests/test_download_validation_boundary.py (32 tests) covers the _reject_* functions in download/validation.py, which are in the mutation scope but had no direct tests -- they were only reached incidentally through whole-command tests, which is why mutants there survived. It asserts observable outcomes (exit code, failed_step, machine-readable fields, credential redaction) and deliberately not error-message prose. That module went 30-31% -> 40.3%. scripts/syntax_smoke.py now discovers scripts/ and tests/ instead of carrying a curated list that named two files this commit deletes -- the same stale-parallel- list failure CLAUDE.md warns about. 87 -> 157 files compiled. Gates: 1247 passed (was 1214), coverage 87.1%, ruff/ruff-format/mypy/bandit/ layers/drift-check/syntax/help/workflow/package-contents green; mutation 50.7% against the new floor of 48.
The redaction test wrote a literal user:pass@host URL, which tests/privacy_smoke.py flags as an email address and a credential-bearing URL — so the test proving credentials get redacted was itself committing one. All five test legs red; lint was green, which is why it was not caught by the lint-only checks. Switch to username-only + IP host, the convention the sibling tests in test_job.py and test_mqtt_print_and_setup.py already use with the same explanatory comment. Still exercises the userinfo-stripping path. My miss: privacy_smoke was run before this file existed and not again after.
1406 collected / 1405 passing, 89.1% Linux coverage — measured after rebasing onto main with the TUI merged, not carried over from either side of the conflict. The rebase left placeholders precisely so these would be measured rather than guessed; test_docs_consistency rejected them until they were.
DLANSAMA
force-pushed
the
refactor/test-decomposition
branch
from
August 5, 2026 14:41
efa86ec to
c6e6b56
Compare
DLANSAMA
marked this pull request as ready for review
August 5, 2026 14:50
DLANSAMA
added a commit
that referenced
this pull request
Aug 5, 2026
…#107) * docs: 0.5.0 truth pass — correct what the docs now get wrong Audits every tracked doc against the 0.5.0 code and the release commit's own CI run (31044588411 on 5b08720), and fixes what had gone false or stale. Actively false, now corrected: - CONTRIBUTING claimed the four 2026-07 audit gaps were open. All four had closed: B.4 cli helper extraction, B.5 single TLS pin helper, the remaining JSON schemas (generated now), and the camera bind/pin hardenings. It also cited ~82% coverage against a measured 89. - quality-roadmap said "Phase D schemas largely landed but not complete for every command" while docs/api.md and a contract test say the opposite and enforce it. Schemas are complete and generated. - quality-roadmap still framed the TUI as unmerged work on feat/tui that "has not reached CI yet". It shipped in 0.5.0 (#97 + #104). - test-backlog put the mutation floor at 40 (it is 48) and asked for a mutmut re-run on slicer/output.py that already happened on 2026-08-04 and disproved the prediction it was based on. - manual's OrcaSlicer detection table listed only the legacy Flatpak app id and no Flatpak profile paths at all, contradicting the prose two paragraphs above and the real candidate list in config.py. - docs/plans/interactive-mode-plan.md still read "Draft for implementation" and told the reader not to start until 0.4.0 was tagged; plate go shipped in 0.4.0. Marked implemented, historical text kept as written. Stale or incomplete: - Measured numbers refreshed with their source and date: 1419 passing / 1420 collected, 89.1% branch coverage over 8120 statements on local Linux, and the full CI matrix (Windows 88.8% binding, macOS 89.1%, Linux 89.2-89.3%). Local and CI figures are cited separately, not reconciled. - tui/ coverage row was one module behind (#104 added widgets/summary.py): now 14 of 18 at 100%, package minimum 95.8%. - AGENTS module table was missing bambu.py and printer.py; its repo-only list was missing docs/releasing.md, docs/README.md and docs/plans/*. - AGENTS and CONTRIBUTING gate lists omitted pip-audit, gen_schemas --check and check_layers, all blocking in the same CI job. - releasing.md's dev-bump example still said 0.5.0.dev0. - README told pipx / uv tool users to `pip install 'platecli[tui]'`, which installs into the wrong environment; and its agent section did not mention that go and tui refuse --json. No grade was inflated: the scoreboard stays A- / A, coverage 89 against a target of 92, per-module floors still unenforced, camera residuals still open. * fix: plate tui — stop the scrollbar clipping which material was detected On a terminal short enough that the prepare form overflows, #prepare-inputs grows a vertical scrollbar, and a Textual scrollbar takes real cells rather than overlaying. The longest radio label is the AMS-detected material — "PLA — easy, rigid, most models (detected in AMS)", 49 cells, plus 4 for the toggle and its padding = 53 — against a column that offered 54. One cell of slack, which the scrollbar consumed: the label rendered as "(detected in AMS" with the closing paren shaved off, so the screen silently truncated the one thing that tells you which filament it found. Widening the column to 62 leaves 54 cells even while scrolling. Found by recording the TUI and looking at the frames — every existing assertion passed, because the label *string* was intact and only its rendering was cut. The new test asserts the RadioSet has room for its widest button while the scrollbar is up, and fails when the column goes back to 60. * chore: untrack the raw hero capture, and ignore raw captures going forward docs/job-hero.mp4 is the full-length VHS capture of the hero run — 2.3 MB, most of it the dead air while OrcaSlicer works. The postable cut that scripts/ trim_hero.sh produces from it (docs/job-hero-post.mp4, 427 KB) is already committed and is what the launch material uses, so the raw file was pure repo weight and no doc referenced it. It was never meant to be tracked. It was untracked AND unignored, which the post-audit gameplan explicitly called out as a trap, and it duly got swept into #102 — a test-decomposition PR — by a `git add`. Ignoring it closes the trap. The file stays on disk for re-cutting the video without re-running a real print; it stays in history, which is fine: frames were extracted and checked and it leaks no IP, serial, or home path. This only removes it from HEAD. * docs: record the TUI against a real print, and put each GIF under what it shows The 0.5.0 headline feature had no imagery at all, and the picture below its README section was the `plate doctor` GIF — health-check output captioned as if it illustrated the live dashboard. Meanwhile `plate doctor` is introduced 40 lines earlier with nothing beside it. Both are now under what they actually show: doctor beside the `plate doctor` line, the new capture under "Watch the printer while it works". docs/tui.gif is a real printer mid-print, not a mockup and not --sim: the dashboard at 13% with nozzle 219.9->220C, bed 60->60C and layer 4/240, then the job monitor (including its "the print keeps going; nothing here can stop it" line), then the two-column prepare screen. Frames were extracted and read before committing — no IP, serial, access code or home path in any of them, and the shell prompt is bare. docs/tui.tape records how, and carries the two things that cost time here: vhs Width/Height are PIXELS, not cells, so the obvious 1100px window is only ~93 columns and silently captures the NARROW single-column prepare layout rather than the two-column one; and an idle printer records as a static 0%, so the print has to be running before the tape is rolled. --------- Co-authored-by: DLANSAMA <258674612+DLANSAMA@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 4 of 4. Stacked on #101.
The splits
Three modules held a third of the suite:
test_printer_commands.py(1333)test_json_contracts.py(1031)json_contract_base.pytest_camera_cmd.py(927)Verbatim moves. Each verified by diffing the collected test-id set before and after — identical in all three cases.
That check turned out to be necessary but not sufficient: it passed while two module-level helpers had been swept into the wrong file, because collection does not run anything. The full suite caught it. Both checks are worth keeping, and the PR keeps both.
Mutation testing: measured, not asserted
You asked me to fix a "high mutation survivor rate". I re-ran the baseline on a clean tree before changing anything, because the floor is CI-enforced and narrowing scope moves the score.
50.7% (killed 1061 / 2091), up from the documented 41.2%. Per-module numbers derived from the mutant sources and survivor list — they reconcile to the 50.7% total, so they are measured, not estimated:
job/payload.pynetsafety.pyslicer/options.pydownload/naming.pyjob/predict.pydownload/validation.pyslicer/output.pyTwo findings worth your attention
1. The C.4 prediction did not pan out.
docs/mutation-baseline.mdclaimed the hermetic Orca stub "should score materially higher" forslicer/output.pyand asked for a re-run. Re-run: 21.8%, up 0.8 points. Line coverage went 79.8% → 92.7% while the mutation score did not move — the textbook signal that those tests execute_finalize_slicewithout constraining it. That module now holds 269 of the 1027 survivors (26%). The doc now says this instead of the optimistic prediction.I did not try to fix it here. The honest options are to extract the pure decision logic out of
_finalize_slice, or to stop counting it — both are production refactors, not test changes.2. A real bug in the mutation runner.
scripts/run_mutation_baseline.shassignedMUTATION_SCORE_FLOORwithout exporting it, so the score check — a child python process — never saw it and fell back to its own hardcoded default. A local run printedfloor: 40%in its header and enforced a different number a few lines later. CI was unaffected only because the workflow sets the variable at job level.Now exported, one value, and the child errors out rather than inventing a default.
Floor raised 40 → 48, just under the measured 50.7% — the same discipline already used for the coverage floor. The old 40 was set against 41.2% and had ~10 points of silent-drift room.
New boundary tests
tests/test_download_validation_boundary.py(32 tests) covers the_reject_*functions indownload/validation.py— in the mutation scope, but with no direct tests. They were only reached incidentally through whole-command tests, which is exactly why mutants there survived: nothing asserted what they do.It asserts observable outcomes (exit code,
failed_step, machine-readable fields, credential redaction) and deliberately not error-message prose. Pinning message text couples tests to copy-editing; those mutants stay recorded as equivalent.While writing them I found
.exeis not on the refuse-list andtext/htmldeliberately passes (it is the HTML-scrape path). Both are correct — the list names archive/document/image types, not a security allowlist — and are now documented in the tests.Also
scripts/syntax_smoke.pynow discoversscripts/andtests/instead of carrying a curated list that named two files this PR deletes — the same stale-parallel-list failureCLAUDE.mdwarns about. 87 → 157 files compiled.Gates