Post-implementation follow-ups: deferred structural work, docs, and the release runbook

[cnicholas]

Follow-ups from the 2026-08-24 implementation pass. Workstreams A–C of the release-prep plan are merged (8276e46..d886341); this tracks what was deliberately left, what was discovered along the way, and the owner-only release runbook.

Cross-references, so nothing here is duplicated: #77 (release prep phase 3 — most of its checklist is now done in-tree), #81 (API contract hardening — several items closed below), #82 (maintenance polish — analysis.py, mypy, coverage), #84 (decompose analysis_result.py), #100 (ODS ↔ SDS terminology — still open, see note), #86 (CI matrix restoration — superseded, the full matrix is live).

---

Closed by this pass — worth verifying before ticking elsewhere

From #81's API-contract list:

• get_statistics shape — the four-key contract {N, center, lpl, upl} now actually holds on X/mR, pinned by TestStatisticsKeyContract over chart × design state × stratification.
• result.dataset mutability leak — now copy-on-read, matching Study.dataset.
• Raw ValueError on user-facing paths — 8 converted; ~25 internal-invariant sites remain deliberately.
• ChartNotAvailableError reparented under ValidationError, which is what the exception-convention docs already claimed.

From #77:

• Full CI matrix, wheel smoke-test, trusted publishing, SHA-pinned actions, Dependabot — all present. Added since: a tag↔__version__ guard, workflow_dispatch removed from publish.yml, docs build on PRs, pinned mystmd, pytest timeout, pre-commit hygiene hooks, issue/PR templates, CITATION.cff.

---

1. Deliberately deferred (structural)

These were listed as non-goals in the plan. None is blocked; all are invasive.

• R6 materialises onto the study-level frame. Study._compute_r6 writes a column onto the shared dataset from inside execute(), making execute non-pure. Contained today by one metadata gate in AnalysisResult._locate_residual. Computing it onto the per-request frame instead (Analysis already copies per calculation) would delete the wart and the gate defending it. Touches CLAUDE.md-protected code — do not "simplify" the gate without doing this properly.
• Chart identity lives in three parallel authorities — VALID_BASE_CHARTS (spc_constants), VALID_CHART_TYPES (plotting/contracts), and an inline set in analysis.py. Cheap first step: make the latter two import the first. The full per-chart registry stays deferred until a second chart family is real. (Overlaps Tier 3: Maintenance polish — analysis.py refactor, vocabulary, mypy, coverage #82.)
• analysis.py is 2,897 lines with structurally duplicated stratified/unstratified pairs. This pass extracted the ungrouped replication raises into a shared helper, but the mR family (~950 lines across three near-parallel calculators) is untouched. The changelog's own "three fixes from one cause" entry is the evidence for why. (Overlaps Tier 3: Maintenance polish — analysis.py refactor, vocabulary, mypy, coverage #82.)
• Effects dict namespace. Keys are the user's own factor names sharing a namespace with library vocabulary — a factor literally named time or main_effect collides silently. Docstrings now describe the real keys and name the hazard; the structural fix (namespacing, plus a tidy long-format effects_table()) is a breaking change best made before 1.0.
• Pre-1.0 surface pruning. AnalysisResult carries a dict-emulation shim "for backward compatibility" with no release to be compatible with; plotting.__all__ exports Plotter while labelling it internal. Free to delete now, a deprecation cycle later.

2. Typing (#47 / #82)

• 16 mypy errors remain in analysis.py, all TypedDict mismatches in the chart-payload contract. Flip continue-on-error: false in ci.yml once cleared.
• Extend the per-module strict list. Currently 7 modules re-enable arg-type/return-value/assignment. Verified not yet clean: data_preparation (5), focused_result (4), result_tabular (1), exporters.excel_exporter (1).
• signals/config.py has 2 annotation-unchecked notes (untyped function bodies).

Note on the mypy config: python_version = "3.12" is the stub-parsing version, not the support floor (requires-python = ">=3.10"). numpy's bundled stubs use 3.12 type statements and mypy aborts on numpy/__init__.pyi below that — which is why type-checking had been silently dead. Revisit if numpy's stubs change.

3. Docs

• Six tutorials are pulled from the TOC but still on disk, awaiting a rewrite against the current API: basic-imr, signal-detection, stratified-analysis, xbar-s-analysis, sds-validation, sds1-complete-analysis. They rejoin docs/myst.yml and the CI nbmake gate together — not one without the other.
• Two tracked docs files are unreachable from the TOC: docs/sds-detection.md (505 lines) and docs/reference/sds_definitions.md (133 lines). The first is not a duplicate of docs/user-guide/sds-detection.md — it is a deeper "how classification works" reference. Neither has stale-API markers. Decision needed: publish (after verifying their examples) or delete. Left untouched rather than guessed at.
• The API reference is hand-written and drifts. docs/reference/api.md is 702 lines of hand-maintained signatures; it was missing statistics(), stratify_by, and supports_calibration. mystmd has no mature Python autodoc, so this needs an explicit mitigation: a generation script, or minimally a CI check that every __init__.py export appears in the reference.
• Comparison page + SEO landing pages (planned as C4, not written). A "vs. other Python SPC options" page and short worked-example pages for "XmR chart in Python" / "control chart in Python" / "process behavior chart". Deliberately left to the maintainer: they make factual claims about other projects and are positioning copy.
• Docs versioning strategy before 1.0 — mystmd has none built in.

4. Terminology — #100 stays open

⚠️ This pass initially relabelled the Design Report's observed row SDS (Sampling) → ODS (Observed), treating it as an inconsistency. That was reverted (d886341) once #100 surfaced: the label is deliberate, and #100 records that reconciling it needs a naming decision to settle with Tom, since "SDS" already carries the umbrella meaning. A comment now sits at the site so the next reviewer finds the open question instead of an apparent typo.

Nothing else in this pass touched ODS/SDS naming. #100 is unaffected and still the right place for that decision.

5. Validation

• ODS 4–6 end-to-end against Bishop's Minitab output. Smaller than it looks: detection and the deterministic ODS→ADS collapse (4→1, 5→2, 6→3) are covered by ~118 tests, and the collapse rule is definitional within the lineage model, so the ADS 1–3 numbers are the ones that apply. What remains is analysing the tidied PM SDS 4/5/6 subsets in Minitab and adding a spot-check assertion block — roughly three runs, not a validation campaign. Until then docs/reference/validation.md says so plainly.

6. Owner-only — release runbook

Sequential, one sitting, not to be automated. Full detail is in the local plan file.

• Freeze — no open PRs (a history rewrite orphans them), fresh-clone backup.
• Pre-rewrite audit — git ls-files sweep for internal planning files beyond the three known ones.
• History rewrite — git filter-repo --invert-paths over those files; force-push; verify the paths are gone from all history and the tip tree is unchanged.
• Retro-tag v0.1.0 / v0.1.1 at their (rewritten) commits — the CHANGELOG's compare links point at tags that have never existed. Push these before configuring the PyPI trusted publisher so the runs they trigger fail harmlessly at OIDC.
• Version cut — __version__ → 0.2.0, move the ~582-line [Unreleased] into a dated section, add its compare link.
• Enable GitHub Pages (the docs deploy job is gated on it).
• Enable private vulnerability reporting (Settings → Security) — SECURITY.md now points reporters there; the feature is public-repo-only, so it can only be switched on at go-live. Also fold the historical author-email scrub (personal gmail on 797 commits, and 20 commits with a *.gha.chartermi.net machine email) into the history-rewrite step.
• Configure the PyPI trusted publisher — cnicholas/processbehavior, workflow publish.yml, environment pypi.
• Publish — push v0.2.0; verify pip install processbehavior in a clean venv runs the README quickstart end to end.
• Post-publish — CODECOV_TOKEN secret, then flip fail_ci_if_error: true and add the coverage badge; reconsider pip-audit --strict (it can only work once the project is on PyPI); submit the conda-forge feedstock.

7. Compatibility posture

• Test against the pandas 3 RC and plan lifting pandas<3; library upper caps export resolver conflicts downstream, so the cap needs an expiry rather than riding indefinitely.
• Verify and publicly declare numpy 2.x support — the numpy>=1.23 floor spans the 1.x/2.x break.

[cnicholas]

Additions from the first CI run on the pushed work

The push exercised the workflow changes for real. Three items to add.

Go-live: turn on branch protection

Required-checks enforcement on main is currently off (gh api .../branches/main/protection → "Branch not protected"). Until it is on, a failing check is a signal rather than a gate — the validation run against Bishop's reference results can go red and a change still lands.

That is acceptable while the maintainer is the only person accepting changes. It stops being acceptable the moment the repository is public and outside contributions are possible, because the claim that the reference numbers are a hard acceptance criterion depends on it. Add to the go-live sequence, alongside enabling Pages and configuring the trusted publisher: protect main, require the CI checks (at minimum the test matrix and the Bishop validation step), and disallow bypass.

audit dependencies fails on an ambient CVE

setuptools 79.0.1 / PYSEC-2026-3447, fixed in 83.0.0. This project does not declare setuptools — it builds with hatchling — so this is whatever version the runner ships. Pre-existing and unrelated to any code change, but it is a permanent red job. Fix is one line in the audit job: pip install --upgrade pip setuptools before pip-audit.

A performance test is flaky on Windows

test_performance.py::TestScalability::test_formulate_scaling[1000] asserts an absolute floor of 10,000 rows/sec. On the py3.13/windows-latest leg it measured 9,619 — 4% under — while passing on Windows 3.10 and 3.11 and on every other leg (2,165 passed, 1 failed). An absolute wall-clock floor on a shared runner will keep doing this. Options: lower the floor, make the perf tests advisory in CI, mark them slow so the default CI slice deselects them, or assert against a relative baseline rather than an absolute rate.

Worth settling before branch protection goes on, since a flaky required check blocks real work.

Also confirmed by this run

• Both 3.10 legs pass — the dependency resolution after dropping the 3.9-conditional pins is fine, which could not be verified locally.
• All five wheel smoke-tests pass, including Windows and macOS.
• The previously-broken notebook gate is fixed: the last pre-push run failed with ERROR: file or directory not found: docs/tutorials/calibration.ipynb (exit 4); it now executes six notebooks clean.
• Docs / build succeeds and renders the new validation page. Docs / deploy fails with HttpError: Not Found — Pages not yet enabled, failing on every run since 2026-07-29, already in the go-live list.

[cnicholas]

Pre-PyPI pass complete (d3c8f99)

Closed from this issue:

• Pre-1.0 surface pruning (§1) — the AnalysisResult dict shim and Plotter in plotting.__all__ are both gone. Verified against the app first: it uses the public .charts attribute and never references Plotter. 18 test call sites migrated, each keeping its original expected value.
• Orphaned docs (§3) — resolved in opposite directions. docs/reference/sds_definitions.md joins the TOC (clean, no code to rot, already cross-linked). docs/sds-detection.md deleted: it referenced study.sds_description and study.sds_reason, neither of which exists, and one of its 14 code blocks does not parse. Every inbound link already pointed at the published user-guide page.
• audit dependencies — setuptools upgraded explicitly in the audit job.
• Flaky perf test — test_performance.py marked slow at module level. All 34 tests deselect from the gating slice and still run in a full pytest tests/, under -m benchmark, and in scripts/benchmark.py, which tracks drift against a baseline rather than an absolute floor.
• Version cut (§6) — 0.2.0 across __version__, CITATION.cff and the CHANGELOG; the 584-line [Unreleased] is now ## [0.2.0] - 2026-08-31 with its compare link.

Corrections and additions to the runbook

GitHub Pages is still not enabled. An earlier read of mine was wrong: recent green Docs runs were pull-request runs, which skip the deploy job by design. The last main-branch deploy failed and gh api repos/.../pages returns 404. Still a go-live step.

The [0.2.0] changelog date must match the tag date. It is currently 2026-08-31. If publication slips, update it before tagging.

git filter-repo --invert-paths --path does not rewrite commit messages. This is a hole in the history-rewrite step as written. Commit c978208's message body names analysis_pack.py, "Bishop's IP" and "training material"; purging the file leaves that message intact and public. Two amendments:

1. Add --message-callback (or --replace-message) to the filter-repo invocation, mapping analysis_pack.py, processbehavior-app and the IP phrasing to neutral wording. c978208 is the only commit needing it today — re-check at freeze time.
2. Strengthen the verification from "the path is gone" to a full-history string sweep: git log --all --grep= for the private names plus git grep -I <term> $(git rev-list --all) for content. Confirm zero hits for analysis_pack, processbehavior-app and Bishop & Associates before the repo goes public.

For the record, CLAUDE.md itself contains no recipe — no fold names, ordering or gating rules, only the statement that a recipe exists and where. It is also the only tracked file naming the private repo or file; CONTRIBUTING.md's scope section states the boundary without naming either. Editing its text now would not help, since 14 historical commits touch it and every prior version is equally readable. The purge plus the message rewrite is what actually closes this.

Still open and unchanged

§1 (R6 rematerialization, chart registry, analysis.py split, effects namespace), §2 (mypy, #47), the six stale tutorials and API-reference autodoc in §3, #100, §5 ODS 4–6 validation, §7 compatibility posture, and the owner-only runbook in §6.

[cnicholas]

§1 item 1 and all of §3 are done

R6 per-request frame (§1, item 1) — landed on main in 7aea779 (bit-exact formula pins) + b004d43. R6/RCR6 now compute onto the requesting result's own frame in Analysis.calculate() (residual_calculator.calculate_r6_residuals); execute(), why_not(), and supports_calibration() no longer mutate the study frame, and the _locate_residual metadata gate is deleted because its precondition no longer exists. Numeric output verified bit-identical across 48 scenarios (T100 PM SDS 1–3 + unbalanced synthetic; all chart types, by= shapes, recentered forms). Full CI matrix was green on the push.

Docs workstream (§3, all five) — merged via #112 (b322378):

• Six tutorials — all rewritten against 0.2.0, freshly executed, and back in myst.yml + the ci.yml nbmake gate together (allowlist now 13 paths; local gate run: 13 notebooks in 17s). basic-imr returned as the entry-level Your First X/mR Chart; sds-validation split into Bishop Reference Validation + a new Design-State Tour, and dropping its 'notebook' plotly renderer took it 5.1 → 4.1 MB, clear of the 6 MB hook.
• Two unreachable docs files — this box was already resolved by d3c8f99 (deleted docs/sds-detection.md; published reference/sds_definitions.md into the TOC). Docs workstream: six tutorials return, API reference rewritten, comparison + examples pages, versioning policy (#109 §3) #112 fixed the leftovers: the published guide's calls to nonexistent study.sds_reason/sds_description, and the stale 0.1.1 pin in installation.md.
• API reference — rewritten end-to-end from an inspect.signature dump (18 previously-undocumented exports; the get_c4/ResidualCalculator/DataPrepConfig fictions removed; the {N, center, lpl, upl} contract and real exception tree documented). Guarded by the new tests/test_api_reference.py: every __all__ symbol must appear, known fictions must not — the "CI check that every __init__.py export appears" mitigation, chosen over a generation script.
• Comparison + SEO pages — docs/comparison.md plus docs/examples/{xmr-chart-in-python,control-chart-in-python,process-behavior-chart}.md. Drafted under a VERIFY-marker protocol; maintainer reviewed and cleared the external-project claims (ec4c0a1).
• Versioning strategy — decided + documented in CONTRIBUTING.md → "Docs versioning": one live site tracking main; no version strings in docs prose outside generated/drift-guarded pages; hand-pinned mystmd bumped as release housekeeping; versioned deploy paths deferred to 1.0.

Bonus fix found en route: detect_signals(rules='standard') raised despite being documented — presets bypassed SignalConfig.__post_init__ (a6c73da, with regression tests).

Still open elsewhere in this issue: §1 items 2–5, §2 typing, §5 validation, §6 runbook, §7 compatibility.