Batch modernization phase 1: offline BatchPCA, dataset loaders, converter completion#459
Open
kgdunn wants to merge 8 commits into
Open
Batch modernization phase 1: offline BatchPCA, dataset loaders, converter completion#459kgdunn wants to merge 8 commits into
kgdunn wants to merge 8 commits into
Conversation
…ter completion Opens the Phase 1 work described in the batch data processing modernization plan: an offline batchwise (BWU) PCA model with batch-level diagnostics, dataset loader functions for the bundled nylon/dryer/fake batch data, completion of the three unimplemented data_input converters, and a sweep of small lint/dead-code cleanups. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SyEpexCyqHn1rwTMYQSiP6
load_nylon(), load_dryer(), and load_batch_fake_data() return the standard batch-data dictionary directly, following the loader-function precedent in experiments/datasets.py. Until now every consumer had to locate the CSVs by path and convert them by hand; the book's worked examples need a one-line reproducible load. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SyEpexCyqHn1rwTMYQSiP6
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
melted_to_wide returned an empty dict, wide_to_melted an empty frame, and wide_to_dict None; the tests pinned that placeholder behaviour. All three are now implemented by composing the existing, tested conversions (melted_to_dict, dict_to_wide, dict_to_melted), and wide_to_dict inverts the (tag, sequence) pivot directly. The tests now assert round-trip identities instead of emptiness, including a truncation-aligned round-trip on the real nylon dataset. wide_to_dict and the new dataset loaders are exported from the batch namespace. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SyEpexCyqHn1rwTMYQSiP6
- plotting.py: fix the silently-ignored 'resonsive' Plotly config typo; replace stdlib random.seed/shuffle with the package-wide check_random_state contract; replace the inert file-level 'flake8: noqa: C901' directive with per-function noqa codes (it was blanket-suppressing C901 for the whole file under ruff). - preprocessing.py: remove the dead guarded plotly import (go was never used in this module) and route the DTW progress print through the module logger. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SyEpexCyqHn1rwTMYQSiP6
BatchPCA brings the Nomikos-MacGregor batch modelling approach to the library. It unfolds an aligned batch-data dictionary batchwise (one row per batch) via dict_to_wide, optionally joins an initial-conditions (Z) block onto that row, mean-centres and scales every column with MCUVScaler (which removes the average trajectory), and fits the existing multivariate.PCA by composition. Batch-level scores, SPE and Hotelling's T2 with control limits, plus contribution and score/SPE/loading plot forwarders, are exposed as batch-indexed views. Loadings keep the (tag, sequence) index so they reshape to a variable-by-time grid. Until now the batch module stopped at feature extraction with no multiway model; the unfold-then-PCA path existed only privately inside find_reference_batch. This is the headline capability the batch case study in the book depends on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SyEpexCyqHn1rwTMYQSiP6
time_varying_loading_plot draws one component's loadings as a function of time (one trace per tag), reading the batchwise-unfolded model's (tag, sequence) loading grid; initial-condition loadings are shown as a marker group before time zero. contribution_at_time_plot renders the per-tag SPE or T2 contribution at a chosen time sample for one batch, to localize an abnormal event to the responsible variable. Both are plotly, guarded behind the plotting extra, and use the package theme. The batch score/SPE/T2 plots reuse the multivariate plot functions on the internal model, so only these two time-resolved plots are new. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SyEpexCyqHn1rwTMYQSiP6
Version bump for the new batch capabilities (BatchPCA, batch model plots, dataset loaders, completed data_input converters). CITATION.cff version and date-released synced to 1.55.0 / 2026-07-16; CHANGELOG section added with the compare-link footer; docs/api/batch.rst now autodocuments BatchPCA, the dataset loaders, and the batch plots. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SyEpexCyqHn1rwTMYQSiP6
Adds tests for the validation and error paths flagged as uncovered: initial-conditions type/numeric/missing-value guards, group_by_batch with a Z block, fit_transform, diagnose with initial conditions, ellipse_coordinates, and the contribution-plot unknown-batch and out-of-range-time errors. Raises patch coverage of the new batch modules (_batch_pca to 97%, _batch_plots to 94%). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SyEpexCyqHn1rwTMYQSiP6
This was referenced Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BatchPCA, an offline batchwise-unfolded (BWU) PCA model for batch data: dict-of-batches in, batch-indexed scores/T2/SPE out, with an optional initial-conditions (Z) block joined onto the one-row-per-batch unfolded matrix. Built by composition over the existingmultivariate.PCA+MCUVScaler, so it reuses the library's proven numerics. Loadings keep the(tag, sequence)index so they reshape to a variable-by-time grid.time_varying_loading_plot,contribution_at_time_plot) and dataset loaders (load_nylon,load_dryer,load_batch_fake_data).data_inputconverters (melted_to_wide,wide_to_dict,wide_to_melted) and sweeps small cleanups inbatch/(dead plotly import,resonsivetypo, stdlibrandom->check_random_state,print-> logger, stray flake8 noqa).This is Phase 1 of the batch data processing modernization plan (offline model + loaders + cleanups). Online Nomikos-MacGregor monitoring, sklearn transformer facades, and BatchPLS-to-quality are planned follow-up PRs.
Test plan
tests/batch/test_batch_pca.py: fit on aligned nylon data (shapes, monotone R2, loadings reshape to variable x time), transform round-trip, diagnose consistency, synthetic injected-fault batch flagged by SPE/T2, Z-block join and mismatch validation, wrong-length rejectiontests/batch/test_batch_plots.py: both plots build; component/column-shape guardstests/batch/test_datasets.py: loaders return valid batch dictstests/batch/test_data_input.py: stub-pinning assertions replaced with round-trip identities (including a real-nylon round-trip)uv run pytestfull suite: 2026 passed, 2 pre-existing skipsuv run ruff check .anduv run mypy src/process_improveboth passChecklist
pyproject.toml(MINOR: 1.54.0 -> 1.55.0, new modules/API) andCITATION.cffsyncedruff check .passesCHANGELOG.mdupdated🤖 Generated with Claude Code
https://claude.ai/code/session_01SyEpexCyqHn1rwTMYQSiP6