SignalMap

A finished measurement study of unsupervised condition monitoring — and the code that produced it.

This repository is archived. It started as a product — fit a detector on healthy recordings, flag deviations — and the measurements said the product was not worth building. What is left is worth reading: a preregistered study on nine public datasets, the null controls that killed our own method, and ten amendments recording our errors. Nothing here is maintained. Issues and pull requests are closed.

The result

Across seven tallied public domains (MIMII, Paderborn, MAFAULDA, plus two synthetic controls), under one frozen recording-level protocol with label shuffle nulls and bootstrap CIs over recordings:

1. Nothing beats the RMS of the raw signal, overall — not even the published state of the art. We tested our own searched nine-program feature grammar, the eight classic condition indicators, catch22 (22 features distilled from 4791 hctsa candidates over 93 classification problems) and MiniRocket (~10 000 convolutional features). Against the honest RMS: catch22 better on 1 of 7 domains and worse on 2; MiniRocket better on 2 and worse on 1.

2. Which method wins flips per domain, and the spread is enormous.

domain	raw RMS	catch22	MiniRocket	our spec
mimii_fan	0.742	0.530 ᴺ	0.637	0.615
mimii_pump	0.864	0.675	0.859	0.792
mimii_slider	0.934	0.901	0.988	0.888
mimii_valve	0.744	0.828	0.808	0.740
paderborn_kat_n15	0.510 ᴺ	0.974	1.000	0.649 ᴺ
synth_neg	0.855	0.800	0.841	0.810
synth_pos	1.000	1.000	1.000	1.000

Direction-free AUC*; ᴺ = does not clear its own label-shuffle null. On MIMII fan, plain RMS wins and catch22 falls below chance. On Paderborn phase current, RMS is at chance and MiniRocket separates perfectly. Without labels you cannot tell which case you are in — and a healthy-only detector has none.

3. The sign is not identifiable from healthy data alone. On two of nine domains the anomaly score ranks faults as more normal than normal (AUC 0.2786 and 0.3172, bootstrap CI clear of 0.5). A detector that assumes "far from healthy means faulty" is worse than a coin flip there. This is textbook — no optimal test without a specified alternative (Neyman–Pearson), one-class methods define outlierness without ever seeing an outlier (Schölkopf 2001, Tax & Duin 2004) — and a hostile review said so before we did.

4. Our own searched grammar is best on 0 of 9 domains. It loses to catch22 outright. There is no reading of this where the search earns its complexity.

Full tables: study/REPORT_external.md (the challenge against catch22 and MiniRocket, prereg study/PREREG_EXTERNAL.md) and study/REPORT.md (the sign-identifiability study, prereg study/PREREG.md).

The errors we found in ourselves

The amendments are the part we would ask a reader to check first, because they are what the nulls were for.

• A selection bonus in our own benchmark. The "best of six baselines" was chosen and scored on the same recordings — an advantage our fixed spec never had. Corrected: the baseline is picked on one half of the eval recordings and both methods judged on the other. The verdict did not change.
• A baseline that measured floating-point residue. distill.window() z-normalises every 1024-sample window, so the std baseline is 1.000000 on all nine banks with a spread of 1e-16. It "won" on four domains because that residue is a monotone proxy for the level the normaliser had removed. Recomputed on the raw recording the numbers move by less than 0.03 — right verdict, wrong mechanism. See REPORT_baseline.md, Amendment 2.
• A capacity gate that was inert. budget = 50 × n_recordings was meant to keep the search proportional to the evidence. The grammar enumerates 2128 programs, so above ~43 recordings it admits everything. On the study banks it did nothing at all.
• Ten further amendments are in study/, including the failed null controls that removed two banks from every tally.

What the code still does

Installable (pip install 'signalmap[all]') and tested — 408 tests pass — but unmaintained, and the study above is the honest description of what it can do.

• signalmap checkup --bank recordings/ — fits on half your healthy recordings, scores the held-out half plus your labelled bad ones, and answers SEPARATES or REFUSED with a confidence interval and a direction. Given result 2 above, this go/no-go answer is the most defensible thing in the package: it tells you the method does not work on your data instead of handing you a number.
• Signed receipts. Every verdict-producing command writes a JSON receipt — the claim, the verdict, the sha256 of every input, an Ed25519 signature — and tools/verify_receipt.py checks it while importing nothing from this project (stdlib plus cryptography). It is hardened against 23 concrete attacks on the gap between the signed bytes and the fields a human reads: duplicate JSON keys, NaN/Infinity, integers past 2⁵³, unpaired surrogates, and a pinned-key check that once compared transcriptions rather than key bytes. Verification proves attribution and integrity, never truth.
• signalmap distill — the searched feature grammar. Experimental, and the study says it does not pay for itself. Kept because the receipts it emits are what let a reader tell a found feature from a lucky one.
• The eight classic indicators in signalmap/indicators.py are a plain, dependency-free panel with a dual-holdout selector. For real work prefer pycatch22 (Lubba et al. 2019) — the same idea, distilled from a far larger candidate pool and validated on far more problems.

On CWRU bearing data, fitting on 945 healthy frames and monitoring 1183 catches 238/238 faults with 2/945 false alarms. That number is real and it does not transfer; the tables above are what happened when we measured instead of assuming.

Reproducing

python3 -m pytest -q                                    # 408 tests
.venv/bin/python study/tools/baseline_benchmark.py --refresh
.venv-ext/bin/python study/tools/external_benchmark.py --refresh   # needs aeon + pycatch22

study/manifests/ carries the sha256 of all 7623 recordings behind the banks. examples/ reproduces the CWRU result end to end.

What is in this repository

study/         the studies: preregistrations, reports, receipts, bank manifests
signalmap/     the package: detectors, the feature grammar, receipts, CLI
tools/         verify_receipt.py — the standalone verifier, imports nothing from us
research/      the full working record: every preregistration, analysis and log,
               including the dead ends
examples/      runnable recipes, including the CWRU result
docs/          ARCHITECTURE.md — the two detector paths and the known limits
firmware/      Rust edge-core for capture on constrained hardware (experimental)
sensors/ infra/ schema/   microphone capture, MQTT config, on-wire frame format

research/ is kept deliberately. For a negative result the working record is the evidence, not clutter.

License

Apache-2.0.