M5: the evaluation harness before any model — scoring as numbers, streams, the report, the DET notebook - #49
Merged
Merged
Conversation
…cs as numbers The hit window [h - T, h + L + T] with h = e // hop and L the §7 ceiling of 20 hops; the reference decision stage (trailing moving average over W hops, upward threshold crossings, a refractory period of R hops under which false accepts merge) that kws.h must match at M6; the exact two-sided Poisson interval on a count (chi-square form), the one-sided zero-event bound ln 20 / H and the Wilson interval, each checked in _self_check against hand-computed values (3 events in 0.5 h -> 6.0 FA/h, [1.237, 17.535]; 0 events in 3.0 h -> 0.9986 FA/h bound; 190/200 -> [0.9104, 0.9726]). requirements-notebook.txt pins the DET notebook's toolchain, which CI never installs (the DET is the notebook verification layer, plan §7). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ZkpA4PWuaan5ek2CL2DDm
…eams, the report, the DET notebook The wake-word plan's M5 (docs/wake-word-plan.md §6 M5, §7): the meter before the model, every scoring rule a number and every pass item a test that can fail. tools/ml/kws/ - kws_scoring.py (committed earlier on this branch): the hit window [h - T, h + 20 + T] inclusive with h = e // hop; the reference decision stage — a trailing moving average over W = 10 hops, an upward crossing fires, a crossing fewer than R = 100 hops after an event merges into it (t - last >= R) — the numbers kws.h must match at M6; the exact two-sided Poisson interval, the zero-event bound ln 20 / H, the Wilson interval, all checked against hand-computed values. - kws_streams.py: every variant-0 eval positive becomes one stream whose audio is the row's mixture as extract featurized it (kws_build.mixture), so the hit window lies inside its hops; each share's negatives, sorted by id, packed into streams of at most 60 s, a clip never split; hours from decoded lengths — the plan's denominator, positives excluded; every mis-accounting refused by name (validate_stream, one rule set the harness applies to the audio it decodes for scoring). - kws_detectors.py: the detector contract (one float64 score in [0, 1] per completed hop, n // hop of them, measured through the bridge), the band-energy sanity baseline through the shipping front end, the oracle's PlantedDetector; the alignment pinned on a planted 1 kHz burst. - kws_holdout.py: M4c's holdout.json (talkers with consent form version and permitted uses; utterances with a FLAC under <store>/holdout/, its sha256, microphone path, distance, SNR, phrase, endpoint), refused on any schema deviation; verify_holdout refuses by file a missing or altered FLAC, a stray FLAC, two rows for one file, a talker without consent or without the evaluation and m7-replay uses; the hold-out set id over the sorted hashes; one positive stream per utterance. - kws_eval.py: one decode per stream, the score contract enforced, the sweep, and the committed report (report.json + report.md, version 1): manifest and eval_set_id per share, the scoring numbers, the detector, the front end's contract version and DspTap commit, hours per share with the packing bound, positives per subshare, then per threshold recall with its Wilson interval, FRR, spurious events, and FA/h on speech, music, TTS speech and noise as separate columns — each with its hours and Poisson interval, <= ln 20 / H at zero events, `absent` for a share the lock lacks — with the eval-tts recall beside the hold-out's. - test_kws_eval.py, test_kws_streams.py (44 tests, 2 s, in the kws-dataset CI job): the planted-event oracle exact to the utterance (window edges inclusive, one hop outside each; 3 of 4 planted events counted under the refractory period in exactly 0.5 h -> 6.0 FA/h [1.237, 17.535]; events R apart fire, R - 1 apart merge; a zero-event share reports ln 20 / H; spurious events never in FA/h), the mis-accounted variants refused by name, the report round trip, the toy swept end to end through the bridge (hours = the pcm tier's decoded lengths, eval_set_id = the lock's), the hold-out record over soundfile-written FLACs, and the CLIs' refusals. tools/ml/build_kws_det_notebook.py -> notebooks/kws_det.ipynb, executed and committed (14 s, 0.12 MB, no personal path): the scoring semantics quoted from kws_scoring, the bring-up corpus from the store, the baseline sweep, the report, the DET per share with the zero-event bound marked, and a closing cell that argues only from printed figures. Measured on the M0 Mac: 412 streams — 195 positives, eval-speech negatives 2.9513 h in 179 streams (bound 1.015 FA/h), eval-noise 0.5021 h in 38; the baseline scores ~2,500 s of audio per second including decoding; sweep 8.6 s wall; best recall 34/195 at 1,977 FA/h on speech — the sanity curve, not a pass. docs/wake-word-plan.md: the M5 Done record; the M4a record's "3.0 h" corrected to the 2.95 h of negatives the denominator counts. HANDOFF's wake-word bullet records M5 as done and names M4b and M6 as next. kws_manifest.check_keys is now public (additive) so the hold-out and report schemas share the manifest's key check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ZkpA4PWuaan5ek2CL2DDm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
M5 of the wake-word plan (
docs/wake-word-plan.md§6 M5): the evaluation harness, built before any model.tools/ml/kws/gains the scoring semantics as numbers (kws_scoring.py), stream assembly from a lock (kws_streams.py), the detector contract with the band-energy sanity baseline and the oracle's planted detector (kws_detectors.py), M4c's hold-out record verified by hash and consent before scoring (kws_holdout.py), and the sweep with the committed report format (kws_eval.py); plustools/ml/build_kws_det_notebook.pyand the executednotebooks/kws_det.ipynb. Stacked on #48 (M4a); retargets tomainwhen that merges.Why
The plan puts the meter before the model: M6's architecture choice and operating point are read off this harness, so its rules must be numbers and its pass must be a test that can fail. The hit window [h − T, h + 20 + T], the reference decision stage (W = 10, R = 100, upward crossings, t − last ≥ R) that
kws.hmust match at M6, the hours from decoded durations, the exact Poisson and Wilson intervals and the ln 20 / H zero-event bound are all pinned here.Verification
Run on the M0 Mac (CPython 3.12.14, numpy 2.5.3, scipy 1.18.1, soundfile 0.14.0, DspTap
5ca3b1c):python -m unittest discover -s tools/ml/kws -p 'test_*.py': 118 tests, OK, 24 s (M4a's 74 + M5's 44; thekws-datasetCI job runs the same command).validate_streams: a wrong, non-integer or negative hours count, an endpoint or window beyond the stream, duplicate stream ids, a positive id used twice, a positive stream with hours, NaN / short / out-of-range / complex scores, thresholds outside [0, 1] or repeated.manifests/speech_commands_v2_bringup.json, from M4a: the keyword-spotter dataset builder, its contracts, the toy fixture and its CI job #48): 412 streams — 195 positives; eval-speech negatives 2.9513 h in 179 streams (bit-identical to the lock's summary; zero-event bound 1.015 FA/h), eval-noise 0.5021 h in 38; music and TTS absent and reported so. The band-energy baseline through the bridge scores ~2,500 s of audio per second including decoding;kws_eval.py sweep8.6 s wall. Best recall over the sweep 34/195 at 1,977 FA/h on speech: the sanity curve the plan asks for, useless as expected, not a pass.MUTAP_KWS_STORE); its closing cell argues only from figures the preceding code cell prints.kws-datasetjob is the gate for the 44 new tests on Linux.Notes for the reviewer
report.jsonversion 1) is new in this PR and committed by name;max_stream_sis recorded because every negative stream is decided after a fresh reset, so each FA/h row depends on the packing by at most streams / H (60.65 FA/h at θ = 0 on this corpus; 12–40 FA/h, 0.6–3.1 %, at thresholds 0.1–0.9).kws_manifest.check_keysis public so the hold-out and report schemas share the manifest's key check; M4a's tests are untouched and green.requirements-notebook.txtpins the notebook toolchain, which CI never installs (the DET evaluation is the notebook layer by design, §7).kws_eval.py's CLI printskws_eval: <Type>: messagewith rc 1 on a refusal where the other CLIs printrefused:with rc 2; cosmetic, left for a follow-up.🤖 Generated with Claude Code
https://claude.ai/code/session_016ZkpA4PWuaan5ek2CL2DDm