M4a: the keyword-spotter dataset builder, its contracts, the toy fixture and its CI job - #48
Merged
Conversation
…nifest and lock, store, audio The first slice of the wake-word plan's M4a (docs/wake-word-plan.md §6 M4, rev 3): the modules every later stage builds on, plus the package README that states the design contract (store tiers, manifest schema, lock with its identity/derived classification, stages, labels, split rules, shard format, the resolved-draw schema). - kws_features.py: `Geometry` mirroring log_mel_geometry field for field; `FrontEnd` driving the shipping front end through dsptap_py.LogMel (double in, float32 stored); `assert_band_support` through the reference's mel_weights, since log_mel.h zero-fills a band with no bin silently; `self_check` of the bridge against the numpy oracle on both paths at any geometry — measured 2026-09-09 on the M0 Mac: 1.5e-14 / 3.4e-14 at the reference geometry, 1.6e-15 / 5.3e-15 at DspTap's TUNED geometry; pinned at DspTap's double pin, 1e-13. - kws_manifest.py: sources[] / recipe with validation that refuses by name (repository origin without redistributable, unknown kind or role, empty licence, invalid geometry or band support, bad split fractions), the canonical-JSON manifest hash, the lock (Clip/Shard/Lock), eval_set_id over sorted clip ids, and compare_locks over the identity fields only. - kws_store.py: --store / MUTAP_KWS_STORE with no default; fetch never downloads (a repository-relative origin is copied, a store-only one is verified where a human placed it, both against sha256 and size, with a .verified marker every later stage checks); extraction with filtered members (no links, no absolute or .. paths). - kws_audio.py: soundfile decode, scipy.signal.resample_poly as the one resampler, int16 pcm and its sha256, the endpoint trim rule and its analytic transform, the RIR onset rule, draws keyed by (seed, clip id, variant), the keyed-hash split rule, and the augmentation primitives. DspTap pin 58210ae -> edf5441 (feat/reference-geometry: the parameterized numpy reference, so the self-check runs at the manifest geometry from the first build). requirements.txt pins the environment; .venv/ is ignored. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ZkpA4PWuaan5ek2CL2DDm
…; DspTap pin -> d3c1109 DspTap CI measured the tuned-geometry double parity at 6.8e-14 (log) / 1.5e-13 (PCEN) on Linux GCC x86-64, against 2.4e-15 / 7.1e-15 on macOS arm64 (FMA contraction differs, HANDOFF working note 2), so the self-check follows the re-pinned DspTap test: 5e-13, rounding-level, both measurements recorded in the docstring. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ZkpA4PWuaan5ek2CL2DDm
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ZkpA4PWuaan5ek2CL2DDm
…ure and its CI job The first stage of the wake-word plan's M4 (docs/wake-word-plan.md §6 M4, rev 3): the builder and its contracts on a redistributable bring-up corpus, every M4a pass item a test that can fail. tools/ml/kws/ - kws_build.py: fetch · decode · synth · augment · extract · shard · all over a manifest into the store (--store / MUTAP_KWS_STORE, no default; --jobs a spawn process pool that renders and featurizes but never draws). Every draw is a function of the manifest seed, the clip id and the variant, resolved in clip-id order before dispatch, so lock.json is byte-identical between --jobs 1 and --jobs 4. Positives are featurized embedded in same-split negative material with the draw's RIR and noise applied to the whole mixture (no keyword-bounded burst), one reset per mixture; the stored endpoint is e''' = round(e/f) + rir_delay + offset, never re-trimmed, with f the rational factor the resampler actually applied. Shards carry the README's arrays and a header (contract version, DspTap commit, geometry, kws_features version, manifest hash) written with fixed zip timestamps so a shard's digest is a function of its arrays. Every stage refuses out of order, on an unverified archive, under another front-end contract version, or over rows extracted at another DspTap commit. - kws_sources.py: adapters for speech_commands_v2 (official split lists, speaker-hash key, endpoints by the trim rule), musan (per-file attribution from the subset LICENSE classified into a licence id the card checks; an unclassifiable block is refused), openslr_28_simulated (rir_list room key, simulated subset only), keyword_list, piper (a pinned voice archive) and a holdout stub that refuses naming M4c. Layouts read from the real archives. - kws_features.py: the bridge library is keyed to the submodule commit it was built from (.dsptap_commit marker; rebuilt Release on a pin move; a dirty submodule refused), so the commit the lock and shard headers record is the one that computed the features. Self-check pinned at 5e-13 with both platforms' measurements recorded. - kws_manifest.py: schema-key diffing on every document (unknown or missing fields refused by name), single-segment ids and bare archive file names, in-repository origins confined to tools/ml/kws/fixtures/, stored_path tied to geometry.pcen.enabled, ranges as [lo, hi] of finite numbers, context_s >= 2 s, mic_model "none" only (nothing else is implemented), the TTS block tied to piper sources. kws_store.py: every tier path confined to the store. - verify_splits.py: R1-R5 as named rules over a lock (R2: no decoded-PCM leak across splits, every draw from the clip's own split, hold-out mixtures from eval-noise only), one planted-violation fixture per rule under fixtures/planted/, --self-test requiring each to be rejected by exactly its rule; conventions for the M4b person keys in the docstring. - kws_dataset_card.py: DATASET.md and ATTRIBUTION.csv rendered from the manifest and lock, nothing typed; --check refuses an unlicensed clip, a flagged clip and any NC licence id per file; the hand-run record cited when a build ran no synth. - fixtures/toy/: 662,293 bytes of committed audio (15 Speech Commands clips, six 2 s MUSAN excerpts with their LICENSE blocks, two SLR28 small-room RIRs chosen by the builder's own RT60 rule, the 35 Speech Commands keywords), cut deterministically by fixtures/make_toy_fixture.py from the verified upstream archives; expected/ holds the lock, the three shards and the card of a real build on the M0 Mac at DspTap 5ca3b1c. - fixtures/synth_hand_run/: the M4a hand-run of synth over en_US-kristin-medium (piper-tts 1.8.0, onnxruntime 1.29.0, as an isolated subprocess tool): the manifest and a lock excerpt of its six TTS rows with every per-clip TTS field, no audio (the voice and its output stay in the store). - test_kws.py (+ test_verify_splits.py, test_dataset_card.py): 74 tests. The toy rebuild into an empty temporary store reproduces the committed lock's identity fields and shards exactly (max |feature difference| 0.0, 3/3 shard digests identical; pinned at 1e-6 relative for other toolchains), --jobs 1 == --jobs 4, a corrupted or unverified archive refused, each planted leak rejected by rule name, the planted unlicensed clip and NC track failing the card, the self-check at both geometries, a non-redistributable in-repository source refused, plus the robustness and schema refusals. Measured on the M0 Mac (CPython 3.12.14, numpy 2.5.3, scipy 1.18.1, soundfile 0.14.0): all --jobs 4 2.65 s, --jobs 1 0.82 s; the suite 22 s. .github/workflows/ci.yml: the kws-dataset job (checkout with submodules, Python 3.12, the pinned requirements, the bridge built Release through kws_features.py --ensure-bridge, unittest discover). Not yet run on Linux from here: it is the PR's gate. THIRD_PARTY_NOTICES.md: the toy fixture's sources and the build-time subprocess tools (piper-tts, espeak-ng embedded in it) that are never imported, linked, fetched in CI or redistributed. HANDOFF's wake-word bullet records M4a's state and what remains (the full Speech Commands v2 ingestion). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ZkpA4PWuaan5ek2CL2DDm
…he Done record tools/ml/kws/manifests/speech_commands_v2_bringup.json: Speech Commands v0.02 in full under its official split, MUSAN's noise partition (the one unattributed sound-bible file excluded), SLR28's 60,000 simulated RIRs and the keyword list; K = 2, M = 1, dry share 0.25, context 2 s. Built on the M0 Mac with --jobs 8 (measured 9 September 2026): fetch verifies 14.8 GB; decode 166,758 clips in 39.6 s; augment 96,729 variants in 65.2 s; extract 200,653 rows, 27.5 M frames, in 49.6 s; shard 784 shards in 227.6 s; 396 s in all at 2.3 GB peak resident. marvin positives 1,710 / 195 / 195 (train / dev / eval), negatives 83,133 / 9,786 / 10,810; the eval-speech share 3.0 h (zero-event bound ln 20 / 3.0 = 1.0 FA/h, the bring-up figure the plan predicted), eval-noise 0.5 h; 27,455 of the RIRs inside rt60_s [0.2, 1.0]. verify_splits R1-R5 clean over 263,487 rows; the card's licence checks pass. The rendered card (DATASET.md, ATTRIBUTION.csv) is committed beside the manifest as the record M5 cites; the lock (164 MB) and the features stay in the store. docs/wake-word-plan.md gains M4a's Done record with every number above and the CI figures; HANDOFF's wake-word bullet now says M4a is done and names M5's bring-up and M4b's phrase as next; the kws README documents manifests/. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ZkpA4PWuaan5ek2CL2DDm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
M4a of the wake-word plan (
docs/wake-word-plan.md§6 M4, rev 3):tools/ml/kws/— the keyword-spotter dataset builder and its contracts on a redistributable bring-up corpus — plus akws-datasetCI job that rebuilds a committed toy fixture, the notices for the committed audio and the build-time GPL tools, and the DspTap pin moved tomain's5ca3b1c(tap/DspTap#16, the parameterized numpy reference).Why
Rev 3 staged M4 so that the builder, its refusals and its reproducibility contract exist and are tested before any corpus is assembled or any talker recorded. Training features come from the shipping front end through DspTap's C ABI bridge (the 8 September decision); the numpy reference is the oracle the module self-checks against, now at any geometry. Every M4a pass item in the plan is one test here.
Verification
Built and run on the M0 Mac (CPython 3.12.14, numpy 2.5.3, scipy 1.18.1, soundfile 0.14.0, DspTap
5ca3b1c):python -m unittest discover -s tools/ml/kws -p 'test_*.py': 74 tests, OK, 22 s. The toy manifest rebuilds into an empty temporary store and reproduces the committed lock's identity fields and its three shards exactly (max |feature difference| 0.0, 3/3 shard digests identical; pinned at 1e-6 relative for other toolchains);lock.jsonis byte-identical between--jobs 1and--jobs 4(0.82 s / 2.65 s wall); a wrong sha256 and a one-byte-changed archive are refused byfetchanddecoderefuses the unverified archive; each planted split-leak fixture is rejected by exactly its rule (R1–R5,verify_splits.py --self-test); a planted unlicensed clip and a planted CC BY-NC track fail the card; the self-check holds at the reference and manifest geometries (1.5e-14 / 3.4e-14 here); an in-repository source withredistributable: falseis refused; plus the schema, path-confinement, silence, contract-version and stale-commit refusals.synthhand-run overen_US-kristin-medium(piper-tts 1.8.0 and onnxruntime 1.29.0 as an isolateduv tool, never imported): six TTS rows with every per-clip field,verify_splitsclean, card--checkpassing; the record isfixtures/synth_hand_run/(manifest and lock excerpt, no audio).tools/ml/kws/manifests/speech_commands_v2_bringup.json, second commit): Speech Commands v0.02 in full under its official split, MUSAN's noise partition, SLR28's 60,000 simulated RIRs, the keyword list; on the M0 Mac with--jobs 8: fetch verifies 14.8 GB, decode 166,758 clips 39.6 s, augment 96,729 variants 65.2 s, extract 200,653 rows / 27.5 M frames 49.6 s, shard 784 shards 227.6 s — 396 s in all, 2.3 GB peak.marvin1,710 / 195 / 195, negatives 83,133 / 9,786 / 10,810; eval-speech 3.0 h (zero-event bound 1.0 FA/h, as the plan predicted for Speech Commands);verify_splitsR1–R5 clean over 263,487 rows; card--checkpasses; the rendered card is committed beside the manifest, the 164 MB lock and 8.5 GB of tiers stay in the store.kws-datasetjob passed on ubuntu-latest in 1 m 20 s on the first push (the pinned wheels, the bridge build, the toy rebuild inside the 1e-6 pin).Notes for the reviewer
58210ae→5ca3b1c(Parameterize the numpy front-end reference and pin parity at a tuned geometry DspTap#16 merged; identical tree onmain). The pin'smake_frontend_reference.pyis what the self-check imports.include/moves. The plan's M4 text and HANDOFF's wake-word bullet record M4a's state.tools/ml/kws/fixtures/toy/(Speech Commands CC BY 4.0, MUSAN CC BY per file, SLR28 Apache-2.0), cut deterministically byfixtures/make_toy_fixture.py, withTHIRD_PARTY_NOTICES.mdpointing at the manifest's rows and the generatedATTRIBUTION.csv. No Common Voice. The expected outputs (fixtures/toy/expected/, 993 KB) are three float32 shards, the lock and the card of a real build.requirements.txtpins Python ≥ 3.12 because numpy 2.5.3 and scipy 1.18.1 require it; the plan's floor of 3.10 stays as the statement of intent..venv/is now git-ignored.🤖 Generated with Claude Code
https://claude.ai/code/session_016ZkpA4PWuaan5ek2CL2DDm