Skip to content

v2.0.0

Latest
Latest

Choose a tag to compare

View all tags
@notsubash notsubash released this 22 Aug 15:54
v2.0.0
2cee6ff
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

What's Changed

New Contributors

Full Changelog: v1.0.0...v2.0.0

Changelog

[2.0.0] - 2026-08-22

First citable release of the har package. Replaces the student notebook workflow
(v1.0.0 git tag, June 2026) with a session-safe 20 Hz pipeline, subject-grouped
evaluation, frozen 51-subject reports, and a CPU FastAPI that scores one 5 s window.

Baseline for these notes: 369ae56 (installable skeleton, locked WISDM constants,
notebooks archived). Delivered as PRs #1–#6 (feat/package-setup through
feat/containerize). Package version in pyproject.toml is still 0.1.0 until
this bump.

Primary metric is macro-F1. Accuracy is secondary. Every public number names a
protocol and a config.

Headline

  • Same flattened 5 s phone windows: leaky Protocol A2 0.8925 vs GroupKFold
    Protocol B 0.2924. That drop is the leakage finding
    (configs/protocol_a2_phone_raw_flat_xgb.yaml vs
    configs/protocol_b_phone_raw_flat_xgb.yaml).
  • Honest 18-class number to cite: watch statistical XGBoost 0.7031 under
    5-fold GroupKFold (configs/protocol_b_watch_stat_xgb.yaml).
  • Phone statistical XGBoost is 0.3272 on the same protocol. Trees beat dummy,
    logreg, RF, and flattened raw, so no 1D CNN / TCN ships.
  • Served default is that watch bundle (ONNX trees; statistical features stay in
    Python). Phone windows against a watch bundle are 422.

Added

Data

  • Installable har package (src/ layout), frozen 18-class maps
    (CODE_TO_NAME, GROUP_OF; A–S skipping N).
  • WISDM parser that strips the trailing ; on z and splits subject-sensor
    files into activity runs (gap > 2 s or time reversal starts a new run).
  • Download CLI that skips when the extract sentinel exists. Zip stays out of git.
    Identity of this dump is Weiss row counts (15,630,426), not a pinned
    zip_sha256 yet.
  • Dataset audit: 51 × 18 × 4 coverage, missing cells, implied Hz modes
    (20 / 25 / 50 / 100). 35 empty cells. Student concat had 15,649,253
    rows; that extra is a loader artifact.
  • Repair: interpolate each session onto a shared 20 Hz grid, align accel and
    gyro by coverage intersection (not an exact-timestamp join), optional
    phone-accel reorient, optional start trim. Default is reorient: false,
    trim_start_s: 0.0.
  • Session-safe windows: 5.0 s / 1.0 s hop inside one
    (subject, activity, device) session. Never slide across identity boundaries.
  • Statistical features (104 dims on 6 channels) plus raw_flat for the student
    representation. Magnitude-only path is 32 dims.

Evaluation

  • Protocols: A1 (80-sample leaky clone), A2 (5 s leaky flatten), B (GroupKFold 5
    on subject_id), C (46/5 × 3 grouped holdout, not 51-fold LOSO). Protocol D
    (phone↔watch transfer) is specified and not run.
  • Metrics: accuracy, balanced accuracy, macro-F1, per-class F1, per-group F1
    (locomotion / posture / hand / eating).
  • Training CLI (python -m har.train --config ...) with MLflow tracking.
    Missing --config errors; it does not silently train default.yaml.
  • Honest ladder: dummy, logreg, RF, XGBoost on phone statistical features;
    watch statistical XGBoost; concat as stacked 6-channel windows (not 12-channel
    fusion).
  • Ablations on Protocol B phone statistical XGBoost: window 2 s / 10 s, 15 s
    trim, reorient, magnitude, hierarchical group-then-expert.
  • Frozen JSON under docs/reports/ plus README figures from
    python -m har.eval.plots. Compact ladder via
    python -m har.evaluate --from-reports docs/reports.

Serving

  • Export: python -m har.models.export writes ONNX + sidecar JSON (joblib
    remains a stub / non-XGBoost fallback). Export refuses device: both and
    hierarchical, and forces device: cpu even if the YAML says cuda.
  • FastAPI: GET /health, GET /labels, POST /predict. Wrong T, C,
    device, or hz is 422. Bodies over 1 MiB are 413. Abstain is
    max(proba) < threshold (default 0.0, never abstain; uncalibrated).
  • Inference-only Docker image (python:3.13-slim). Does not install MLflow,
    XGBoost, or pyarrow. Mount $PWD/models:/models.

Tooling

  • GitHub Actions: Python 3.13, pip install -e ".[dev]", ruff, pytest on
    committed fixtures. No WISDM download in CI.
  • Makefile: install, test, audit, prepare, train, eval, figures,
    serve. Default CONFIG is watch statistical XGBoost.
  • Docs: docs/protocol.md, docs/data_card.md, docs/limitations.md,
    docs/model_card.md, serving/README.md. README rewritten as the case study.

Changed

  • Student notebooks live under notebooks/archive/ and are not the training
    path. Student 0.8559 accuracy lives next to them, not in docs/reports/.
  • Config names are protocol_{rung}_{device}_{features}_{model}.yaml. Ablations
    write docs/reports/ablations/<stem>.json so they do not pollute
    ladder_summary.json.
  • Honest B/C XGBoost is 200 trees, max_depth 6. Student 982-tree family is
    Protocol A and the B raw-flatten leakage pair only.
  • pyarrow pinned to 19.0.1 (MLflow 2.22.5 requires <20). Runtime
    onnx==1.22.0 (1.17 had no wheel).
  • CI Python aligned to 3.13 with the serve image.

Fixed

  • Parser no longer glues ; onto every z value.
  • Failed / truncated WISDM zip is deleted so the next download can retry
    (zip_sha256 is still null).
  • Empty or nested-wrong --raw-root no longer overwrites docs/data_card.md
    with a zero grid and exit 0.
  • Hierarchical experts remap local 0..K-1 so locomotion class M (label 12)
    does not crash XGBoost.
  • src/har/models/export.py is tracked (was gitignored), so serving tests can
    import it.
  • Makefile kept LF via .gitattributes so Windows checkout does not break GNU
    make.

Removed / not shipped

  • TensorFlow / PyTorch. No src/har/models/tcn.py, no configs/phone_tcn.yaml.
  • 12-channel time-aligned phone+watch fusion. Concat is extra 6-channel rows.
  • 51-fold LOSO as a shipped config (helper remains in code).
  • Hardware transfer (Protocol D).
  • Calibrated abstain. Default threshold stays 0.0.

Frozen results (full 51-subject UCI 507, repaired 20 Hz)

Cell Config macro-F1
Student notebook (leaky, unrepaired) notebooks/archive/student_evaluation.txt accuracy 0.8559 (macro-F1 not reported)
A1 leaky, 80-sample flatten protocol_a1_phone_raw_flat_xgb 0.8490
A2 leaky, 5 s flatten protocol_a2_phone_raw_flat_xgb 0.8925
B GroupKFold, same 5 s flatten protocol_b_phone_raw_flat_xgb 0.2924
B phone statistical dummy protocol_b_phone_stat_dummy 0.0151
B phone statistical logreg protocol_b_phone_stat_logreg 0.2767
B phone statistical RF protocol_b_phone_stat_rf 0.3131
B phone statistical XGBoost protocol_b_phone_stat_xgb 0.3272
B watch statistical XGBoost protocol_b_watch_stat_xgb 0.7031
B concat statistical XGBoost protocol_b_concat_stat_xgb 0.5236
C phone statistical XGBoost (46/5 × 3) protocol_c_phone_stat_xgb 0.2985

Ablations (phone statistical XGBoost, Protocol B): 10 s windows 0.3422 (only
knob that clearly beats 5 s control); hierarchical 0.3271 (eating group F1
0.5855 vs 0.4945, still not an 18-way win); reorient 0.3230; trim 15 s 0.3247.
Defaults stay 5 s, unreoriented, untrimmed, flat 18-way.

Watch sandwich (L) is the hard served class (per-class F1 0.2816). Phone eating
per-class F1 is 0.07–0.11. Do not serve a phone window to a watch bundle.

Known limits in this tag

  • Served ONNX is a refit on all windows (one subject held out only for early
    stopping). Cite GroupKFold from the metrics JSON, not from export.
  • Statistical features are still Python; only the tree head is ONNX.
  • No subject demographics, so no fairness slice.
  • 10 s windows beat 5 s on phone GroupKFold by about 1.5 macro-F1 points.
    Changing the default is a product choice, not a free latency win.
  • p95 2.7 ms is FastAPI TestClient on this CPU, not Docker/uvicorn.

Upgrade notes

From the June notebook tree: stop concatenating raw files, stop windowing by
row count on mixed Hz, stop shuffling windows for the number you publish.
make install && make audit && make prepare && make train && make eval.
Full 51-subject XGBoost is overnight local, not CI.

Contributors

notsubash
Assets 2
Loading