Sensor Context Encoder

Can an inertial sensor be connected to a frozen language model through a single continuous context embedding — a learned vector spliced directly into the model's input-embedding sequence, never converted to text — and is that interface worth developing? This repo answers that with three conditions on the UCI HAR dataset, compared by macro-F1:

1. Direct sensor classifier — engineering baseline: encoder → linear head.
2. Context-embedding model — encoder → projector → frozen SmolLM2-360M → linear head. The projected vector replaces <SENSOR> in a fixed prompt; the hidden state after Activity: is read by a trainable head.
3. Sensor-dependence (shuffle) check — the trained context model re-evaluated with the projected test embeddings shuffled across examples (no retraining). A large F1 drop confirms the model actually reads the sensor input.

Results (seed 42)

Condition	Macro-F1
Direct sensor classifier	0.9291
Context-embedding model	0.9120
Context model with shuffled embeddings	0.1662

Seed 42. Numbers are written to results/results.json by the scripts below. The shuffled-embedding macro-F1 (0.1662) collapses to chance level (~0.167 for six balanced classes), a 0.746 drop — confirming the frozen LLM genuinely reads the injected sensor vector rather than exploiting a prompt/label shortcut.

Setup

pip install -r requirements.txt
python scripts/download_data.py      # downloads + verifies UCI HAR (~60 MB)

Python 3.9+, PyTorch, transformers, scikit-learn, numpy. A GPU is recommended for the context model (the direct classifier trains fine on CPU).

Reproduce

All three conditions use the same data splits, standardization and seed.

# Condition 1 — direct classifier (fast; runs locally, even on CPU)
python -m src.train_direct --seed 42

# Condition 2 — context-embedding model (GPU recommended)
python -m src.train_context --seed 42 --epochs 15 --batch-size 16

# Conditions 2 & 3 — re-evaluate + shuffled-embedding sensor-dependence check
python -m src.evaluate --seed 42

The final three-condition table is written to results/results.json under results_table.

Running the context model on Colab / a cloud GPU

The context model backprops through the frozen 360M LLM. On small local GPUs this is slow; open notebooks/context_colab.ipynb on a free T4 and run the cells (clone or upload this repo, install deps, download data, then run conditions 2 & 3). On GPU the backbone is loaded in a half dtype — bf16 when the card supports it (no loss scaling needed), otherwise fp16 with a GradScaler — while the trainable encoder/projector/head stay fp32, so the 360M model stays resident on ~2 GB cards.

How it works

• Data (src/data.py): only the nine raw inertial signals (total_acc_*, body_acc_*, body_gyro_*), stacked to (N, 128, 9). The supplied 561-D engineered features are not used. Per-channel standardization uses train-only statistics. Validation is a subject-wise hold-out from the training split (subjects {27,28,29,30}); the test split is never used for tuning.
• Sensor encoder (src/models.py, SensorEncoder): a compact 1D-CNN (9→64→128→128, stride-2 downsampling, global average pool → 128-D). The same architecture is used by both models (trained separately) so the comparison isolates the LLM interface, not encoder capacity.
• Projector (Projector): MLP 128→256→960 + LayerNorm + a learnable gain initialized to the mean token-embedding norm, producing one (1, 960) context vector kept in-distribution for the frozen model.
• Context model (ContextModel): builds inputs_embeds = [embed(prefix) | sensor_vec | embed(suffix)] from the exact challenge prompt, runs the frozen SmolLM2-360M-Instruct, and reads the last-position hidden state after Activity:. All LLM parameters (including the token-embedding table) are frozen; only encoder + projector + head are trained (599,431 params, well under the 10M budget). Gradients flow through the frozen LLM to the projector/encoder.

Repository layout

src/common.py        seeding, macro-F1, Config, device, param counting
src/data.py          UCI HAR inertial-signal loader, standardization, subject split
src/models.py        SensorEncoder, DirectClassifier, Projector, ContextModel
src/engine.py        shared train/eval loop (identical for both conditions)
src/train_direct.py  condition 1
src/train_context.py condition 2 (saves a small trainable-only checkpoint)
src/evaluate.py      conditions 2 & 3 (shuffle check) → results/results.json
scripts/download_data.py   fetch + verify the dataset
notebooks/context_colab.ipynb   T4 runner
technical_note.md    ≤2-page write-up

Notes

• Seed: 42 (recorded in results/results.json).
• Fair comparison: identical splits, standardization, encoder architecture, loss (cross-entropy), optimizer (AdamW + cosine), and early-stopping rule (val macro-F1) across conditions.