BrowSync v0.3.0 — Observant-Orm

BrowSync v0.3.0 — Observant-Orm

Downloads

Asset	What it is
browsync-model.zip	Trained ONNX model. Unzip both files into models/.
browsync-vrcft-module.zip	VRCFT plugin. Unzip the folder into %APPDATA%\VRCFaceTracking\CustomLibs\.

The model is trained on generated synthetic data and is reproducible
from this tag. It is a cold-start baseline — the rule-based estimator
alone still works if you skip it.

"Orm" is Swedish for snake — an animal that reads the world through vibration and heat rather than sight, and is famously difficult to sneak up on. Fitting for a release that is almost entirely about the system perceiving its own state correctly: knowing when it is calibrated, when its data is stale, when its output is going nowhere, and when it has no right to upload anything.

Almost everything in BrowSync was quietly broken in a way that produced no error message. Training could not learn — the objective was degenerate, so the model was mathematically obliged to output zero. Recording captured nothing, because both recorders called a method that does not exist and the exception was swallowed by a handler that only caught disconnects. Model output never reached the avatar, because the module silently returned early on a state VRCFT sets whenever another plugin owns expression. The head tracker reported itself calibrated-in-progress forever whenever the headset was asleep. The smoother's two most prominent tuning knobs did nothing at all.

None of these announced themselves. The server looked healthy, the TUI showed numbers moving, and nothing worked. This release fixes the causes and — more importantly — makes each of them say so out loud next time.

It also adds consented data donation, so the model can start learning from real faces instead of only synthetic ones.

---

The training pipeline could not learn. Below is why, and the input-distribution mismatches that would have kept a trained model from behaving in VRChat even once it could.

Training — correctness

The residual objective was degenerate (critical)

• generate_synthetic.py labelled every frame with RuleBasedEstimator, and train.py re-derived the same targets for unlabelled frames. The loss was (rule + 0.4*residual - target)^2 with target == rule, so the optimal residual was exactly zero everywhere — the GRU was being trained to output nothing
• Synthetic sessions now carry an independent ground-truth brow pose per expression state, derived from FACS co-occurrence rather than from the rule estimator. Several states (sadness/frown, pensive) encode AU1+AU4 blends the rule base structurally cannot express, since the rules make pinch oppose inner raise
• Added label_source to BrowFrame (quest_pro / openface / synthetic / rule) driving the loss weight. Rule pseudo-labels drop to 0.05 and act as a "stay near the baseline" anchor instead of the whole objective
• Training now warns and explains when every training frame is rule-labelled

Label noise

• RuleBasedEstimator carries a time-accumulating procedural noise phase, and train.py called it twice per sample at different phases without resetting between sessions, so target - baseline was a random ±0.025 jitter — the only signal left in the objective. Added deterministic=True and reset(), now used for all offline labelling

Train/serve baseline skew

• apply_head_motion_rules was applied only at inference. Head motion contributes up to ~1.0 to the inner brow, so residuals were learned against one baseline and applied to a different one. Added RuleBasedEstimator.estimate_baseline(), now used by both sides

The temporal smoothness loss was not temporal

• It diffed adjacent samples within a shuffled batch, which have no temporal relationship — a variance penalty pulling every prediction toward the batch mean, actively suppressing expressiveness. It now compares two genuinely consecutive predictions

Training — stability

• Output layer is zero-initialised, so training starts as an exact no-op on top of the rule baseline instead of emitting random ±0.4 residuals at epoch 0
• Added a residual L2 anchor (--residual-anchor, default 0.02) so out-of-distribution input degrades toward the deterministic baseline rather than producing large unconstrained corrections
• Weight EMA with warmup for the exported checkpoint. A fixed 0.999 decay left the average pinned near initialisation on short runs — fixing this took validation from +1.5% to +36% over the rule baseline
• Validation now reports per-AU MAE for the model and the rule baseline, and warns if the model fails to beat it
• Augmentation is applied per block so paired windows share a transformation; added tracker gain/offset jitter; removed the temporal-shift augmentation, which destroyed the consecutive-window relationship

Input distribution parity

Synthetic features were generated in ways live data never produces:

• Prosody now updates at 10 Hz and is held between updates, matching ANALYSIS_WINDOW = 0.10s. It was smooth at 90 Hz
• PitchDelta/EnergyDelta use the live definition — (newest - oldest) * 3.0 over a 10-sample history (~0.9s). They were an 11ms single-frame difference scaled by 9.0, a different quantity by two orders of magnitude
• IsSpeaking is binary, matching the live VAD threshold, rather than a smooth ramp
• PitchDelta2/EnergyDelta2 were never written by any live code path and sat at exactly 0.0 during inference while synthetic supplied real values. inject_frame_deltas now populates them, and the generator matches the definition

Synthetic data quality

• Expressions now have onset/hold/offset envelopes. Previously the generator interpolated across the entire state duration, so an expression never held — it was always mid-morph toward the next one, and the model never saw a sustained pose
• Per-session subject profiles: expressivity, resting brow height, L/R asymmetry, flash gain, lid baseline, tracker noise, head gain, audio latency, blink rate
• Left/right asymmetry throughout, plus a skeptical state with a unilateral brow raise. Everything was previously perfectly bilateral, so the model could never learn independent sides
• Blinks are fast overlay events instead of a Markov state that got smeared across 45+ frames by interpolation — the eye never actually closed
• Tracker realism: sensor noise, 8-bit quantisation, and brief tracking stalls
• Fixed signed-feature clipping that zeroed EmotionValence (-0.60 in frown) and GazeVertical (-0.20 in pensive) — the exact bug the previous docstring claimed to have fixed, still present for every non-head bipolar feature. SIGNED_FEATURES in schema.py is now the single source of truth
• Ground truth passes through brow-muscle dynamics and gains speech-emphasis flashes tied to syllabic energy bursts

Smoother

• attack_hz and decay_hz had no effect at all. omega was computed from them and then never used; the spring constant came from stiffness alone, so asymmetric attack/decay never worked and the entire config.json smoother section was a no-op
• Response is now derived as omega = 2*pi*f, making the frequencies real. Effective natural frequency was ~1.4 Hz regardless of configuration; brows are now noticeably more responsive (measured 2.43x attack/decay asymmetry, previously exactly 1.0)
• Sub-stepped integration so the spring cannot diverge at the 20fps dt clamp, where 8 Hz would have given omega*dt = 2.5
• Integrator state is clamped with velocity zeroed at the rails, preventing windup; loop vectorised (~16.5us/frame)

Data collection was entirely non-functional

• Both recorders recorded nothing. record_session.py and record_quest_raw.py called self._mic.get_features() / self._head.get_features(); neither method exists (the API is .latest.to_feature_dict() / .latest.to_array()). The AttributeError escaped the WebSocket handler — which caught only ConnectionClosedError — and dropped the client on the very first frame. Recording only ever worked with --no_mic --no_head. Verified fixed: 300/300 frames captured with both sources enabled
• The donation ingest path was dead code. _handle_donation fires only on labelled_frame messages carrying brow_ground_truth and consent_version; the C# module has never sent any of those. data/sessions/donated/ was therefore always empty and donate_to_hub could only ever report "No donation files to upload"
• Donated data was never trained on. train.py globbed only train/ and val/, so even a working donation flow produced data no run would read. TrainConfig.extra_train_dirs now includes donated/
• The recorder never flushed — only closing the file on a clean Ctrl+C, so any crash lost the whole session. SessionWriter flushes every 90 frames
• The recorder computed time.monotonic() - self._vrcft_ts / 1000.0, subtracting an epoch-millisecond timestamp from a monotonic clock. The result was meaningless and unused
• The recorder never wrote PitchDelta2/EnergyDelta2 despite a comment claiming the mic supplied them, reintroducing the train/serve skew fixed earlier. It now uses the server's own inject_frame_deltas
• convert_raw.py used a non-deterministic RuleBasedEstimator, baking time-varying procedural noise into pseudo-labels — the same bug fixed in the training path

Recording from the TUI

• Ctrl+R toggles recording while inference runs, so brows keep working on the avatar while data is captured. Previously recording meant running a separate script that replaced the server
• The tee writes the exact assembled vector inference consumed, deltas already injected — recording through a second assembly path is how training and serving drift apart
• Record indicator top-right: dim ○ Record when idle, blinking ● Recording mm:ss · N frames when active. Blink is driven by the panel's own timer, so...

v0.2.1 - Nimble-Näbbdjur

"Näbbdjur" is Swedish for platypus — a creature that looks improbable on paper but turns out to be remarkably capable in practice. Fitting for a release that makes the model smarter without touching its architecture.

This release focuses on making BrowSync fully aware of all its input sources. The ML model previously only knew what the face tracker told it; it now also understands head motion and speech prosody during training. Data collection grows from a single path (VRCFT WebSocket) to three — including a direct OpenXR recorder that captures raw Quest hardware values before VRCFT remaps them, with labeled brow ground truth for Quest Pro users.

---

Server

Mode hysteresis and crossfade

• Source loss now requires 300ms of absence before the inference mode degrades — eliminates mode thrashing from momentary tracker dropouts
• Mode transitions blend over 150ms using pre-smoother AU targets, removing the hard jump artifacts that occurred when switching between ml, rules_only, and mic_head

Head motion tracking: OpenVR backend

• Replaced the OpenXR head motion backend with OpenVR (VRApplication_Background init mode)
• SteamVR's OpenXR runtime requires a D3D11/Vulkan graphics context even for headless pose queries; OpenVR bypasses this entirely, making the tracker reliable for background use
• Rotation matrix → quaternion conversion uses Shepperd's numerically stable method; existing Euler extraction and calibration pipeline unchanged

GRU residual median filter

• A 5-frame rolling median is applied to the GRU residual output before it is added to the rule base, eliminating single-frame spikes without introducing lag

Per-session prosody normalization

• Each recording session now maintains its own pitch and energy baseline using Welford's online mean/variance algorithm
• Normalization blends progressively from global stats → session-relative over the first 300 frames (~3.3s), so cold-start frames are not distorted by the speaker's absolute pitch range
• Microphone recalibration (r key / recalibrate_head message) resets session stats

Configurable smoother parameters

• Spring-damper attack/decay rates and stiffness are now configurable per brow AU via config.json
• Server loads config.json at startup; missing or malformed entries fall back to compiled defaults
• No restart required to apply changes between sessions (reload on next server launch)

New control messages

• get_calibration — returns calibration phase (settling / locked / degraded) and per-AU output variance over the last 3s; useful for diagnosing whether sources are stable
• get_status — extended to include current calibration phase alongside mode and source flags

Session event logging

• Structured JSONL event log written to logs/ for each server session
• Records source connect/disconnect events, mode transitions, and session start/end
• Crash-safe: each event is flushed immediately; an atexit handler closes the log on abnormal exit

---

VRCFT Plugin

Input forwarding

• The module now sends current VRCFT eye and face expression data to the Python server as frame WebSocket messages — the same data path used during live inference
• This enables the Python server to record labeled sessions without a separate capture tool, and ensures the inference server always has the latest tracker state
• Fixed VRCFT enum name mismatches discovered via reflection: EyeClosedLeft/Right (→ derived from 1 - Openness), LipCornerPull (→ MouthCornerPull), LipCornerDepressor (→ MouthFrown), CheekRaiser (→ CheekSquint), LipStretch (→ MouthStretch), MouthOpen (→ JawOpen)

Mode awareness

• The module reads the mode field from every server response
• In noise_only mode (no active tracking sources), brow shapes are zeroed rather than applying the anti-freeze noise to the avatar

Connection reliability

• Frame send queue changed to a capacity-1 Channel<string> with DropOldest policy — the module always sends the freshest data and never blocks behind a stale backlog
• All WebSocket send operations serialized through a SemaphoreSlim to prevent concurrent-send exceptions

---

Training

Faster training loop

• Rule estimates for each sequence's target frame are now pre-computed and cached at dataset construction time
• Previously, the training loop re-ran RuleBasedEstimator.estimate() per sample, per batch, per epoch — approximately 3 million calls for a 60-session dataset at batch size 64
• With caching, rule estimates are computed once at startup (~50K total) and reused across all epochs; remaining per-batch Python overhead is minimal
• Batch size increased from 64 to 256; num_workers set to 0 (eliminates Windows process-spawn overhead for small in-memory datasets)
• PyTorch thread count explicitly set to all available CPU cores at startup
• torch.compile enabled on platforms with MSVC (cl.exe) available; skipped gracefully on Windows without it (Inductor backend requires MSVC to JIT-compile C++ kernels)

---

Synthetic Data

Full 52-feature coverage

• The synthetic session generator now produces temporally coherent data for all 52 input features, including the 11 head motion and 6 prosody features that were previously zero
• Head motion driven by a per-axis spring-damper (K=16, B=8, critically damped at ω₀≈4 rad/s) with expression-state targets; outputs all of HeadPitch/Roll/Yaw, HeadY/Z, angular/linear velocity and acceleration
• Prosody driven by a speech dynamics model with syllabic energy bursts (~5–6/s), smooth VAD onset/offset, and intonation contours; produces first and second derivatives for PitchDelta2 and EnergyDelta2
• Sign convention and normalization constants match the live OpenVR head tracker exactly, so synthetic and real sessions are directly interchangeable
• New expression states: whisper, excited, pensive, recoil_surprise
• Bug fix: head motion features were being clipped to [0, 1] in state_to_vector(); they live in [−1, 1] and HeadPitch=−0.20 (concentration posture) was silently zeroed

---

Data Collection

Three recording paths are now available:

data/record_session.py — VRCFT-based live recorder

• Runs as a drop-in replacement for the inference server on port 7720
• Receives eye/face frames from the C# module, merges microphone prosody and head motion, assembles full 52-feature frames, and writes unlabeled .jsonl sessions
• Sends dummy zero brow responses so the C# module stays connected throughout
• Live status line: elapsed time, frame count, frame rate, active sources

data/record_quest_raw.py — direct OpenXR recorder

• Reads the 70 raw XR_FB_face_tracking2 expression weights from the Quest's face tracking hardware via Meta's OpenXR runtime, bypassing VRCFT and its UnifiedExpressions remapping entirely
• Requires Meta Quest Link to be running and set as the active OpenXR runtime; face tracking enabled in Quest privacy settings
• Quest Pro: brow AUs (INNER_BROW_RAISER, OUTER_BROW_RAISER, BROW_LOWERER) are present in the hardware output and saved as labeled targets (has_labels=True), giving full-weight ground truth for training
• Quest 3: face data is captured unlabeled; rule pseudo-labels applied at training time
• Also captures microphone prosody and SteamVR head motion in the same frame, so sessions are fully feature-complete
• Session strategy: attempts XR_MND_headless first (no graphics context); falls back to a minimal D3D11 device created via ctypes if the runtime requires a graphics binding

data/convert_raw.py — offline converter

• Converts already-captured recordings to BrowSync .jsonl training sessions
• Accepts WebSocket frame log format (newline-delimited JSON with inputs dict) or feature CSV with schema.py column names
• Injects delta features (eye openness delta, jaw delta) computed across frames
• Batch directory conversion with configurable val split

data/synthetic/openface_converter.py — OpenFace 2.0 support (new in this milestone)

• Converts OpenFace FeatureExtraction CSV output to labeled BrowSync sessions
• Maps AU01/AU02/AU04 (brow AUs) to target vector with ground truth labels
• Includes head pose from pose_Rx/Ry/Rz columns and eye/face features from AU intensities and gaze landmarks

---

Dependencies

• openvr>=1.23.701 replaces pyopenxr for head motion (OpenVR background mode, no graphics context)
• huggingface_hub>=0.20.0 added for HF Hub training data donation
• pyopenxr>=1.0.3 added as optional dependency for record_quest_raw.py

---

Migration Notes

• config.json is optional; the server runs with compiled defaults if it is absent. To customize smoother parameters, create config.json in the project root — see CLAUDE.md for the schema.
• Sessions generated with the previous synthetic generator (head features all zero) can coexist with new sessions but will not teach the model head motion correlations. Re-generate synthetic data with python data/synthetic/generate_synthetic.py --sessions 60 before retraining.
• The VRCFT module DLL version is 0.2.1; rebuild with dotnet build -c Release and replace the DLL in %APPDATA%\VRCFaceTracking\CustomLibs\.

BrowSync v0.1.5 — Lively Lodjur

Initial Release

BrowSync is a real-time eyebrow tracking system for VRChat that estimates VRCFT Unified Expression brow parameters without requiring a Quest Pro headset. It uses a hybrid rule-based + ML approach, combining eye tracking, lower face tracking, and microphone prosody to drive expressive brow animation at 90fps.

---

Features

Inference Pipeline

• Hybrid architecture: deterministic rule base (RuleBasedEstimator) plus a lightweight GRU residual model (~15K parameters) for learned corrections
• 52-feature input schema spanning eye/face tracking, microphone prosody, head motion, and computed deltas
• 8 VRCFT Unified Expression brow output AUs, all clamped to [0, 1]
• Spring-damper smoother with asymmetric attack/decay (raises fast, lowers slow) per AU

Automatic Mode Fallback
The server selects the best available inference mode at runtime and degrades gracefully as sources go offline:

Mode	Sources active
ml	Eye + face + mic + head + GRU model
rules_only	Eye + face + mic + head
mic_head	Mic + head only
head_only	Head motion only
noise_only	Procedural anti-freeze noise

Input Sources

• VRCFT eye/face tracking (14 eye + 13 face features, WebSocket push)
• Microphone prosody via librosa: pitch, energy, speech rate, speaking detection
• Head motion via OpenXR: pitch/roll/yaw, translation, velocity, acceleration — self-calibrating from first 2.5s of data
• Optional SpeechBrain emotion context (SER)

Server

• Async WebSocket server on port 7720 (ws_server/)
• 90fps inference clock on a background thread
• Control messages: ping, reset, recalibrate_head, set_mode, get_status
• ONNX model is self-contained with embedded normalization stats

TUI

• Textual-based terminal UI with live per-AU bar meters, FPS counter, source status indicators, and color-coded mode display
• Keys: q quit, r recalibrate head, Ctrl+L dev log

VRCFT Plugin (BrowSyncModule/)

• C# net7.0 plugin; drop-in install to %APPDATA%\VRCFaceTracking\CustomLibs\
• Persistent WebSocket connection with auto-reconnect (3s backoff) and 5s ping keepalive
• Writes only brow shapes — leaves eye and lower-face tracking to your existing VRCFT modules
• Zeroes brow shapes on disconnect; sends reset on reconnect to clear the GRU buffer

Training

• Supervised training from labelled .jsonl session files with unlabelled pseudo-label support (0.25 weight)
• Custom loss: MSE + temporal smoothness penalty + asymmetric raise/lower weighting
• Exports to ONNX with embedded normalization for zero-config deployment

---

Known Limitations

• Head motion tracking requires an OpenXR runtime to be active
• VRCFT data expires after 0.5s — a slow tracker will cause fallback mode switching
• The GRU residual scale (0.4×) and sequence length (30 frames) are fixed; changing either requires retraining
• No GUI installer — manual setup required (see README)

---

Getting Started

pip install -r requirements.txt
python -m ws_server.server --model models/browsync.onnx

Then build and install the VRCFT plugin:

cd BrowSyncModule && dotnet build -c Release