BrowSync v0.3.0 — Observant-Orm

BrowSync v0.3.0 — Observant-Orm

Downloads

Asset	What it is
browsync-model.zip	Trained ONNX model. Unzip both files into models/.
browsync-vrcft-module.zip	VRCFT plugin. Unzip the folder into %APPDATA%\VRCFaceTracking\CustomLibs\.

The model is trained on generated synthetic data and is reproducible
from this tag. It is a cold-start baseline — the rule-based estimator
alone still works if you skip it.

"Orm" is Swedish for snake — an animal that reads the world through vibration and heat rather than sight, and is famously difficult to sneak up on. Fitting for a release that is almost entirely about the system perceiving its own state correctly: knowing when it is calibrated, when its data is stale, when its output is going nowhere, and when it has no right to upload anything.

Almost everything in BrowSync was quietly broken in a way that produced no error message. Training could not learn — the objective was degenerate, so the model was mathematically obliged to output zero. Recording captured nothing, because both recorders called a method that does not exist and the exception was swallowed by a handler that only caught disconnects. Model output never reached the avatar, because the module silently returned early on a state VRCFT sets whenever another plugin owns expression. The head tracker reported itself calibrated-in-progress forever whenever the headset was asleep. The smoother's two most prominent tuning knobs did nothing at all.

None of these announced themselves. The server looked healthy, the TUI showed numbers moving, and nothing worked. This release fixes the causes and — more importantly — makes each of them say so out loud next time.

It also adds consented data donation, so the model can start learning from real faces instead of only synthetic ones.

---

The training pipeline could not learn. Below is why, and the input-distribution mismatches that would have kept a trained model from behaving in VRChat even once it could.

Training — correctness

The residual objective was degenerate (critical)

• generate_synthetic.py labelled every frame with RuleBasedEstimator, and train.py re-derived the same targets for unlabelled frames. The loss was (rule + 0.4*residual - target)^2 with target == rule, so the optimal residual was exactly zero everywhere — the GRU was being trained to output nothing
• Synthetic sessions now carry an independent ground-truth brow pose per expression state, derived from FACS co-occurrence rather than from the rule estimator. Several states (sadness/frown, pensive) encode AU1+AU4 blends the rule base structurally cannot express, since the rules make pinch oppose inner raise
• Added label_source to BrowFrame (quest_pro / openface / synthetic / rule) driving the loss weight. Rule pseudo-labels drop to 0.05 and act as a "stay near the baseline" anchor instead of the whole objective
• Training now warns and explains when every training frame is rule-labelled

Label noise

• RuleBasedEstimator carries a time-accumulating procedural noise phase, and train.py called it twice per sample at different phases without resetting between sessions, so target - baseline was a random ±0.025 jitter — the only signal left in the objective. Added deterministic=True and reset(), now used for all offline labelling

Train/serve baseline skew

• apply_head_motion_rules was applied only at inference. Head motion contributes up to ~1.0 to the inner brow, so residuals were learned against one baseline and applied to a different one. Added RuleBasedEstimator.estimate_baseline(), now used by both sides

The temporal smoothness loss was not temporal

• It diffed adjacent samples within a shuffled batch, which have no temporal relationship — a variance penalty pulling every prediction toward the batch mean, actively suppressing expressiveness. It now compares two genuinely consecutive predictions

Training — stability

• Output layer is zero-initialised, so training starts as an exact no-op on top of the rule baseline instead of emitting random ±0.4 residuals at epoch 0
• Added a residual L2 anchor (--residual-anchor, default 0.02) so out-of-distribution input degrades toward the deterministic baseline rather than producing large unconstrained corrections
• Weight EMA with warmup for the exported checkpoint. A fixed 0.999 decay left the average pinned near initialisation on short runs — fixing this took validation from +1.5% to +36% over the rule baseline
• Validation now reports per-AU MAE for the model and the rule baseline, and warns if the model fails to beat it
• Augmentation is applied per block so paired windows share a transformation; added tracker gain/offset jitter; removed the temporal-shift augmentation, which destroyed the consecutive-window relationship

Input distribution parity

Synthetic features were generated in ways live data never produces:

• Prosody now updates at 10 Hz and is held between updates, matching ANALYSIS_WINDOW = 0.10s. It was smooth at 90 Hz
• PitchDelta/EnergyDelta use the live definition — (newest - oldest) * 3.0 over a 10-sample history (~0.9s). They were an 11ms single-frame difference scaled by 9.0, a different quantity by two orders of magnitude
• IsSpeaking is binary, matching the live VAD threshold, rather than a smooth ramp
• PitchDelta2/EnergyDelta2 were never written by any live code path and sat at exactly 0.0 during inference while synthetic supplied real values. inject_frame_deltas now populates them, and the generator matches the definition

Synthetic data quality

• Expressions now have onset/hold/offset envelopes. Previously the generator interpolated across the entire state duration, so an expression never held — it was always mid-morph toward the next one, and the model never saw a sustained pose
• Per-session subject profiles: expressivity, resting brow height, L/R asymmetry, flash gain, lid baseline, tracker noise, head gain, audio latency, blink rate
• Left/right asymmetry throughout, plus a skeptical state with a unilateral brow raise. Everything was previously perfectly bilateral, so the model could never learn independent sides
• Blinks are fast overlay events instead of a Markov state that got smeared across 45+ frames by interpolation — the eye never actually closed
• Tracker realism: sensor noise, 8-bit quantisation, and brief tracking stalls
• Fixed signed-feature clipping that zeroed EmotionValence (-0.60 in frown) and GazeVertical (-0.20 in pensive) — the exact bug the previous docstring claimed to have fixed, still present for every non-head bipolar feature. SIGNED_FEATURES in schema.py is now the single source of truth
• Ground truth passes through brow-muscle dynamics and gains speech-emphasis flashes tied to syllabic energy bursts

Smoother

• attack_hz and decay_hz had no effect at all. omega was computed from them and then never used; the spring constant came from stiffness alone, so asymmetric attack/decay never worked and the entire config.json smoother section was a no-op
• Response is now derived as omega = 2*pi*f, making the frequencies real. Effective natural frequency was ~1.4 Hz regardless of configuration; brows are now noticeably more responsive (measured 2.43x attack/decay asymmetry, previously exactly 1.0)
• Sub-stepped integration so the spring cannot diverge at the 20fps dt clamp, where 8 Hz would have given omega*dt = 2.5
• Integrator state is clamped with velocity zeroed at the rails, preventing windup; loop vectorised (~16.5us/frame)

Data collection was entirely non-functional

• Both recorders recorded nothing. record_session.py and record_quest_raw.py called self._mic.get_features() / self._head.get_features(); neither method exists (the API is .latest.to_feature_dict() / .latest.to_array()). The AttributeError escaped the WebSocket handler — which caught only ConnectionClosedError — and dropped the client on the very first frame. Recording only ever worked with --no_mic --no_head. Verified fixed: 300/300 frames captured with both sources enabled
• The donation ingest path was dead code. _handle_donation fires only on labelled_frame messages carrying brow_ground_truth and consent_version; the C# module has never sent any of those. data/sessions/donated/ was therefore always empty and donate_to_hub could only ever report "No donation files to upload"
• Donated data was never trained on. train.py globbed only train/ and val/, so even a working donation flow produced data no run would read. TrainConfig.extra_train_dirs now includes donated/
• The recorder never flushed — only closing the file on a clean Ctrl+C, so any crash lost the whole session. SessionWriter flushes every 90 frames
• The recorder computed time.monotonic() - self._vrcft_ts / 1000.0, subtracting an epoch-millisecond timestamp from a monotonic clock. The result was meaningless and unused
• The recorder never wrote PitchDelta2/EnergyDelta2 despite a comment claiming the mic supplied them, reintroducing the train/serve skew fixed earlier. It now uses the server's own inject_frame_deltas
• convert_raw.py used a non-deterministic RuleBasedEstimator, baking time-varying procedural noise into pseudo-labels — the same bug fixed in the training path

Recording from the TUI

• Ctrl+R toggles recording while inference runs, so brows keep working on the avatar while data is captured. Previously recording meant running a separate script that replaced the server
• The tee writes the exact assembled vector inference consumed, deltas already injected — recording through a second assembly path is how training and serving drift apart
• Record indicator top-right: dim ○ Record when idle, blinking ● Recording mm:ss · N frames when active. Blink is driven by the panel's own timer, so it stays steady regardless of engine frame rate
• New data/session_writer.py is the single definition of a recorded frame, shared by the standalone recorder and the tee

Consented data donation

• New inference/consent.py. allows_upload requires an explicit grant and a matching CONSENT_VERSION; it defaults to opt-out and fails closed on an unreadable config. Verified with 8 gate cases including corrupt config and stale-version consent
• First-run TUI modal states plainly what is and is not collected (no audio, no video, no identifiers), rendered from the same constants as CONSENT_VERSION so text and version cannot drift. Shift+D reopens it to revoke at any time
• New ingest_space/ — a FastAPI Space that holds the dataset write token server-side. Contributors never need a token; shipping one would let anyone extract it and rewrite or delete the dataset
• Every frame is validated before anything is committed: exact vector lengths, all values finite, and each value within its declared per-feature range. One bad frame rejects the session. Rate limited per IP with a daily quota, and stored filenames are server-assigned so a client cannot overwrite existing data. 14/14 validation cases pass
• DRY_RUN=1 validates and acknowledges without storing, so a deployment can be verified before the token is attached
• New inference/donation.py — gzip + POST with bounded retry, no retry on 4xx. Failed uploads stay on disk and retry next launch; successful ones are renamed *.uploaded.jsonl so they are never re-sent
• Privacy fix caught in testing: the --subject tag was being written into session_id, which lives inside every frame and is echoed in the upload metadata — it would have been published to a public dataset. session_id is now an opaque UUID and the tag stays in the local filename only
• tests/test_ingest_schema_sync.py guards the constants the Space necessarily duplicates from schema.py. Writing them by hand got the signed-feature indices wrong on the first attempt (15, 17, 20 instead of 28, 30, 33), which would have rejected every legitimate negative PitchDelta, EnergyDelta and EmotionValence

Server

• Unhandled QueueFull at 90fps. call_soon_threadsafe(q.put_nowait, msg) schedules the put to run later, so the surrounding try/except guarded only the scheduling and never the deferred call. Any client stalling past ~110ms raised an unhandled exception inside the event loop on every subsequent frame. Now routed through a helper that drops the oldest entry — a late consumer should catch up on current values, not replay a backlog
• DONATION_DIR.mkdir() ran at import time, so importing the server for training or tests created directories as a side effect. Now resolved on demand
• recalibrate("mic") returned ready_in_ms or 10000, reporting a 10s wait precisely when the mic was already settled (0)
• Recording subject tags are sanitised against path traversal — verified that test user!!/../evil becomes testuserevil

Avatar output not reaching VRChat

Two independent causes, both of which produce "the server is clearly outputting but the avatar never moves":

• Update() was gated on Status == ModuleState.Active and returned early. VRCFT grants each capability to the first module that claims it, so with Project Babble (or any expression module) loaded first, BrowSync is offered expressionAvailable == false and gets marked Idle — but VRCFT keeps calling Update() anyway. The self-gate meant BrowSync silently wrote nothing forever while its WebSocket stayed connected and the TUI kept streaming values. The gate is gone; BrowSync writes only the eight brow shapes, and neither Babble (mouth) nor ETVR (eyes) produces brow data, so there is nothing to contend with
• The module was deployed as a loose DLL to the CustomLibs root, with no module.json. VRCFT 5.x discovers modules as CustomLibs\<ModuleId>\{dll, module.json} — the layout every registry-installed module uses. Added a manifest and fixed the deploy target, which also deletes the stale loose DLL so two copies can't run two WebSocket clients against the same shapes
• Module state transitions and the Initialize offer are now logged, so this is answerable from the VRCFT log instead of by inspection

Brow shape indices were verified by reflection against the bundled VRCFaceTracking.Core.dll v5.4.5.0 and are correct (4–11). The full Unified Expressions brow reference — including the Simple Expression blend weights (BrowUp = OuterUp×0.6 + InnerUp×0.4, BrowDown = Lowerer×0.75 + Pinch×0.25) — is now documented in BrowSyncModule/README.md.

Head calibration stuck on "calibrating"

• HeadMotionTracker.is_available returned True as soon as openvr.init() succeeded, which happens whenever SteamVR is running — even with the headset asleep on a desk. _process() was then never reached, so _calib_start stayed None, settling stayed True, and the TUI showed "calibrating" indefinitely with ready_in_ms reporting 0. Availability now requires a valid HMD pose within the last second
• CalibrationState.finalise() silently returned False below 30 samples, leaving complete unset and retrying every frame with no escape. Added a 10s timeout that locks in with whatever samples exist
• Added a distinct waiting for HMD pose state so the UI can say why: the TUI now shows "waiting for HMD pose (headset asleep?)" or "unavailable (SteamVR not running?)" instead of a misleading "calibrating"
• waiting_for_pose reports as degraded rather than settling, since no amount of waiting resolves it
• The server logs once when SteamVR is up but no valid pose is arriving, instead of silently feeding zeroed head features into the rule base

VRCFT module (C#)

Brought in line with the current server and schema after a long gap.

• MouthOpen was a byte-for-byte duplicate of JawOpen — CollectInputs passed W(JawOpenIdx) for both, so one of the 52 model inputs carried no independent information. Now JawOpen * (1 - MouthClosed), the actual lip aperture
• BlinkLeft/BlinkRight were sent as 1 - Openness, but the schema defines them as a blink event and the training data generates them that way. A user with a resting openness of 0.65 reported a constant blink of 0.35, which the rule estimator's blink_suppress_raise turned into a permanent brow-raise damping that varied per headset fit. Replaced with a self-calibrating BlinkDetector: measured 0.000 at rest for resting openness 0.55/0.65/0.80, peaking 0.92 during a 120ms blink
• Frozen brows on server stall. Output had no expiry, so if the inference thread stopped while the socket stayed open, the last values stayed applied forever. Added a 400ms staleness timeout mirroring the server's own 0.5s input expiry
• Brows now fade to neutral over 200ms on disconnect / stale / noise_only instead of snapping to zero
• EyeLidTightener documented as intentionally zero — Unified Expressions has no such shape, and synthetic data leaves it zero too, so deriving one here would create a skew
• Exponential reconnect backoff (1s → 15s), replacing a fixed 3s retry
• Version is read from the assembly; the hardcoded v0.3.0 string had drifted from the project version

Build

• VRCFaceTracking.Core resolved only from $(APPDATA)\VRCFaceTracking, which does not exist without VRCFT installed — the build worked by accident because MSBuild resolved the name out of libs/. Now explicitly prefers libs/ with an installed-VRCFT fallback
• Replaced Microsoft.Extensions.Logging 10.0.8 with Microsoft.Extensions.Logging.Abstractions (all the module uses), dropping four unused transitive packages. Version still matches what VRCFT binds against — dropping to 7.0.x to match the TFM looks tidier but would disagree with the host process
• Build warnings 8 → 1, with the remaining System.Text.Json conflict documented in the .csproj

Hot path

• Utf8JsonWriter and ArrayBufferWriter are cached per thread instead of allocated on every frame at 90Hz
• Inbound messages parse directly from UTF-8, removing two string allocations per message
• Module status text only written when it changes

Fixes

• ONNX export crashed on Windows. torch.onnx prints a U+2705 on success and stock cp1252 stdout cannot encode it, so a full training run completed and then died without ever writing models/browsync.onnx. train.py and the generator now force UTF-8 output
• residual_scale was hardcoded in three places; it is now written into ONNX metadata at export and read back by the server
• The inference buffer is primed with the first real frame instead of 30 frames of zeros, removing ~333ms of reacting to a fabricated pose after every connect or reset
• BrowSequenceDataset stores sessions once and slices windows, instead of duplicating each frame 30x (~340MB → ~34MB for 60 sessions)
• Removed per-feature re-imports inside apply_head_motion_rules, which ran an import lookup per feature per frame at 90fps

---