Skip to content

v0.2.1 - Nimble-Näbbdjur

Choose a tag to compare

View all tags
@SillyNickDev SillyNickDev released this 31 May 12:40
Nimble-Näbbdjur
48f828b

"Näbbdjur" is Swedish for platypus — a creature that looks improbable on paper but turns out to be remarkably capable in practice. Fitting for a release that makes the model smarter without touching its architecture.

This release focuses on making BrowSync fully aware of all its input sources. The ML model previously only knew what the face tracker told it; it now also understands head motion and speech prosody during training. Data collection grows from a single path (VRCFT WebSocket) to three — including a direct OpenXR recorder that captures raw Quest hardware values before VRCFT remaps them, with labeled brow ground truth for Quest Pro users.

Server

Mode hysteresis and crossfade

  • Source loss now requires 300ms of absence before the inference mode degrades — eliminates mode thrashing from momentary tracker dropouts
  • Mode transitions blend over 150ms using pre-smoother AU targets, removing the hard jump artifacts that occurred when switching between ml, rules_only, and mic_head

Head motion tracking: OpenVR backend

  • Replaced the OpenXR head motion backend with OpenVR (VRApplication_Background init mode)
  • SteamVR's OpenXR runtime requires a D3D11/Vulkan graphics context even for headless pose queries; OpenVR bypasses this entirely, making the tracker reliable for background use
  • Rotation matrix → quaternion conversion uses Shepperd's numerically stable method; existing Euler extraction and calibration pipeline unchanged

GRU residual median filter

  • A 5-frame rolling median is applied to the GRU residual output before it is added to the rule base, eliminating single-frame spikes without introducing lag

Per-session prosody normalization

  • Each recording session now maintains its own pitch and energy baseline using Welford's online mean/variance algorithm
  • Normalization blends progressively from global stats → session-relative over the first 300 frames (~3.3s), so cold-start frames are not distorted by the speaker's absolute pitch range
  • Microphone recalibration (r key / recalibrate_head message) resets session stats

Configurable smoother parameters

  • Spring-damper attack/decay rates and stiffness are now configurable per brow AU via config.json
  • Server loads config.json at startup; missing or malformed entries fall back to compiled defaults
  • No restart required to apply changes between sessions (reload on next server launch)

New control messages

  • get_calibration — returns calibration phase (settling / locked / degraded) and per-AU output variance over the last 3s; useful for diagnosing whether sources are stable
  • get_status — extended to include current calibration phase alongside mode and source flags

Session event logging

  • Structured JSONL event log written to logs/ for each server session
  • Records source connect/disconnect events, mode transitions, and session start/end
  • Crash-safe: each event is flushed immediately; an atexit handler closes the log on abnormal exit

VRCFT Plugin

Input forwarding

  • The module now sends current VRCFT eye and face expression data to the Python server as frame WebSocket messages — the same data path used during live inference
  • This enables the Python server to record labeled sessions without a separate capture tool, and ensures the inference server always has the latest tracker state
  • Fixed VRCFT enum name mismatches discovered via reflection: EyeClosedLeft/Right (→ derived from 1 - Openness), LipCornerPull (→ MouthCornerPull), LipCornerDepressor (→ MouthFrown), CheekRaiser (→ CheekSquint), LipStretch (→ MouthStretch), MouthOpen (→ JawOpen)

Mode awareness

  • The module reads the mode field from every server response
  • In noise_only mode (no active tracking sources), brow shapes are zeroed rather than applying the anti-freeze noise to the avatar

Connection reliability

  • Frame send queue changed to a capacity-1 Channel<string> with DropOldest policy — the module always sends the freshest data and never blocks behind a stale backlog
  • All WebSocket send operations serialized through a SemaphoreSlim to prevent concurrent-send exceptions

Training

Faster training loop

  • Rule estimates for each sequence's target frame are now pre-computed and cached at dataset construction time
  • Previously, the training loop re-ran RuleBasedEstimator.estimate() per sample, per batch, per epoch — approximately 3 million calls for a 60-session dataset at batch size 64
  • With caching, rule estimates are computed once at startup (~50K total) and reused across all epochs; remaining per-batch Python overhead is minimal
  • Batch size increased from 64 to 256; num_workers set to 0 (eliminates Windows process-spawn overhead for small in-memory datasets)
  • PyTorch thread count explicitly set to all available CPU cores at startup
  • torch.compile enabled on platforms with MSVC (cl.exe) available; skipped gracefully on Windows without it (Inductor backend requires MSVC to JIT-compile C++ kernels)

Synthetic Data

Full 52-feature coverage

  • The synthetic session generator now produces temporally coherent data for all 52 input features, including the 11 head motion and 6 prosody features that were previously zero
  • Head motion driven by a per-axis spring-damper (K=16, B=8, critically damped at ω₀≈4 rad/s) with expression-state targets; outputs all of HeadPitch/Roll/Yaw, HeadY/Z, angular/linear velocity and acceleration
  • Prosody driven by a speech dynamics model with syllabic energy bursts (~5–6/s), smooth VAD onset/offset, and intonation contours; produces first and second derivatives for PitchDelta2 and EnergyDelta2
  • Sign convention and normalization constants match the live OpenVR head tracker exactly, so synthetic and real sessions are directly interchangeable
  • New expression states: whisper, excited, pensive, recoil_surprise
  • Bug fix: head motion features were being clipped to [0, 1] in state_to_vector(); they live in [−1, 1] and HeadPitch=−0.20 (concentration posture) was silently zeroed

Data Collection

Three recording paths are now available:

data/record_session.py — VRCFT-based live recorder

  • Runs as a drop-in replacement for the inference server on port 7720
  • Receives eye/face frames from the C# module, merges microphone prosody and head motion, assembles full 52-feature frames, and writes unlabeled .jsonl sessions
  • Sends dummy zero brow responses so the C# module stays connected throughout
  • Live status line: elapsed time, frame count, frame rate, active sources

data/record_quest_raw.py — direct OpenXR recorder

  • Reads the 70 raw XR_FB_face_tracking2 expression weights from the Quest's face tracking hardware via Meta's OpenXR runtime, bypassing VRCFT and its UnifiedExpressions remapping entirely
  • Requires Meta Quest Link to be running and set as the active OpenXR runtime; face tracking enabled in Quest privacy settings
  • Quest Pro: brow AUs (INNER_BROW_RAISER, OUTER_BROW_RAISER, BROW_LOWERER) are present in the hardware output and saved as labeled targets (has_labels=True), giving full-weight ground truth for training
  • Quest 3: face data is captured unlabeled; rule pseudo-labels applied at training time
  • Also captures microphone prosody and SteamVR head motion in the same frame, so sessions are fully feature-complete
  • Session strategy: attempts XR_MND_headless first (no graphics context); falls back to a minimal D3D11 device created via ctypes if the runtime requires a graphics binding

data/convert_raw.py — offline converter

  • Converts already-captured recordings to BrowSync .jsonl training sessions
  • Accepts WebSocket frame log format (newline-delimited JSON with inputs dict) or feature CSV with schema.py column names
  • Injects delta features (eye openness delta, jaw delta) computed across frames
  • Batch directory conversion with configurable val split

data/synthetic/openface_converter.py — OpenFace 2.0 support (new in this milestone)

  • Converts OpenFace FeatureExtraction CSV output to labeled BrowSync sessions
  • Maps AU01/AU02/AU04 (brow AUs) to target vector with ground truth labels
  • Includes head pose from pose_Rx/Ry/Rz columns and eye/face features from AU intensities and gaze landmarks

Dependencies

  • openvr>=1.23.701 replaces pyopenxr for head motion (OpenVR background mode, no graphics context)
  • huggingface_hub>=0.20.0 added for HF Hub training data donation
  • pyopenxr>=1.0.3 added as optional dependency for record_quest_raw.py

Migration Notes

  • config.json is optional; the server runs with compiled defaults if it is absent. To customize smoother parameters, create config.json in the project root — see CLAUDE.md for the schema.
  • Sessions generated with the previous synthetic generator (head features all zero) can coexist with new sessions but will not teach the model head motion correlations. Re-generate synthetic data with python data/synthetic/generate_synthetic.py --sessions 60 before retraining.
  • The VRCFT module DLL version is 0.2.1; rebuild with dotnet build -c Release and replace the DLL in %APPDATA%\VRCFaceTracking\CustomLibs\.
Assets 2
Loading