v1.2.0-preview.9 - Pulsar

Release Notes - 1.2.0

Release Date: March 2026
Code Name: "Pulsar"

---

🎙️ Overview

"Pulsar" introduces the Wyoming Voice Platform — a full local speech pipeline that turns Lucia into a Wyoming protocol-compatible voice satellite for Home Assistant. This release delivers streaming speech-to-text, speaker verification, wake word detection, speech enhancement, a multi-engine model management system, and a new dashboard Voice Platform page for configuring everything from one surface. In addition, we introduce the Conversation Command Parser — a fast-path processing pipeline that pattern-matches common voice commands (lights, climate, scenes) and executes them directly against Home Assistant, bypassing the LLM entirely for sub-50ms response times. We also introduce user-facing personalization to Lucia's orchestration layer, letting users define a personality prompt that rewrites all agent responses through an LLM before delivery.

🚀 Highlights

• Wyoming Protocol Server — Native TCP server implementing the Home Assistant Wyoming satellite protocol for real-time voice processing.
• Multi-Engine STT Pipeline — Hybrid streaming STT with progressive re-transcription, Sherpa streaming, Sherpa offline (Parakeet TDT/CTC), and IBM Granite 4.0 1B Speech ONNX engines.
• Speaker Verification & Profiling — Cosine-similarity speaker identification with enrolled profiles, provisional auto-discovery, adaptive profile updates, and guided voice enrollment onboarding.
• ONNX Auto-Detection — Automatic GPU/accelerator selection (CUDA, ROCm, OpenVINO, DirectML, CoreML) across all six inference engines — no manual configuration required.
• GTCRN Speech Enhancement — Real-time streaming noise reduction for cleaner audio in noisy environments.
• Voice Platform Dashboard — New unified control room for model management, speaker profiles, wake words, engine status, and real-time session monitoring.
• Personality Prompt — Configurable system prompt that rewrites the fan-in aggregated response through an LLM, giving Lucia a customizable personality (pirate speak, formal assistant, casual friend — you name it).
• Conversation Command Parser — New POST /api/conversation endpoint with pattern-matching pipeline that executes common smart home commands (lights, climate, scenes) directly via skills — zero LLM latency for recognized commands, SSE-streamed LLM fallback for everything else.
• Response Templates — Customizable response templates stored in MongoDB with {placeholder} interpolation. Manage templates per skill/action from the dashboard with guided dropdowns and token insertion buttons.
• Home Assistant Component v1.2 — Simplified integration migrated from A2A JSON-RPC to structured REST. No more agent catalog selection — just point to the host and go.
• Separate Model Support — Personality rewriting can use a different (cheaper/faster) model than the orchestrator, selectable from a dropdown of configured chat-type providers.
• Zero-Cost Opt-In — When no personality is configured, the pipeline is unchanged — no LLM call, no added latency.

✨ Features

Wyoming Protocol Server

• Full Wyoming satellite protocol implementation (audio-start, audio-chunk, audio-stop, transcribe, transcript, detect, not-detected)
• Persistent TCP connections with per-session state management
• Zeroconf/mDNS service advertisement for automatic Home Assistant discovery
• VAD-driven voice activity detection with configurable speech/silence thresholds
• Wake word detection with custom phrase support and per-speaker calibration

Multi-Engine Speech-to-Text

• HybridSttEngine — Streams audio through a lightweight online model, then re-transcribes the full utterance with a high-accuracy offline model for best-of-both-worlds latency and accuracy
• SherpaSttEngine — Pure streaming CTC/transducer inference for ultra-low-latency transcription
• SherpaOfflineSttEngine — Offline NeMo Parakeet TDT+CTC support for high-quality batch transcription
• GraniteOnnxEngine — IBM Granite 4.0 1B Speech ONNX with 3-model pipeline (audio encoder, embed tokens, auto-regressive decoder with 40-layer KV cache)
• Progressive re-transcription with burst detection and stability-based early stopping
• Per-engine model catalog with download, install, activate, and delete lifecycle

Speaker Verification & Voice Profiles

• Sherpa-onnx speaker embedding extraction with cosine similarity matching
• Enrolled speaker profiles with configurable verification threshold
• Provisional auto-discovery profiles for unknown speakers with interaction tracking
• Adaptive profile updates (exponential moving average) for high-confidence matches
• Guided onboarding flow with audio quality validation (SNR, duration checks)
• Profile merge, rename, and provisional-to-enrolled promotion via API and dashboard
• Audio clip capture, playback, reassignment, and per-profile management
• Redis-cached enrolled profiles for sub-millisecond lookup latency
• MongoDB-backed persistent storage with in-memory fallback

ONNX Provider Auto-Detection

• OnnxProviderDetector singleton probes OrtEnv.Instance().GetAvailableProviders() at startup
• Automatic selection priority: CUDA → ROCm → OpenVINO → DirectML → CoreML → CPU
• Applied to all six engines: SherpaDiarization, HybridSTT, SherpaStt, SherpaOfflineSTT, GraniteOnnx, GtcrnSpeechEnhancer
• Detected provider exposed in /api/wyoming/status and displayed on the dashboard status card with GPU badge
• Graceful fallback — if an accelerated provider fails to initialize, falls through to CPU

Speech Enhancement

• GTCRN (Gated Temporal Convolutional Recurrent Network) streaming noise reduction
• Per-session enhancement with isolated overlap-add state
• Raw audio preserved separately for speaker verification to avoid spectral mismatch with enrollment data

Model Management System

• Centralized model catalog with architecture-aware compatibility filtering
• Background download with real-time progress tracking via SSE
• Multi-stage progress bars (download → extract → validate)
• Hot-reload model activation via ActiveModelChanged events — no restart required
• Per-engine model views (STT, VAD, Wake Word, Speaker Embedding, Speech Enhancement)
• Startup model validation with dummy inference warmup

Voice Platform Dashboard

• Status tab — Engine readiness tiles, active model indicators, ONNX provider detection display, and guided next-steps checklist
• Models tab — Browse, download, activate, and delete models per engine type with real-time download progress
• Profiles tab — View enrolled and provisional speaker profiles, inline rename, promote to enrolled, merge profiles, manage audio clips with playback
• Wake Words tab — Register custom wake phrases with optional speaker enrollment, manage and delete entries
• Monitor tab — Real-time session monitoring via SSE with audio level meters, transcript log, and connection status
• Config panel — Voice verification threshold, provisional profile settings, adaptive profiles, and retention controls — persisted to MongoDB via ConfigStoreWriter

🎭 Personality Prompt (PR #80)

Users can now define a personality prompt on the /configuration page under the new Personality Prompt section. When set, the ResultAggregatorExecutor passes the composed multi-agent response through an LLM with the personality instructions as the system prompt, rewriting the output to match the desired tone.

• PersonalityPromptOptions — New config model with Instructions (the personality system prompt) and ModelConnectionName (optional separate LLM for rewriting)
• WorkflowFactory — Accepts IOptionsMonitor<PersonalityPromptOptions> for live config reload without restart. Conditionally resolves a separate IChatClient when ModelConnectionName is set, or falls back to the orchestrator's default model.
• ResultAggregatorExecutor — After composing the raw message from agent responses, calls the personality LLM with system=instructions + user=composed message. Graceful fallback to raw message on LLM failure or empty response.
• Resilient resolution — A misconfigured ModelConnectionName logs a warning and disables personality for that request instead of failing the entire orchestration pipeline.
• Cancellation-safe — OperationCanceledException propagates correctly through the personality rewrite path.

🖥️ Dashboard

• Model provider dropdown — The ModelConnectionName field renders as a searchable dropdown populated from configured chat-type model providers, matching the pattern used in Agent Definitions.
• Textarea field type — New textarea field type in the configuration page for multi-line prompt editing with vertical resize support.
• Auto-discovery — The Personality Prompt section appears automatically in the configuration sidebar via schema API.

🧺 Project Laundry

• SDK Pinned — Pinned .NET 10 to the latest appropriate minimal SDK
• Removed outdated docs — Removed a lot of outdated docs and old AI assistance templates and prompts.
• Package Updates — Updated all central projects to the latest versions from .NET 10.0.4 security patch

🐛 Bug Fixes

• CUDA GPU acceleration not activating — Microsoft.ML.OnnxRuntime NuGet package was CPU-only. Replaced with Microsoft.ML.OnnxRuntime.Gpu.Linux 1.23.2 and aligned managed/native versions (was mismatched 1.23.2/1.22.0), enabling automatic CUDAExecutionProvider detection on hosts with NVIDIA GPUs. Docker voice image verified working with RTX 4090 + CUDA 12.8 + cuDNN 9.20.
• Wyoming describe response missing STT — WyomingServiceInfo injected a single ISttEngine? via DI, which resolved to the last registered engine (SherpaSttEngine). If that engine wasn't ready, STT was omitted from the info response even when HybridSttEngine was ready. Now injects IEnumerable<ISttEngine> and reports STT available if any engine is ready.
• Parakeet model download "not found" — ModelCatalogService.GetModelById(string) only searched EngineType.Stt, but Parakeet TDT is registered as EngineType.OfflineStt. Now searches all engine types. Also fixed the download endpoint to resolve the model base path per engine type instead of hardcoding the STT path.
• Activating Parakeet crashes server — ModelManager.SwitchActiveModelAsync(string) hardcoded EngineType.Stt, causing the streaming OnlineRecognizer to load an offline transducer model (native crash: 'window_size' does not exist in the metadata). Now resolves engine type from the catalog, routing offline models to HybridSttEngine correctly.
• Cannot switch back to streaming STT after activating offline model — Both engines remained ready with no user preference tracking, so FirstOrDefault(e => e.IsReady) always picked HybridSttEngine (registered first). Added ModelManager.PreferredSttEngineType that tracks the user's last activation choice. Status endpoint, session engine selection, and active model API all respect the preference.
• HuggingFace API key not persisting in dashboard — ConfigurationPage.entriesToValues stripped only the first colon segment from stored keys (e.g., Wyoming:HuggingFace:ApiToken → HuggingFace:ApiToken). For nested config sections, this didn't match the schema property name ApiToken. Fixed to strip the full section prefix.
• Speaker identification always returning unknown — Speech enhancement was altering audio used for embedding extraction, causing ~0.39 cosine similarity against enrollment profiles. Now uses raw (unenhanced) audio for speaker verification.
• Verification threshold changes from GUI ignored — SpeakerVerificationThreshold config was never passed to IdentifySpeaker(). Now read via IOptionsMonitor.CurrentValue at identification time.
• Embedding dimension mismatches silently failed — After model changes, CosineSimilarity threw for mismatched dimensions, caught by outer try/catch returning null. Now gracefully skips with a warning log.
• Voice config written to local JSON file — VoiceConfigApi was writing to voiceconfig.json instead of the platform's MongoDB ConfigStoreWriter. Migrated to match established pattern.
• Thread pool deadlock in STT finalization — GetFinalResult blocked the thread pool. Converted to async with proper continuation.
• Aspire resilience timeout killing model downloads — 3-minute default timeout was insufficient for large model downloads. Bypassed Aspire service discovery for external URLs.
• Progressive re-transcription mixing raw and enhanced audio — STT now always receives raw audio; enhanced audio used only for clip storage.
• WAV protocol wire format corrections — Proper transcribe/transcript event ordering per Wyoming protocol spec.

⚡ Performance

• Hybrid STT achieves ~90ms finalization latency at 0% WER on benchmark audio
• Redis-cached speaker profiles reduce diarization lookup from ~500ms to ~1ms
• Short command fast-path skips progressive re-transcription overhead
• ONNX model warmup on startup eliminates cold-start inference latency
• CUDA 12.8 + cuDNN 9 GPU Dockerfile for accelerated voice inference
• Workstation GC mode and thread pool tuning for development responsiveness
• OTEL export frequency reduced from 1s to 5s to lower telemetry overhead

📊 Test Coverage

Area	Tests
Wyoming session integration	50+
Speaker verification components	17
Speech enhancement validation	20+
Engine hot-reload	10+
Engine readiness	10+
Wyoming status API	2
Command pattern matching	30+
Profile store (InMemory)	10+
Total Wyoming Tests	230

🔄 Upgrade Notes

• New infrastructure dependencies — MongoDB (luciaconfig database) is used for speaker profiles and voice config persistence. Redis is optional but recommended for profile caching.
• Model downloads required — On first launch, navigate to the Voice Platform → Models tab to download and activate at least one STT model and supporting models (VAD, Wake Word, Speaker Embedding).
• Wyoming integration — Add the Lucia Wyoming satellite in Home Assistant under Settings → Devices & Services → Add Integration → Wyoming. The server advertises via Zeroconf automatically.
• GPU acceleration — The project now ships with Microsoft.ML.OnnxRuntime.Gpu.Linux for automatic CUDA support. For local development, install CUDA Toolkit 12.x and cuDNN 9.x. The OnnxProviderDetector will find and use CUDA automatically — no configuration required. The Docker voice image (Dockerfile.voice) includes all GPU dependencies out of the box.
• Existing voice config — If you previously had a voiceconfig.json, those settings will need to be re-entered through the dashboard Voice Platform config panel (they now persist to MongoDB).