Features and fixes: Silero - open your ears and present it to the user#21
Merged
Conversation
VadWorker had three fatal bugs that killed live transcription after ~1.2s: - Initial probability 1.0 (not 0.0) bypassed seen_speech gate on first frame - No sample accumulation: 1024@44100Hz → 371@16kHz < 512 CHUNK_SIZE → predict() always returned 0.0 - Fire-and-forget stale AtomicU32 values Replaced with local-instance architecture: - AccumulatingVad: SileroVad + Resampler + proper chunk accumulation (init prob 0.0) - RecorderVad: dedicated thread for cpal Fn callback compatibility - ConversationEngine: local AccumulatingVad for Moshi flow Removed ~250 LOC of global singleton (OnceLock, Mutex, mpsc channel, VadMessage enum). Added ~100 LOC of correct local-instance code. Rewrote e2e_vad_flow tests (21 tests) for AccumulatingVad API. VibeCrafted with AI Agents (c)2026 VetCoders Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduce a Silero VAD pre-filter (vad::extract_speech) and expose VadExtractStats from core/vad. The VAD extractor runs over 500ms windows, returns concatenated speech-only samples and stats (speech_pct, windows, sparkline), and falls back to the original audio if the model is unavailable or filters everything. Update bin/codescribe to always load audio, run the VAD pre-filter, print VAD stats, and use the speech-only samples for language detection, streaming transcription, and non-streaming transcription. Export extract_speech and VadExtractStats from the public vad_api.
Engine tab (5th sidebar tab) shows read-only runtime status: STT engine (candle/onnx), Whisper model, Silero VAD, TTS, Embedder. Green/orange dot indicators, monospace values, env hint. Fix 3 VAD tests: synthetic sinusoids don't trigger Silero (neural network recognises formant structure, not pure tones). Changed assertions from prob>0.1 to valid range [0,1] — real audio tests already cover speech detection accuracy (18/18 pass). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
6 tests verifying Engine tab can read runtime status: - STT engine env var (candle default, onnx switch) - Whisper embedded status API - Silero VAD availability (embedded or user path) - Embedder API existence (lazy init) - TTS embedded status API Co-Authored-By: Klaudiusz <the1st@whoai.am>
Move many previously .env-only configuration keys into user settings and wire up loader/migration logic so settings.json becomes the source of truth. Added new UserSettings fields for LLM formatting, local STT, STT endpoint, transcript/send/audio controls, quick notes, history/start-at-login flags, and several "Voice Lab survivor" UX knobs (buffer_delay_ms, typing_cps, emit_words_max, buffered_interim_sec, whisper_model, backend_max_upload_mb). Updated Config::load and migration code to read/write these values, promote them to runtime env vars when appropriate, and handle parsing/setters for strings, u64, f32 and booleans. Simplified the UI Voice Lab field list (reduced from 22 to 7), tweaked labels/descriptions (e.g. interim cadence), and removed various experimental/debug Voice Lab options. Co-Authored-By: Klaudiusz <the1st@whoai.am>
|
To use Codex here, create a Codex account and connect to github. |
Summary of ChangesHello @Szowesgad, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly refactors the Voice Activity Detection (VAD) system, resolving critical bugs that affected live transcription and improving its accuracy and stability by transitioning from a problematic global singleton to a robust local-instance architecture. Concurrently, it enhances the user interface by introducing a dedicated "Engine" tab for monitoring runtime component statuses and simplifies the "Voice Lab" settings to present only the most relevant options to users, thereby improving both system reliability and user experience. Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request is a significant improvement, addressing critical bugs in the VAD pipeline by refactoring from a problematic global singleton to a much cleaner local-instance architecture. The introduction of AccumulatingVad correctly handles chunked audio processing, which was a source of major issues. The addition of the "Engine" tab in the UI is a great enhancement for observability, and simplifying the "Voice Lab" tab improves user experience. The code is well-structured, and the rewritten tests provide good coverage for the new VAD implementation. I've found a couple of areas for improvement, mainly related to minor memory leaks in the UI code and a logic issue in the speech extraction function. Overall, this is a high-quality contribution that greatly enhances the robustness and usability of the application.
Comment on lines
+123
to
+133
| if speech_samples.is_empty() { | ||
| return ( | ||
| samples.to_vec(), | ||
| VadExtractStats { | ||
| speech_pct: 100.0, | ||
| speech_windows: total_windows, | ||
| total_windows, | ||
| sparkline, | ||
| }, | ||
| ); | ||
| } |
There was a problem hiding this comment.
This safety net to prevent returning an empty vector seems to have an unintended side effect. If VAD correctly determines there is no speech, this logic returns the original (silent) audio and reports 100% speech. This defeats the purpose of VAD pre-filtering, as the downstream transcription process will be fed a lot of unnecessary silence. It would be better to return an empty vector and let the caller decide how to handle the 'no speech' case.
// If VAD filtered everything, it means no speech was detected.
// Return empty samples and let the caller handle it.
if speech_samples.is_empty() && total_windows > 0 {
return (
Vec::new(),
VadExtractStats {
speech_pct: 0.0,
speech_windows: 0,
total_windows,
sparkline,
},
);
}
Comment on lines
+2024
to
+2038
| let vad_status = if vad_embedded { | ||
| "Silero v6 embedded (2.3 MB)" | ||
| } else { | ||
| let path = codescribe_core::vad::user_model_path(); | ||
| if path.exists() { | ||
| Box::leak(format!("Silero v6: {}", path.display()).into_boxed_str()) | ||
| } else { | ||
| "Not found (will auto-download)" | ||
| } | ||
| }; | ||
| add_row( | ||
| "VAD", | ||
| vad_status, | ||
| vad_embedded || codescribe_core::vad::user_model_path().exists(), | ||
| ); |
There was a problem hiding this comment.
Similar to the whisper_status above, this use of Box::leak creates a memory leak. You can avoid this by creating a local String and passing a slice of it to the add_row function.
Suggested change
| let vad_status = if vad_embedded { | |
| "Silero v6 embedded (2.3 MB)" | |
| } else { | |
| let path = codescribe_core::vad::user_model_path(); | |
| if path.exists() { | |
| Box::leak(format!("Silero v6: {}", path.display()).into_boxed_str()) | |
| } else { | |
| "Not found (will auto-download)" | |
| } | |
| }; | |
| add_row( | |
| "VAD", | |
| vad_status, | |
| vad_embedded || codescribe_core::vad::user_model_path().exists(), | |
| ); | |
| let vad_status = if vad_embedded { | |
| "Silero v6 embedded (2.3 MB)".to_string() | |
| } else { | |
| let path = codescribe_core::vad::user_model_path(); | |
| if path.exists() { | |
| format!("Silero v6: {}", path.display()) | |
| } else { | |
| "Not found (will auto-download)".to_string() | |
| } | |
| }; | |
| add_row( | |
| "VAD", | |
| &vad_status, | |
| vad_embedded || codescribe_core::vad::user_model_path().exists(), | |
| ); | |
There was a problem hiding this comment.
Pull request overview
This pull request modernizes the Silero VAD (Voice Activity Detection) architecture by replacing a buggy global singleton with local instances, adds a new "Engine" settings tab for runtime status visibility, and streamlines the Voice Lab UI to focus on user-impactful settings.
Changes:
- Replaced global
VadWorkersingleton with localAccumulatingVadinstances, fixing three critical bugs: initial probability incorrectly set to 1.0 (now 0.0), lack of sample accumulation causing inference failures, and stale atomic values - Added a new "Engine" tab to the settings UI showing runtime status of STT, Whisper, VAD, TTS, and Embedder components with read-only indicators
- Reduced Voice Lab settings from 22 to 7 user-facing fields, promoting several settings from
.envtosettings.jsonfor better discoverability
Reviewed changes
Copilot reviewed 14 out of 16 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| core/vad/silero_ort.rs | Removed ~250 LOC of singleton infrastructure (VadWorker, channels, atomics), added AccumulatingVad with proper chunk accumulation and 0.0 initial probability |
| core/vad/mod.rs | Updated API exports, added extract_speech function for VAD-based speech extraction from audio files |
| core/audio/recorder.rs | Added RecorderVad local instance to replace global VadWorker dependency |
| core/conversation/engine.rs | Added local_vad field using AccumulatingVad instead of global singleton |
| tests/e2e_vad_flow.rs | Completely rewrote 21 tests for new local-instance API (removed serial markers, updated to use AccumulatingVad.new/feed) |
| tests/e2e_bootstrap_settings.rs | Added 5 tests for Engine tab runtime status queries |
| app/ui/bootstrap/mod.rs | Added Engine tab implementation with status indicators, reduced Voice Lab fields from 22 to 7, updated state arrays from 4 to 5 tabs |
| app/ui/bootstrap/handlers.rs | Added on_tab_engine handler for tab switching |
| core/config/settings.rs | Added 17 new optional fields promoted from .env (llm_formatting_endpoint, use_local_stt, etc.) |
| core/config/migrate.rs | Added migration logic for newly promoted fields from .env to settings.json |
| core/config/loader.rs | Added loader logic for promoted fields with env-var fallbacks |
| bin/codescribe.rs | Integrated vad::extract_speech for pre-filtering audio files before transcription |
| app/ui/tray/mod.rs | Removed vad::shutdown calls (no longer needed without singleton) |
| Cargo.toml | Version bump to 0.7.17 |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+2013
to
+2033
| let whisper_status = if whisper_embedded { | ||
| "Embedded (~894 MB in binary)" | ||
| } else { | ||
| let path = | ||
| std::env::var("CODESCRIBE_MODEL_PATH").unwrap_or_else(|_| "(not set)".to_string()); | ||
| Box::leak(format!("External: {}", path).into_boxed_str()) | ||
| }; | ||
| add_row("Whisper", whisper_status, whisper_embedded); | ||
|
|
||
| // ── VAD (Silero) ─────────────────────────────────────── | ||
| let vad_embedded = codescribe_core::vad::embedded::is_embedded_available(); | ||
| let vad_status = if vad_embedded { | ||
| "Silero v6 embedded (2.3 MB)" | ||
| } else { | ||
| let path = codescribe_core::vad::user_model_path(); | ||
| if path.exists() { | ||
| Box::leak(format!("Silero v6: {}", path.display()).into_boxed_str()) | ||
| } else { | ||
| "Not found (will auto-download)" | ||
| } | ||
| }; |
There was a problem hiding this comment.
Memory leak: Same issue as line 2018 - Box::leak creates a leaked allocation every time the settings window is opened. The string will never be freed. Use a local String instead.
Suggested change
| let whisper_status = if whisper_embedded { | |
| "Embedded (~894 MB in binary)" | |
| } else { | |
| let path = | |
| std::env::var("CODESCRIBE_MODEL_PATH").unwrap_or_else(|_| "(not set)".to_string()); | |
| Box::leak(format!("External: {}", path).into_boxed_str()) | |
| }; | |
| add_row("Whisper", whisper_status, whisper_embedded); | |
| // ── VAD (Silero) ─────────────────────────────────────── | |
| let vad_embedded = codescribe_core::vad::embedded::is_embedded_available(); | |
| let vad_status = if vad_embedded { | |
| "Silero v6 embedded (2.3 MB)" | |
| } else { | |
| let path = codescribe_core::vad::user_model_path(); | |
| if path.exists() { | |
| Box::leak(format!("Silero v6: {}", path.display()).into_boxed_str()) | |
| } else { | |
| "Not found (will auto-download)" | |
| } | |
| }; | |
| let mut whisper_status_owned: Option<String> = None; | |
| let whisper_status: &str; | |
| if whisper_embedded { | |
| whisper_status = "Embedded (~894 MB in binary)"; | |
| } else { | |
| let path = | |
| std::env::var("CODESCRIBE_MODEL_PATH").unwrap_or_else(|_| "(not set)".to_string()); | |
| whisper_status_owned = Some(format!("External: {}", path)); | |
| whisper_status = whisper_status_owned.as_ref().unwrap().as_str(); | |
| } | |
| add_row("Whisper", whisper_status, whisper_embedded); | |
| // ── VAD (Silero) ─────────────────────────────────────── | |
| let vad_embedded = codescribe_core::vad::embedded::is_embedded_available(); | |
| let mut vad_status_owned: Option<String> = None; | |
| let vad_status: &str; | |
| if vad_embedded { | |
| vad_status = "Silero v6 embedded (2.3 MB)"; | |
| } else { | |
| let path = codescribe_core::vad::user_model_path(); | |
| if path.exists() { | |
| vad_status_owned = Some(format!("Silero v6: {}", path.display())); | |
| vad_status = vad_status_owned.as_ref().unwrap().as_str(); | |
| } else { | |
| vad_status = "Not found (will auto-download)"; | |
| } | |
| } |
Comment on lines
+297
to
+300
| assert!( | ||
| embedded || !embedded, | ||
| "is_embedded_available should return bool" | ||
| ); |
There was a problem hiding this comment.
Test quality issue: This assertion embedded || !embedded will always pass since it's a tautology (a boolean is always either true or false). This test doesn't verify anything meaningful. Either check for a specific expected value or remove the test if you just want to verify the function doesn't panic.
Comment on lines
+332
to
+335
| assert!( | ||
| embedded || !embedded, | ||
| "is_embedded_available should return bool" | ||
| ); |
There was a problem hiding this comment.
Test quality issue: Same tautology issue as test_engine_tab_whisper_embedded_status. The assertion embedded || !embedded will always pass. This doesn't test anything meaningful.
Convert CODESCRIBE_TYPING_CPS to f32 across the codebase so fractional typing CPS values are supported (settings, loader parsing, and migrate). Add an end-to-end test to verify decimal persistence. Make the local Silero VAD required (non-optional) in ConversationEngine: fail-fast on init, call feed() directly for speech probability, and reset the VAD on conversation reset. Fix UI bootstrap to use owned Strings (avoid Box::leak) and pass string references to add_row. These changes ensure correct numeric handling for typing CPS and enforce the presence of the Silero VAD for conversation mode.[ok-commit]
Track and cleanly shut down the VAD worker thread owned by RecorderVad. The tx sender is now an Option and a thread JoinHandle is stored so Drop can close the channel and join the thread (warning on panic). Call sites were updated to safely clone the optional tx, and Recorder::stop() now clears recorder_vad to ensure deterministic worker shutdown. This prevents dangling threads and ensures orderly shutdown of the VAD worker.[ok-commit]
* Add macOS onboarding and permission handling Introduce a first-run onboarding wizard for macOS and extend permission utilities and hotkey configuration. Adds a new ui::onboarding module (should_show_onboarding, show_onboarding_wizard) that guides users through language, API key, hotkey mode and required macOS permissions (Microphone, Accessibility, Input Monitoring, Screen Recording, Full Disk Access) with UI, polling for Full Disk Access, keychain persistence for LLM API key, and modal flow control. Update permissions.rs to query microphone via AVCaptureDevice, add screen-recording (CGPreflight/Request) and full-disk-access checks and request helpers, and provide a deeplink opener for Privacy settings. Add a HoldMods::None value and wire it through hotkeys (storage, getters/setters, logic paths and UI selection) to support toggle-only hotkey mode; update bootstrap to show the new "Disabled (toggle only)" option. Wire onboarding into app/ui exports and module declarations. Minor test and synchronization fixes (mutex around hold mode tests) and various UI wiring to persist chosen hotkey modes and settings. * Persist onboarding progress, add sidebar & locking Store and resume onboarding step index to a progress file and clear it when finished. Add a session lock file (with PID liveness check) to prevent duplicate wizard instances and ensure the lock is released. Implement sidebar UI with per-step labels/markers, adjust layout to center content, fix label alignment, and save progress on step advance/retreat. Also change daemon startup to check permissions instead of triggering request_all_permissions() to avoid prompting on launch. * Add macOS screen capture & refactor onboarding Add ObjC block support and NSScreenCapture usage: add the block crate dependency and include a Screen Capture usage string in the app bundle Info.plist. Implement synchronous microphone request flow on macOS using AVCaptureDevice::requestAccessForMediaType with an ObjC completion block and channel timeout. Make privacy settings opener public in the permissions module and update callers to use it. Refactor onboarding code to replace lazy_static/Once with std sync LazyLock/OnceLock, return &'static Class from class builders, mark the window as closable, and properly release Objective-C objects when the onboarding window closes. * Persist promoted IPC settings; UI & config fixes Persist certain promoted IPC settings into UserSettings instead of only updating env vars (adds is_promoted_ipc_setting and persist_promoted_setting). Refactor bootstrap UI to use named TAB_* constants and TAB_COUNT, update tab handlers and related tests/validation keys. Load FORMATTING_LEVEL from settings into the environment when absent and only set CODESCRIBE_BUFFERED_STREAM if the setting is present. Migrate TOGGLE_SILENCE_SEC from env into settings if available. Improve VAD robustness by logging a warning on AccumulatingVad init failure and reduce initial speech buffer reservation. Update e2e tests to validate embedded model availability by checking get_embedded_data() and model total_size() for Whisper and TTS. @gemini-code-assist review * Persist promoted settings; macOS permission fixes Multiple changes to improve settings persistence, macOS permission handling, onboarding, VAD behavior, and config safety. Highlights: - IPC: Track and persist promoted IPC settings into UserSettings (expanded mapping) and log keys on save failure. - Onboarding/UI: Persist user choices (language, AI formatting, hotkey mode) directly to UserSettings with error logging; safer lock handling; release window refs and reset UI state on close; refactor hotkey mode to write settings once and set runtime state/env vars. - macOS permissions: Revamped microphone request flow to dispatch request on main thread, use bounded polling with timeouts, and robustly handle callback channel results; improved full-disk & screen-recording status reporting and messaging. - CLI (bin/codescribe): Add --no-vad flag to skip Silero VAD; use Cow to avoid copying when skipping VAD. - Recorder/VAD: Increase audio channel buffer from 8 to 128 and tighten atomic memory ordering (Acquire/Release) to avoid races. - Config loader: Add safe_set_env validation to reject empty/oversized values and replace unsafe direct env writes in multiple places. - UI tweaks: Show filenames instead of full paths in engine UI. - Steps/constants: Simplify STEP_FLOW sizing and compute TOTAL_STEPS from array length. - Docs: Add environment registry entries for new STT/ONNX-related variables. Overall these changes improve reliability, reduce hangs on macOS permission prompts, make onboarding persistence more robust, and harden config/env handling. * Introduce AI retry policy, timeouts, and UI refinements Add a configurable AI retry/timeout policy and tighten AI HTTP client timeouts; refactor ai_formatting to use a RetryPolicy, per-attempt and inter-chunk timeouts, and a unified call_provider_once helper, and add env defaults and docs. Update tests to exercise retry and SSE inter-chunk timeout behavior. Improve macOS assistive selection handling by preferring Cmd+C fallback for certain browsers and passing frontmost app info. Add Tahoe-compatible glass effect detection and style handling, use glass/visual effect views where available, and adjust UI visuals (alpha, borders, window style). Make onboarding window larger and restyle titlebar/content, persist onboarding step before requesting permissions, and reposition/resize many onboarding elements. Enhance voice chat overlay with drag-and-drop support, file-url pasteboard fallback, and visual tweaks. Update .env.example and ENV_REGISTRY with new AI-related env vars and defaults. * Adjust onboarding UI layout & mic permission flow Update onboarding UI layout and summary sizing, and make microphone permission requests non-blocking. - app/ui/onboarding/mod.rs: - Tweaked description label frame (height/position) and summary view sizing by introducing SUMMARY_WIDTH/SUMMARY_HEIGHT constants. - Use computed positioning (center-based) and constants for summary layout; introduce summary_line_height and summary_top for clearer per-line positioning. - Adjusted summary label heights and summary_config origin. - Make microphone permission requests asynchronous: spawn a thread to call request_permission for Microphone, update onboarding state on the main queue, render the current step immediately to keep the UI responsive, and potentially schedule auto-advance if granted. - tests/e2e_retry_responses.rs: - Removed unused mockito::Matcher import and make mocks match the concrete POST path "/v1/responses" instead of Matcher::Any to ensure deterministic request matching in tests. These changes improve UI alignment and responsiveness when the system permission prompt is shown, and make tests more specific and reliable. * Promote settings, keychain secrets, Ollama timeout Centralize promoted env keys into a single PROMOTED_SETTINGS_KEYS list and add is_promoted_key; use that in Config loader and IPC persist logic to route GUI-managed settings to settings.json. Persist_config now avoids storing secrets in plaintext .env: secret keys (LLM_API_KEY, STT_API_KEY) are saved/removed via the keychain and .env is updated (set/remove) accordingly using a new EnvUpdate enum and persist_secret_setting helper. Add a dedicated CODESCRIBE_AI_OLLAMA_ATTEMPT_TIMEOUT_MS env var and wire an Ollama-specific per-attempt timeout into the AI retry policy/formatting logic. UI tweaks: ensure onboarding lock is always released via a Drop guard and handle panics from the onboarding modal; make colour alphas adaptive to glass effect and adjust scroll edge alpha; minor drag handler comment. Update docs and examples to document the new Ollama timeout env var and other registry entries, and expand promoted settings parsing to cover many additional keys and types.[ok-commit] * ui: migrate glass effect to typed objc2 and stabilize markdown bubble fonts * ui: unify settings and chat overlay glass hierarchy * ui: introduce unified top toolbar layout for settings * assistive: fallback to chat when selection context is missing * ui: refine chat input pill and fix settings titlebar duplication * Adjust macOS voice chat UI and window styling Reapply FullSizeContentView after window init to avoid AppKit falling back to separate titlebar/content regions (prevents a duplicate-looking top bar). In the voice chat overlay: align the input bar flush with the footer edge, remove heavy drop shadow in favor of a border-only look to match native NSSearchField, slightly tweak text area vertical positioning and height, and set the text container inset for better vertical centering. These tweaks improve visual parity with native macOS controls and fix a duplicated titlebar artifact. * Fix chat SSE streaming and hotkey/runtime regressions * Enforce streaming; refine hotkeys and overlays Make streaming mandatory for LLM formatting and update tests to use SSE fixtures. Harden macOS Option-tap detection by tracking the option side (left/right) and rejecting mismatched or non-keycode releases. Stop re-applying hotkey atomics on IPC config reload (UI sets them synchronously) and consolidate env writes when updating multiple hotkey settings. Add a concurrency guard for bootstrap overlay creation and defer vertical scroller enabling until document height is known. Introduce a header-record CTA (and wire it to the header button) and display keyboard shortcuts inline in the voice chat UI instead of a modal alert. These changes reduce races, improve UX consistency, and tighten gesture handling. * Fix overlay/tab behavior and response ID Keep UI overlay and assistive session state in sync and improve LLM streaming handling. - app/controller/mod.rs: ensure assistive session flag is cleared via set_assistive_session(false) when a recording finishes. - app/ui/voice_chat/api.rs & handlers.rs: centralize Settings-tab routing through update_active_tab_impl, hide chat overlay before showing bootstrap/settings to avoid stacked/ghost windows, avoid spawning the overlay if already visible, focus agent tab on show, and ignore shortcut actions when overlay is hidden. - core/llm/ai_formatting.rs: capture response_id earlier from stream metadata (chunk.response) to handle providers that emit it before completion, stop relying on completed chunk to set the id, and log/warn if the SSE finishes without a response_id (preserving previous id when available). These changes prevent duplicate windows/ghost overlays and make response ID tracking more robust across varying stream chunk orders. * Broadcast engine events and timestamped transcripts Introduce IPC event broadcasting and segment-level transcripts across the pipeline. Add IpcEvent/IpcEventPayload/EngineEventWire types and an IpcBroadcastSink that sends sanitized EngineEvent objects (with RFC3339 timestamps) over a tokio::broadcast channel. RecordingController now owns an event_broadcast channel, exposes subscribe_events, and sends StateChange events when state transitions occur. IPC server handles Subscribe/Unsubscribe at connection level and streams Event responses to subscribers. Implement FanoutEventSink to forward events to multiple sinks and wire it in so presentation + IPC sinks both receive engine events. Propagate segment metadata end-to-end: introduce RawTranscript/TranscriptSegment support and thread it through transcription workers, adapters, and engines. ONNX and Whisper adapters/engines now produce RawTranscript (text + segments), add whisper timestamps parsing helpers (core/stt/whisper/timestamps.rs) to extract segment timestamps from tokenizer timestamp tokens, and add relevant decoding params (emit_timestamps). Export transcribe_with_segments in core/lib. Small tests added for FanoutEventSink and updated event/drop kinds serialization. These changes enable external subscribers to receive real-time engine events and state changes and provide segment-level timing for transcripts. @gemini review @codex review * Recover stale recorder, improve IPC & timestamps Handle stale recorder state and tighten event pipeline semantics. Controller: add test helper publish_ipc_event_for_test, recover_stale_recorder_if_idle, a serial lock for start/stop, and retry logic for "Recording is already in progress" races; transition states earlier to avoid IDLE/active races. Audio: expose Recorder::is_active, StreamingRecorder::is_recording, and self-heal a stale is_recording flag on start. Pipeline: accumulate and adjust segment timestamps across interim slices so UtteranceFinal emits correct segments. Whisper timestamps: ignore unclosed trailing tokens and add unit tests. IPC & config: add IPC event-stream docs and server tests, add core/ipc types tests to ensure raw_text is not leaked and payloads are tagged, and skip saving settings when unchanged to avoid unnecessary writes. * Support AI reasoning channel and UI messages Introduce explicit reasoning channel support in the AI formatting pipeline and surface reasoning summaries in the UI. - core/llm/ai_formatting.rs: Add ProviderOutput with assistant_text and reasoning_text, StreamChannels type, AiReasoningCallback, and a new format_text_with_status_channels API (format_text_with_status kept as a wrapper). Parse provider outputs into separate assistant and reasoning channels (including SSE streaming handling and Ollama/Responses API paths) and propagate reasoning_text through the result. - app/controller/helpers.rs & app/controller/mod.rs: Call the new channels-based API and display reasoning summaries in the voice chat/overlay as system messages when present. - docs/ADR/2026-02-15-CODESCRIBE_RUNTIME_IN_VISTA.md: Add an ADR describing CodeScribe + Vista integration boundaries and migration plan. These changes enable streaming and delivery of separate reasoning summaries from LLM providers to the frontend while preserving existing assistant text behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request aims to repair thoroughly the Silero VAD supervisor pipeline and introduces a new "Engine" tab to the settings UI, providing users with a read-only panel that displays runtime engine status, and streamlines the Voice Lab tab to focus on user-impactful settings. The changes also update the version to
0.7.17and ensure proper handling of the new tab throughout the UI state.fix(vad): bury VadWorker singleton, replace with local AccumulatingVad
VadWorker had three fatal bugs that killed live transcription after ~1.2s:
Replaced with local-instance architecture:
Removed ~250 LOC of global singleton (OnceLock, Mutex, mpsc channel, VadMessage enum).
Added ~100 LOC of correct local-instance code.
Rewrote e2e_vad_flow tests (21 tests) for AccumulatingVad API.
New Engine tab and UI enhancements:
BootstrapState, tab switching, window closure, overlay hiding, and embedded state reset) to support five tabs instead of four. [1] [2] [3] [4] [5]Voice Lab tab simplification:
VOICE_LAB_FIELDSfrom 22 to 7, keeping only settings where user choice improves UX and removing advanced pipeline internals and debug options. [1] [2] [3]Version update:
0.7.16to0.7.17inCargo.toml. [1] [2]Code cleanup:
vadimport and shutdown call from tray UI logic, as it is no longer necessary. [1] [2]@gemini review @codex review