Refactor and enhance VAD integration for robust speech handling#14
Merged
Conversation
[ok-commit] f593a8c Author: Szowesgad <void@div0.space> Date: Tue Jan 27 17:11:17 2026 +0100 Improve E2E env handling and validation scripts Updated .env.example to clarify its role as the E2E ground truth. Added .env.e2e to .gitignore. Enhanced Makefile to auto-generate and use a clean E2E env for E2E and roundtrip tests. Refactored scripts/validate-envs.sh to support new CLI options for env example validation and E2E env emission, improving automation and reliability of environment setup for tests.[ok-commit] commit 1120944 Author: Szowesgad <void@div0.space> Date: Tue Jan 27 17:09:35 2026 +0100 Add initial prompt support for Whisper STT engine Introduces the WHISPER_INITIAL_PROMPT environment variable to guide Whisper transcription with domain vocabulary and formatting hints. Updates decoding parameters, engine logic, and documentation to support and describe this feature. Also adds a new hotkeys contract documentation and minor improvements to .gitignore and UI status messages. commit a77f58c Author: Szowesgad <void@div0.space> Date: Tue Jan 27 16:43:48 2026 +0100 Update VAD defaults and environment registry Changed VAD threshold and silence defaults to be less conservative across documentation, scripts, and examples. Refactored environment variable registry: removed deprecated entries, added new variables for models, embedding, LLM, TTS, quality, and system configuration. Updated code to use CODESCRIBE_VAD_ENABLED for auto_silence logic and removed CODESCRIBE_LOG_STDOUT support. Improved env var validation script to handle more patterns and ignore build system vars.[ok-commit] commit 90b4174 Author: Szowesgad <void@div0.space> Date: Tue Jan 27 15:41:48 2026 +0100 fix(moshi): Cache unavailability to prevent log spam - Add moshi_unavailable flag to RecordingController - Early validate MoshiConfig before trying to create engine - Cache failure state so subsequent hotkey presses don't spam logs - Silent return (Ok) when Moshi unavailable - not an error, just missing models Before: "Moshi model not found" on EVERY Ctrl+Option press After: Single warning, then silent skip Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> commit 23a780a Author: Szowesgad <void@div0.space> Date: Tue Jan 27 14:19:42 2026 +0100 chore: Clean up legacy VAD env var references - Replace CODESCRIBE_VAD_MAX_SILENCE_SEC → CODESCRIBE_VAD_SILENCE_SEC - Replace AUTO_SILENCE → CODESCRIBE_VAD_ENABLED - Fix duplicate CODESCRIBE_VAD_ENABLED in .env.debug.example - Change eprintln spam to tracing::debug in embedded.rs Files updated: - examples/README.md - scripts/download-silero.sh - docs/OVERLAY_STREAMING.md - docs/ADR/2026-01-26-env.md - docs/ADR/2026-01-26-OVERLAY_STREAMING.md - .env.debug.example - core/stt/whisper/embedded.rs Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> commit f248e50 Author: Szowesgad <void@div0.space> Date: Tue Jan 27 14:10:29 2026 +0100 docs(env): Add complete LLM_USE_STREAMING and TRANSCRIPT_SEND_MODE docs Added detailed documentation with defaults and options: - LLM_USE_STREAMING: default 1, options 1/true (SSE) vs 0/false (buffered) - TRANSCRIPT_SEND_MODE: streaming vs buffered modes Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> commit 95c713d Author: Szowesgad <void@div0.space> Date: Tue Jan 27 13:55:34 2026 +0100 fix(vad): Clean VAD config, fix embedding detection, HOLD mode user control BREAKING CHANGES: - Renamed env vars (old ones deprecated): - AUTO_SILENCE → CODESCRIBE_VAD_ENABLED - CODESCRIBE_VAD_MAX_SILENCE_SEC → CODESCRIBE_VAD_SILENCE_SEC - CODESCRIBE_VAD_SENSITIVITY → removed (use THRESHOLD directly) Core fixes: - Fix cfg!(embed_*) not detecting embedded models in workspace builds Now checks weights.len() > 0 instead of compile-time cfg!() macro - VAD defaults less aggressive: threshold 0.35 (was 0.5), silence 2.5s (was 1.2s) - Whisper params stricter: compression 2.0, no_speech 0.5, logprob -0.5 VAD behavior change: - HOLD mode: NO VAD auto-stop (user controls by releasing key) - TOGGLE mode: VAD active (auto-stop after silence) - If user wants to hold in silence for 45 minutes - that's their choice New files: - docs/ENV_REGISTRY.toml: Central registry of all env vars - scripts/validate-envs.sh: Pre-commit validator for env vars Updated docs: - docs/env.md: New VAD var names and defaults - .env.example, .env.debug.example: Clean config templates Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> commit e65aedb Author: Szowesgad <void@div0.space> Date: Tue Jan 27 12:12:45 2026 +0100 Add comprehensive ADR and user documentation Introduced multiple ADR markdown files covering licensing, installation, team setup, architecture, backlog, feasibility analysis, and Whisper Live streaming. Also added a user guide and related documentation to the docs/ADR directory. Removed obsolete local backend section from .env.debug.example and renamed runner.sh script to tools/runner.sh. commit 76c7574 Author: Szowesgad <void@div0.space> Date: Tue Jan 27 09:37:45 2026 +0100 Improve overlay window input and live transcription handling Introduces a custom NSWindow subclass for overlays to allow borderless windows to receive input. Updates live transcription to not auto-stop on silence and adds a stop_without_saving method to audio recorders, preventing unnecessary WAV file creation. Also improves thread safety in overlay state handling and adds an environment variable to control logging to stdout.[ok-commit] commit b53638c Author: Szowesgad <void@div0.space> Date: Tue Jan 27 07:37:26 2026 +0100 Improve logging and overlay handling, add tracing Adds the tracing-appender dependency and initializes structured logging to both file and stdout in the CLI. Replaces eprintln! with tracing macros for better log management. Fixes overlay blur view initialization in bootstrap and voice chat overlays. Updates assistive loop logic and delta callback routing in RecordingController for more robust session handling.[ok-commit] commit 58037e6 Author: Szowesgad <void@div0.space> Date: Tue Jan 27 05:33:42 2026 +0100 Improve overlay logging and update model embedding logic Added more informative logging to overlay and voice chat show/hide functions for better traceability. Updated hotkey event handling in codescribe daemon to apply settings immediately, reducing race conditions. Revised build.rs to embed Whisper model by default in release builds, with opt-out via CODESCRIBE_NO_EMBED, and clarified logic and warnings for model embedding. commit 66bdad1 Author: Szowesgad <void@div0.space> Date: Tue Jan 27 04:10:28 2026 +0100 Add round-trip pipeline demo and tests Introduces interactive and test-based round-trip (TTS→STT→Embeddings) validation for pipeline integrity. Adds Makefile targets, scripts for Silero VAD model download, HuggingFace cache support for models, and updates documentation for quick start and settings. Improves hotkey handling for conversation mode and refines TTS/STT/embedding integration. commit 68769f2 Author: Szowesgad <void@div0.space> Date: Tue Jan 27 04:12:33 2026 +0100 Add onboarding bootstrap overlay for macOS Introduces a bootstrap onboarding overlay for first-time users on macOS, including UI, handlers, and integration with the tray menu. The overlay guides users through initial setup steps and is triggered on first launch or via the tray menu. Packaging scripts and Makefile updated for improved model bundling and release process, with enhanced environment variable support for signing and notarization.[ok-commit] commit e44ec44 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 23:11:45 2026 +0100 Implement assistive voice chat streaming and UI updates Adds support for streaming user transcription into chat bubbles during assistive sessions, including new helpers for appending and finalizing user message text. Refactors overlay routing, session state, and chat bubble styling for improved clarity and Quantum theme consistency. Introduces an assistive VAD loop for hands-off recording and updates UI color palette and font handling for chat bubbles.[ok-commit] commit 17c7a19 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 19:40:40 2026 +0100 Add VAD sensitivity and ONNX support, improve cache handling Introduces CODESCRIBE_VAD_SENSITIVITY and CODESCRIBE_VAD_SILENCE_SEC as preferred VAD controls, mapping sensitivity to threshold and allowing simple silence override. Adds support for ONNX embedder models, improves HuggingFace cache path resolution, and refines environment variable handling for model embedding and cache directories.[ok-commit] commit 9c053a9 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 18:56:46 2026 +0100 Refactor model embedding and HuggingFace cache detection Model embedding is now opt-in for release builds, requiring explicit environment variables to embed models, E5, or TTS. The HuggingFace cache search now checks all possible cache locations instead of just one, improving robustness in both build.rs and hf_cache.rs. Error handling and warnings for missing models have been streamlined to avoid hard failures and provide clearer instructions.[ok-commit] commit effdffe Author: Szowesgad <void@div0.space> Date: Mon Jan 26 17:39:07 2026 +0100 Support both weights.safetensors and model.safetensors Updated model loading logic to accept either weights.safetensors or model.safetensors files for Whisper models. Refactored cache lookup and model path resolution to handle both file names, improving compatibility with different model distributions.[ok-commit] commit def6c72 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 17:23:52 2026 +0100 Add case-insensitive fallback for HF cache lookup Update `find_hf_snapshot` and `find_snapshot` to handle case-insensitive repository matches when locating Hugging Face cache directories. Also, adjust `download-csm.sh` to use multiple --include flags for clarity and compatibility.[ok-commit] commit c9ba7bf Author: Szowesgad <void@div0.space> Date: Mon Jan 26 17:12:17 2026 +0100 Trim whitespace from CODESCRIBE_EMBED_MODEL env var Updated the build script to trim whitespace from the CODESCRIBE_EMBED_MODEL environment variable before checking if it is empty. This ensures that values with only whitespace are treated as empty.[ok-commit] commit 2658fc0 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 16:47:36 2026 +0100 Switch model loading to HuggingFace cache Refactored model loading logic to use HuggingFace cache via the `hf download` CLI, replacing hardcoded local paths and improving flexibility. Added `core/hf_cache.rs` for cache utilities, updated build scripts and runtime model resolution to support repo overrides, and revised download scripts to use the HuggingFace cache directly. Updated documentation and warnings to reference `hf download` commands.[ok-commit] commit 43e6d78 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 16:04:12 2026 +0100 Switch moshi dependency to crates.io version Updated core/Cargo.toml to use moshi 0.6.4 from crates.io instead of the GitHub repository. Updated Cargo.lock to reflect this change and adjusted related dependencies. [ok-commit] commit 0c7c10c Author: Szowesgad <void@div0.space> Date: Mon Jan 26 15:08:44 2026 +0100 Add word-level emission and backspace support to streaming Implements word-level emission in buffered streaming mode with a configurable max words per tick (CODESCRIBE_EMIT_WORDS_MAX). Adds support for backspace (\u{0008}) deltas in both overlay and chat UI, and ensures deltas are applied with proper redaction. Updates environment examples and documentation to reflect new settings.[ok-commit] commit d10a9a8 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 11:27:41 2026 +0100 Switch embedder from fastembed to offline E5 model Replaces the fastembed dependency with a local/embedded E5 model using Candle and tokenizers for offline text embeddings. Adds support for embedding the E5 model directly into the binary in release builds, with new build.rs logic and a generated embedded.rs module. Updates Makefile and scripts to support E5 model download, and adjusts documentation and config to reflect the new embedding approach. Removes fastembed and related dependencies from Cargo files and codebase.[ok-commit] commit 7d5b9f8 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 08:27:59 2026 +0100 Add streaming tag parser for demuxed LLM output Introduces a new demux module with a StreamingTagParser for parsing flat, non-nested tags such as <speak> and <tool> in streamed LLM output. Updates core/lib.rs to include the new module and switches the moshi dependency in Cargo.toml to use the Kyutai Labs GitHub repository. [ok-commit] # Conflicts: # core/Cargo.toml commit 2ee2b10 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 09:23:54 2026 +0100 Switch moshi to git dependency and update configs Replaced the local path dependency for moshi with a git-based dependency for better portability. Updated .gitignore patterns for logs and codescribe memory files. Improved comments and logic in config loader to set history and debug logging as default-on but still overridable by environment variables. Translated comments in helpers.rs from Polish to English. commit b9e7e98 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 08:06:50 2026 +0100 Set formatting-specific LLM env vars in e2e test Added LLM_FORMATTING_ENDPOINT, LLM_FORMATTING_MODEL, and LLM_FORMATTING_API_KEY environment variables in the e2e_retry_responses test to ensure both generic and formatting-specific configurations are covered. commit c546d5d Author: Szowesgad <void@div0.space> Date: Mon Jan 26 07:55:10 2026 +0100 Refactor formatting and improve code consistency This commit applies consistent formatting across multiple modules, including improved indentation, line breaks, and grouping of imports and re-exports. It also updates the 'ort' dependency feature in core/Cargo.toml and makes minor logic and style improvements in tests and core modules for better readability and maintainability. commit cecfbcd Author: Szowesgad <void@div0.space> Date: Mon Jan 26 06:32:56 2026 +0100 Fix clippy warnings for strict -D warnings compliance - Collapse nested if-let statements (clippy::collapsible_if) - Remove useless .into() on anyhow::Error (clippy::useless_conversion) - Use let-chains for cleaner conditional matching Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> commit 051c99e Author: Szowesgad <void@div0.space> Date: Mon Jan 26 05:50:28 2026 +0100 Add session generation counter to RecordingController Introduces a conversation_generation AtomicU64 to RecordingController to track session generations and prevent cross-session race conditions. The audio loop and playback UI updates now check the session generation to ensure only the current session can update the UI, improving reliability when multiple sessions are started in quick succession. commit 96c5a86 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 05:35:24 2026 +0100 Refine conversation mode UI state and audio handling Standardizes conversation state status messages, adds explicit UI state updates, and improves audio playback to be non-blocking for full-duplex operation. Also ensures proper cleanup and state reset on errors and when stopping conversation mode, and updates drawer entry mode detection for conversation logs. commit cc54f39 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 05:26:26 2026 +0100 Add Moshi full-duplex conversation mode Implements a new full-duplex conversation mode using the Moshi engine, allowing simultaneous microphone input and AI response playback. Adds state management, hotkey handling, and UI updates for conversation mode, including a resampler for arbitrary input rates. Updates core conversation context to track total speaking duration and exposes new state enums and helpers throughout the app. commit db848a9 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 04:54:51 2026 +0100 Integrate Moshi LM and Mimi codec into conversation engine This update loads Moshi LM and Mimi codec models in ConversationEngine, implements audio encode/decode, and runs full inference for user audio input. Configuration paths and parameters are updated for new model formats. Adds E2E tests for initialization, audio processing, and state reset to validate the full conversational pipeline. commit 74f77d8 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 02:36:56 2026 +0100 Unify model and data paths to ~/.codescribe directory Standardized all references from .CodeScribe or project-relative paths to ~/.codescribe for models and lexicon files across code, scripts, and documentation. This change ensures consistent model/data location for easier setup and usage. commit 55e89b2 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 01:58:41 2026 +0100 Standardize model paths to ~/.codescribe/models Updated scripts and Rust code to store all model files under ~/.codescribe/models for consistency. Added download-silero.sh to automate Silero VAD model retrieval. Adjusted default_model_path in silero_ort.rs and updated related scripts to use the new location. commit ac69553 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 01:44:03 2026 +0100 Remove deprecated SILENCE_* config and update VAD tests Eliminates legacy SILENCE_DB and SILENCE_HANG_SEC configuration from code, environment, and documentation. Updates VAD-related tests to use the new speech probability threshold and hang_sec parameters, and adds a comprehensive end-to-end VAD flow test suite. # Conflicts: # tests/e2e_vad_auto_stop.rs # tests/e2e_vad_flow.rs commit 991ace5 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 01:31:52 2026 +0100 Migrate silence detection to Silero VAD config vars Replaces deprecated SILENCE_DB and SILENCE_HANG_SEC with CODESCRIBE_VAD_THRESHOLD and CODESCRIBE_VAD_MAX_SILENCE_SEC throughout the codebase and documentation. Updates default values, environment variable handling, and comments to reflect the new Silero VAD-based configuration. Improves VAD initialization logic for thread safety and reliability. Updates documentation to clarify the deprecation and new configuration approach. � Conflicts: � .env.debug.example � core/config/default_env.txt � core/config/loader.rs commit 5b50465 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 01:22:41 2026 +0100 Refactor VAD config and initialization for Silero Updated VAD environment variables and documentation to use Silero-specific parameters, replacing legacy options. Added clamping for VAD config values to ensure valid ranges. Improved VAD worker initialization to confirm model loading with a timeout and propagate errors. Updated code to initialize VAD with the provided config instead of defaults. commit a797865 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 01:10:33 2026 +0100 Centralize VAD config and update usage across modules Moved VAD configuration to a centralized VadConfig in core/vad/config.rs. Updated streaming_recorder.rs to use the new VadConfig, removed the local VADConfig struct, and adjusted related logic and tests. Updated default_env.txt to reflect new VAD environment variable names and structure. Improved error handling for VAD initialization in recorder.rs. Removed unused import in silero_ort.rs. commit 9a3a811 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 01:05:01 2026 +0100 Refactor Silero VAD to use non-blocking worker and update deps Refactors the Silero VAD implementation to use a non-blocking, fire-and-forget worker thread with atomic probability updates, improving audio callback safety and performance. Updates ndarray-npy to 0.10 with default features disabled to remove unnecessary zip dependency, and removes zip and zopfli from dependencies. Adds max_utterance_sec to VadConfig. Cleans up unused silence_db config propagation in controller and improves resampling logic. commit 029033b Author: Szowesgad <void@div0.space> Date: Mon Jan 26 00:49:27 2026 +0100 Add Moshi conversation and E5 embedder modules Introduces a new conversation module with Moshi full-duplex voice AI, including configuration, context, engine, and turn-taking management. Adds an embedder module using fastembed for E5 text embeddings, with singleton/global access. Replaces WebRTC VAD with Silero VAD (ORT backend), updates dependencies (adds moshi, ort, fastembed, removes webrtc-vad), and refactors audio recorder to use VAD-based speech detection. Removes deprecated RMS-based silence detection and updates related code and tests. commit fecce50 Author: Szowesgad <void@div0.space> Date: Mon Jan 26 00:19:52 2026 +0100 Add TTS and VAD modules with embedded model support Introduces a new text-to-speech (TTS) module using the CSM-1B model with support for embedded model data, audio playback, and WAV export. Adds a neural voice activity detection (VAD) module using WebRTC VAD, replacing the previous RMS-based detection. Updates the streaming recorder to use the new VAD, and exposes public APIs for TTS and VAD. Updates build.rs to support embedding TTS models, and documents the new voice stack. Adds new dependencies: webrtc-vad. # Conflicts: # core/Cargo.toml # core/audio/streaming_recorder.rs # core/lib.rs # core/tts/audio_output.rs # core/tts/engine.rs # core/tts/mod.rs # core/tts/singleton.rs # core/vad/config.rs # core/vad/mod.rs # scripts/download-csm.sh commit 9bec9bc Author: Szowesgad <void@div0.space> Date: Sun Jan 25 22:32:28 2026 +0100 Remove legacy backend config and update cloud STT handling Eliminates legacy backend discovery, deprecated config fields, and AI provider selection. Cloud STT and LLM endpoints are now explicit and canonical in .env, with migration logic for legacy keys. Quality daemon state now tracks availability, and tray/menu UI reflects this. Removes unused Node client and related scripts. Refactors AI formatting to use a unified interface and removes legacy formatting code from CLI.[ok-commit] commit e8ba3ca Author: Szowesgad <void@div0.space> Date: Tue Jan 20 15:51:43 2026 +0100 Restructure project layout and update module paths Moved source files from 'src' and 'codescribe-core' to a new modular structure under 'app' and 'core' directories. Updated Cargo.toml and module imports to reflect the new paths, introduced namespacing for core modules, and added new mod.rs files for better organization. This refactor improves codebase maintainability and clarity.[ok-commit] Refactor voice chat overlay to support chat log and sending Reworked the voice chat overlay to maintain a chat log with user, assistant, and error messages. Added input field, send button, and support for streaming assistant responses. Introduced send callback mechanism and improved UI state management for sending, draft text, and error handling. Updated related controller logic to use new chat overlay API. Add persistent hands-off chat overlay with auto-send toggle Introduces a full-featured chat overlay for hands-off mode, including persistent chat history, user/assistant roles, and an input field with an auto-send toggle. The overlay UI is improved with reversed message flow (newest first), selectable text, and a new tray menu action to show the overlay. Voice Activity Detection (VAD) timeout is set to 5 seconds for hands-off mode, and short silences are ignored. The commit also adds a 'Copy Last to Clipboard' tray action, updates documentation, and includes tests for overlay persistence and auto-send logic. Refactor voice chat UI for split view and separate drafts Introduces a split view layout with distinct left (manual input) and right (voice streaming) panels. Separates manual and voice draft buffers, updates state management, and adjusts UI logic and tests to support the new structure. Also adds placeholder for attachment handling and updates default overlay dimensions for improved chat history visibility. Add voice draft management and UI enhancements Introduces voice draft saving, listing, reading, and deletion in history.rs, with corresponding API exports. Updates the voice chat overlay UI to include a right-side panel for drafts and settings, a collapse button, tab bar, and file attachment support. Also adds context menu for copying the last assistant response and refines layout logic. Removes obsolete files layout_patch.rs and test.txt. Add transcription overlay for non-assistive modes Introduces a new macOS transcription overlay (transcription_overlay.rs) for non-assistive recording modes, featuring a minimal floating window with live transcription, status display, and auto-hide. Refactors controller logic to route transcription deltas and overlay display based on session mode, and updates public API in lib.rs. Also updates dependencies, version to 0.7.3, and README formatting. Refactor UI code to use new ui_helpers module Introduced a new ui_helpers.rs module to encapsulate common Objective-C UI patterns and reduce msg_send! boilerplate. Updated transcription_overlay.rs, ui.rs, and voice_chat_ui.rs to use these helpers for window, view, color, and control management, improving code readability and maintainability. Add chat bubbles and drafts panel to voice chat UI Replaces the single text view with a bubble-based chat rendering using NSStackView, including support for streaming updates and per-message copy buttons. Adds a drafts panel in the right sidecar with a scrollable list of draft files, Edit/Copy actions, and animated drawer collapse/expand. Refactors UI state and helper functions to support these features. Always show transcription overlay and add transfer button Transcription overlay is now always shown for live preview, regardless of session mode, and the chat overlay is only opened intentionally by the user. Added a "Do chatu" button to the transcription overlay to allow transferring the current transcription to the voice chat overlay. Improved overlay state management and window delegate handling in voice chat UI to ensure proper cleanup and prevent use-after-free. Extract controller helpers into new module Splits various utility functions such as session state management, transcription delta routing, and voice chat callbacks into a new `helpers.rs` module. Cleans up `controller.rs` by delegating logic to `helpers` and `types`. Adds new unit tests in `tests.rs` to validate controller behavior. Enable always-on history and audio logging, simplify history handling - Default `history_enabled` and `dump_audio_logs` to `true` to ensure data preservation. - Removed toggles for history saving, audio dumping, and related tray menu options. - Simplified history submenu by unifying options into "Open history folder." - Refactored transcription configuration and helpers to enforce mandatory raw transcript saving. - Removed redundant handlers and menu construction logic for history and audio toggle options. - Addressed formatting consistency in `transcription_overlay.rs` for better readability and maintainability. Remove unused imports, redundant attributes, and improve safety in Objective-C calls - Cleaned up unnecessary `#[allow(unused_imports)]` and `#![allow(dead_code)]` attributes across multiple modules for better code hygiene. - Refactored Objective-C bindings to explicitly mark unsafe functions with comments, clarifying requirements and ensuring proper usage. - Renamed constants to SCREAMING_SNAKE_CASE for consistency with Rust coding standards. - Updated transcription logic to include configurable silence detection thresholds (`silence_db` and `silence_hang_sec`) through environment variables. - Enhanced Voice Activity Detection (VAD) session handling by resetting flags and adding custom callbacks in recorder configuration. - Improved Objective-C interop safety by wrapping `msg_send!` calls in `unsafe` blocks where applicable. - Simplified UI helpers and internals with clear separation of Objective-C safety boundaries in `ui_helpers.rs`. - Removed legacy and unused functions, including obsolete chat log rendering helpers. feat: “Streaming improvements with env vars” [ok-commit] [ok-commit] feat(cli): add streaming transcribe + live mode [ok-commit] chore: add codescribe repo runner [ok-commit] chore: support codex runner [ok-commit] chore: use repo mcp config chore: bump codescribe to version 0.7.5 - Updated `Cargo.toml` and `Cargo.lock` to reflect version change from `0.7.4-dev` to `0.7.5`. - Prepares for release with stable versioning for both `codescribe` and `codescribe-core`. - No functional changes introduced, focusing solely on version progression.[ok-commit] chore: remove unused dependencies from project - Cleaned up `Cargo.toml` and `Cargo.lock` by removing unused dependencies like `cpal`, `symphonia`, `candle-core`, `tokenizers`, `indicatif`, and others. [ok-commit] - Reduced dependency footprint for better maintenance and smaller build artifacts. - Ensured no functional changes to the project while simplifying dependency configuration. chore: cleanup and refactor project structure - Updated `.gitignore` to exclude CodeScribe runtime state files (`.codescribe/state/`). - Replaced the default model in `runner.sh` with `claude-opus-4-5-20251101` for consistency with current AI configurations. - Improved markdown formatting in `BACKLOG.md` and related documentation for better readability and alignment. - Expanded chip compatibility requirements in the guide to include Apple Silicon M5 (`README.md`). [ok-commit] - Removed deprecated Python backend management code (`backend.rs`) to streamline the codebase and reduce technical debt. Increase silence hang threshold and improve transcript post-processing Raised the default silence hang duration from 0.8s to 1.5s in environment, config, and audio recorder. Refactored transcript handling to always apply post-processing (lexicon, cleanup, semantic gate) before output, ensuring all output paths receive clean text. Updated embedder model references from BGEM3 to ParaphraseMLMiniLM. Removed unused files: examples/inspect_keys.rs, src/launchd.rs, and src/libraxis.rs. Added a background quality daemon process to main.rs for continuous self-improvement. [ok-commit] Adjust transcription overlay height with tailing Add in-app draft editing to voice overlay Refactor decision mode logic and add tests Replaces decision mode branching with unconditional state reset after recording finishes, ensuring paste works in all modes. Adds hot-reload support for custom lexicon rules, refactors daemon argument construction for testability, and improves UI event handling with additional safety. Includes comprehensive regression and contract tests for controller and lexicon logic. [ok-commit] Add commit recording button to transcription overlay (macOS) Introduces a 'Zakończ' (commit) button to the transcription overlay UI on macOS, allowing users to stop the current recording and enter decision mode immediately. Adds controller registration and commit request logic, and updates overlay state management to handle the new button's visibility and actions.[ok-commit] Enable test mode for audio-free controller tests Controller logic now skips backend health checks and audio device access when running in test mode, allowing tests to run without audio hardware. Removed #[ignore] attributes from controller tests, and added test-mode guards to voice draft finalization to prevent deadlocks. Added comprehensive tests to quality_loop.rs and introduced a detailed architecture vision document for the quality loop. [ok-commit] Bump version to 0.7.6 Update codescribe and codescribe-core package versions from 0.7.5 to 0.7.6 in Cargo.toml and Cargo.lock for a new release. [ok-commit] Remove completed stream chunk output inventory and task docs Deleted documentation files related to streaming chunk output inventory and implementation tasks for CodeScribe CLI transcription. These files contained requirements, context, and recommendations for improving streaming output in the CLI already implemented. [ok-commit] Refactor transcription to use single-pass engine API Replaces streaming chunked transcription with a single-pass engine call in quality report and IPC server, simplifying the flow and ensuring consistent chunking logic. Adds detailed logging and diagnostics to streaming_recorder, improves overlap deduplication with fuzzy matching in whisper engine, and updates .gitignore for new memory files. Also adds resume logic to quality report to skip already-processed pairs.[ok-commit] Refactor endpoint routing and streaming logic for LLM APIs Replaces domain-based endpoint detection with path-based routing using a new EndpointFormat enum. Streaming is now controlled by the LLM_USE_STREAMING environment variable, defaulting to enabled. Refactors Ollama and Responses API handling for clarity and reliability, and simplifies the API key check logic. Removes unused semgrep_pro_20260119.log file. [ok-commit] Add buffered streaming transcription with VAD Introduces a buffered transcription worker using VAD-based utterance segmentation and a typing-speed emitter for smoother streaming output. Adds VAD configuration, segmenter, and supporting tests. Updates StreamPostProcessor with process_utterance for VAD segments, and refactors chunk processing logic to support both buffered and unbuffered streaming modes. [ok-commit] Refactor voice chat UI and drawer, remove draft API Reworks the voice chat overlay UI to use a tabbed interface with Agent and Drawer tabs, removes the voice draft API and related code, and introduces a new card-based drawer for managing transcriptions. Updates the UI helpers with segmented control and card view helpers, and refactors state management and event handlers to support the new design. The commit also cleans up unused code and streamlines the overlay's public API.[ok-commit] Add clipboard copy and search field helpers to UI Introduces a `copy_to_clipboard` function for copying text to the system clipboard and a `create_search_field` helper for creating NSSearchField UI elements. Also updates E2E tests to set LLM_USE_STREAMING=0 for compatibility with mock server responses.[ok-commit] Add exhaustive env example, docs, and env tests Introduces .env.debug.example with all supported environment variables and safe defaults, a comprehensive Polish documentation (docs/env.md) detailing env usage, precedence, and requirements, and new Rust tests for validating required env vars and precedence logic in codescribe-core.[ok-commit] Clamp overlay window position to visible frame Introduces clamp_overlay_position to ensure overlay windows stay within the visible screen area, updating both transcription and voice chat overlays to use it. Also adds tests for the clamping logic and ensures overlays are brought to the front when shown. Documentation is updated to clarify which environment variables require restart or support hot reloading.[ok-commit] Refactor voice chat UI API and handlers Replaces send_agent_message_impl with send_draft_message_impl for improved draft handling. Adds new API functions: clear_voice_chat_text, filter_drawer, is_auto_send_enabled, and send_voice_chat_draft. Updates handler and API usage to use new function names and logic, and removes update_drawer_filter. Improves clarity and separation of concerns in voice chat overlay message sending and drawer filtering. [ok-commit] Add commit/discard for chat drafts and improve test config Introduces commit/discard action bar for draft user messages in the voice chat overlay, including new handler methods and UI updates. Refactors Makefile to load environment variables for tests and adds support for local LLM endpoints via TEST_USE_LOCAL_LLM. Updates .env.example and docs/env.md for simplified and clarified configuration. Reduces transcription overlay auto-hide delay to 5 seconds and prevents auto-hide when hovered. Updates tests and environment validation to support new behaviors. Bumps version to 0.7.7. [ok-commit] Add stream overlap ratio config and update defaults Introduces the CODESCRIBE_STREAM_OVERLAP_RATIO environment variable for finer control of audio chunk overlap in streaming mode. Updates default environment files and documentation to reflect new and revised configuration options, sets buffered streaming as default, and changes the default Ollama model. Also adds logic to enter decision mode after non-assistive recordings and improves .gitignore. [ok-commit]
Introduces the CODESCRIBE_LOOP_USE_CLOUD_STT environment variable to force cloud STT usage in the quality loop without changing app-wide defaults. Updates documentation and environment registry to reflect the new variable, and adds an end-to-end test to validate .env.example against the registry. Also corrects the default for CODESCRIBE_BUFFERED_STREAM in docs. [ok-commit]
Eliminates the CODESCRIBE_VAD_ENABLED flag and all code paths, callbacks, and documentation related to VAD-based auto-stop. VAD now only segments utterances (flushes) instead of stopping recordings, and all references to 'auto-stop' are updated to 'utterance boundary' or 'utterance flush'. Removes the e2e_vad_auto_stop.rs test, updates Makefile, config, and documentation to reflect the new segmentation-only behavior.[ok-commit]
This commit introduces embedded Silero VAD model support for release builds, refactors the streaming and buffered transcription pipeline to use a unified VAD gating mechanism, and separates lexicon postprocessing for buffered mode. It updates build.rs to embed the VAD model, adds core/vad/embedded.rs, and modifies the VAD loader to prefer embedded bytes. Tests and configuration are updated accordingly.[ok-commit]
Introduces a new VAD gate architecture with Supervisor mode, supporting pre-roll and speech padding for improved speech boundary detection. Adds `CODESCRIBE_VAD_GATE_MODE` and `CODESCRIBE_VAD_ITER` environment variables for gate [ok-commit]mode selection, updates documentation and environment examples, and refactors the streaming recorder to use the new gate configuration. Legacy VAD tuning variables are now ignored by the live gate, and documentation reflects these changes.
Refactors Silero VAD wrapper to support v5 model API, including unified state tensor, context window, and updated input/output handling. Adds detailed tests for VAD index synchronization and real audio segmentation in streaming_recorder.rs. Enhances logging and clarifies [ok-commit] Supervisor gate mode behavior.
Enhanced the VAD supervisor test to process multiple WAV files, including hardcoded edge cases and recent files. The test now reports segmentation quality, compares speech-only duration to total audio, and checks for hallucination markers in old transcripts. Improved output formatting and added assertions for speech detection. [ok-commit]
Summary of ChangesHello @Szowesgad, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly refines the Voice Activity Detection (VAD) system and its integration within the transcription pipeline. The primary goal is to enhance segmentation accuracy and boundary detection, while also providing greater flexibility in configuration. These changes aim to improve real-time transcription capabilities for both live and buffered modes, ultimately leading to a more robust and predictable user experience. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This is a massive and impressive pull request that significantly refactors and enhances the VAD integration, model management, and overall speech handling pipeline. The introduction of the "Supervisor" VAD mode, the upgrade to Silero VAD v5, and the removal of the legacy auto-stop logic are major steps forward in segmentation accuracy and robustness. The move to a pure-Candle embedder engine, removing the fastembed dependency, is a great architectural improvement. The extensive work on the build system, release scripts, and testing infrastructure, including the new round-trip tests and E2E environment validation, is commendable and will greatly improve maintainability. The addition of a user onboarding flow is a nice touch for improving user experience.
I have a few suggestions, mainly around code duplication and a minor formatting issue.
| # ============================================================================= | ||
| SHOW_TRAY_GLYPH=1 | ||
| HOLD_INDICATOR=1 # bool (legacy string values ignored) | ||
| HOLD_INDICATOR=1# bool (legacy string values ignored) |
There was a problem hiding this comment.
For better readability and to maintain consistency with the rest of the file, it's good practice to have a space before an inline comment.
HOLD_INDICATOR=1 # bool (legacy string values ignored)
Comment on lines
+287
to
+295
| fn apply_delta_with_backspace(target: &mut String, delta: &str) { | ||
| for ch in delta.chars() { | ||
| if ch == '\u{0008}' { | ||
| target.pop(); | ||
| } else { | ||
| target.push(ch); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
This helper function apply_delta_with_backspace is also defined in app/ui/overlay/mod.rs and bin/codescribe.rs. To avoid code duplication and improve maintainability, consider moving this function to a shared module, for example in app/ui/shared/helpers.rs, and making it public.
| LLM -->|SSE deltas| UI | ||
| CTRL -->|final result| PASTE[Paste / Clipboard]:::box | ||
| ``` | ||
| `flowchart TD %% Monochrome styling classDef default fill:#fff,stroke:#333,stroke-width:1px; classDef box fill:#f5f5f5,stroke:#666,stroke-width:1px; HK[Hotkey: Double Option]:::box --> CTRL[RecordingController]:::box CTRL -->|start()| REC[StreamingRecorder]:::box REC -->|f32 samples| ASR[Whisper Engine]:::box ASR -->|live chunks| POST[Stream Postprocess]:::box POST -->|delta callback| UI[Overlay: voice_chat_ui]:::box REC -->|silence detected| VAD[VAD watchdog]:::box VAD -->|finish_recording()| CTRL CTRL -->|raw transcript| LLM[AI Formatting / Assistive]:::box LLM -->|SSE deltas| UI CTRL -->|final result| PASTE[Paste / Clipboard]:::box` |
There was a problem hiding this comment.
The mermaid diagram seems to have been accidentally converted to a single line and wrapped in backticks, which will prevent it from rendering. Please restore it to a proper multi-line mermaid block to ensure it displays correctly.
Suggested change
| `flowchart TD %% Monochrome styling classDef default fill:#fff,stroke:#333,stroke-width:1px; classDef box fill:#f5f5f5,stroke:#666,stroke-width:1px; HK[Hotkey: Double Option]:::box --> CTRL[RecordingController]:::box CTRL -->|start()| REC[StreamingRecorder]:::box REC -->|f32 samples| ASR[Whisper Engine]:::box ASR -->|live chunks| POST[Stream Postprocess]:::box POST -->|delta callback| UI[Overlay: voice_chat_ui]:::box REC -->|silence detected| VAD[VAD watchdog]:::box VAD -->|finish_recording()| CTRL CTRL -->|raw transcript| LLM[AI Formatting / Assistive]:::box LLM -->|SSE deltas| UI CTRL -->|final result| PASTE[Paste / Clipboard]:::box` | |
| flowchart TD | |
| %% Monochrome styling | |
| classDef default fill:#fff,stroke:#333,stroke-width:1px; | |
| classDef box fill:#f5f5f5,stroke:#666,stroke-width:1px; | |
| HK[Hotkey: Double Option]:::box --> CTRL[RecordingController]:::box | |
| CTRL -->|start()| REC[StreamingRecorder]:::box | |
| REC -->|f32 samples| ASR[Whisper Engine]:::box | |
| ASR -->|live chunks| POST[Stream Postprocess]:::box | |
| POST -->|delta callback| UI[Overlay: voice_chat_ui]:::box | |
| REC -->|silence detected| VAD[VAD watchdog]:::box | |
| VAD -->|finish_recording()| CTRL | |
| CTRL -->|raw transcript| LLM[AI Formatting / Assistive]:::box | |
| LLM -->|SSE deltas| UI | |
| CTRL -->|final result| PASTE[Paste / Clipboard]:::box |
Replaced the footer in all documentation files from 'Created by M&K (c)2026 VetCoders' to 'Copyright © 2024–2026 VetCoders' for consistency and accuracy. [ok-commit]
Introduces a maximum buffer size in the demux parser to prevent unbounded memory growth and potential DoS attacks. Updates Whisper decoding parameters to use a 5-gram repetition block for improved quality and expands documentation for the WHISPER_INITIAL_PROMPT environment variable with usage guidance and examples.
m-szymanska
approved these changes
Jan 29, 2026
* Improve dedup overlap handling and add sink convenience Enhances dedup overlap logic in `strip_suffix_overlap` to use char boundaries, preventing panics with multi-byte UTF-8 (e.g., diacritics, emoji). Adds tests for these cases. In `chunker.rs`, gate-level pre_roll and speech_pad are now derived from VAD config and environment variables. Adds a `from_callback` convenience constructor to `sinks.rs` for easier external use, with corresponding test.[ok-commit] Refactor pipeline deduplication and delta sink handling “Virtual Commit: 5, 6, 7, 8”: Moves chunk and suffix deduplication logic into core/pipeline/dedup.rs and updates the streaming pipeline to use these unified helpers. Introduces DeltaSink trait and concrete sinks (CallbackSink, CollectorSink) in core/pipeline/sinks.rs, replacing direct Fn(&str) callbacks throughout the codebase. Updates all relevant controller and CLI code to use the new sink adapters. Adds integration and unit tests for deduplication and sink behavior. [ok-commit] Improve streaming VAD config and correction handling Adds a safety net to cap raw_buffer growth in chunker during silence, ensures VAD config respects environment overrides, and always applies postprocessing in streaming_recorder. In streaming, corrections are now guarded to prevent rewriting most of the text, and [ok-commit] last_suffix handling is improved to avoid state corruption when postprocessing filters out hallucinations. Includes new tests for suffix and correction guard logic. * Document CODESCRIBE_VAD_SPEECH_PAD_SEC env variable Added documentation for the new CODESCRIBE_VAD_SPEECH_PAD_SEC environment variable in ENV_REGISTRY.toml and env.md, describing its purpose as speech padding after speech offset and its default behavior. Also updated test module in chunker.rs to use serial_test for relevant tests.[ok-commit] * [ok-commit] feat(ui): bubble overlap fixes + multiline input (ranch Fala 1) Cherry-picked from origin/feat/ux-poc-assistive-context: - e9d63c2 fix(ui): stop overlay bubble overlap and enable scrolling - cc03eee fix(overlay): measure bubble height with NSString boundingRect - de934a3 fix(overlay): render full bubble text - acf3c4c fix(overlay): prevent Copy button overlap and bubble clipping - 2861914 fix(overlay): avoid narrow bubbles for long responses - ef9237c fix(overlay): widen streaming bubbles early - a08d944 fix(overlay): widen streaming bubbles sooner - 71cc1ad feat(overlay): multiline agent input Co-Authored-By: Klaudiusz <the1st@whoai.am> * [ok-commit] feat(ui): overlay stability fixes (ranch Fala 2) Cherry-picked from origin/feat/ux-poc-assistive-context: - aace207 fix(overlay): prevent header controls overlap - 7ccdec5 fix(overlay): keep tabs from overlapping header icons - 0828ea3 fix(ui): keep agent chat content visible - 72ac85f fix(ui): make overlay interactive and drawer usable - bd6eab1 fix(ui): force chat overlay visible for responses - 32364ec fix(ui): stabilize voice chat overlay updates - 19c0c33 fix(overlay): avoid deadlock when toggling favorites - 6bc72e3 fix(ui): avoid deadlock when closing voice overlay Also added: set_tooltip helper function. Co-Authored-By: Klaudiusz <the1st@whoai.am> * [ok-commit] fix(ui): re-show chat overlay on assistive output (ranch b05ad18) Ensure voice chat overlay is shown when setting user/assistant text in the controller. Applies to all assistive and formatting paths. Co-Authored-By: Klaudiusz <the1st@whoai.am> * [ok-commit] fix(overlay): enable Cmd+C/Cmd+V in agent app (ranch Fala 3 partial) Cherry-picked from origin/feat/ux-poc-assistive-context: - 3c4e5f1 fix(overlay): enable Cmd+C/Cmd+V in agent app Skipped 3bd43dc, f848580, 0a79ce2 (depend on selection.rs from excluded 27b8b1b). Co-Authored-By: Klaudiusz <the1st@whoai.am> * [ok-commit] feat(ui): context management + agent input (ranch Fala 4 clean) Cherry-picked from origin/feat/ux-poc-assistive-context: - 597952b fix(voice-chat): render overlay contents - 6693cb9 fix(ux): keep assistive overlay on Agent tab (excl. selection.rs) - d2eae30 fix(overlay): make agent input field editable Co-Authored-By: Klaudiusz <the1st@whoai.am> * feat(ui): context management + copy/paste support (ranch Fala 4b) Cherry-picked UI portions from origin/feat/ux-poc-assistive-context: - 34cffcc fix(voice-chat): show context + reduce dead overlay (UI parts) - 0fd49ba feat(voice-chat): add copy/paste last response (UI parts) Added: set_voice_chat_target_app API, last_target_app state field, copy/paste handler infrastructure. Skipped controller hunks from 34cffcc, 0fd49ba, 7d58432, 1f13d8c (heavy controller restructuring conflicts with DeltaSink/CallbackSink).[ok-commit] Co-Authored-By: Klaudiusz <the1st@whoai.am> * Add assistive context capture for selection in macOS Implements best-effort capture of selected text and frontmost app context for assistive mode on macOS, using Accessibility APIs and optional Cmd+C fallback. Updates RecordingController to store and use this context, modifies clipboard handling to support Cmd+C simulation, and adds new helpers in app/ui for retrieving selected text and its length. This enables richer context-aware AI interactions without polluting the clipboard.[ok-commit] Co-Authored-By: Klaudiusz <the1st@whoai.am> * Improve voice chat UX, safety, and accessibility Adds accessibility labels to UI elements, improves send button logic and labeling, and caps chat message history for performance. Introduces path safety checks for file operations in the voice chat overlay to prevent access outside the transcriptions directory. Refactors locking in message send/commit flows to avoid deadlocks. Updates VAD worker for clean shutdown and improves error handling in audio and model initialization. Adds new environment variables for assistive context and logging,and bumps version to 0.7.10. [ok-commit] * Update test and formatting commands in Makefile Consolidate test commands to use --include-ignored for running all tests, including those previously marked as ignored. Restrict Prettier formatting and checking to tracked files using git ls-files to avoid processing untracked files.[ok-commit] --------- Co-authored-by: Klaudiusz <the1st@whoai.am>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This pull request includes significant updates to the Voice Activity Detection (VAD) system and its integration into the transcription pipeline, with an emphasis on improving segmentation accuracy, boundary detection, and overall configuration flexibility.
Key Changes
Improved VAD Gate Configuration:
Supervisor mode.Silero VAD Model Upgrade:
Embedded VAD Model Support:
build.rsand configuration for managing embedded VAD bytes.Removed Legacy VAD Auto-Stop:
Enhancements and Testing
Documentation
.env.examplewith new environment variables (CODESCRIBE_VAD_GATE_MODE,CODESCRIBE_LOOP_USE_CLOUD_STT, etc.).env.md.Impact