Show whisper.cpp: Integrating offline Whisper transcription into an infinite 2D canvas AI workspace (Knovum, Tauri 2 + Rust)

[madebyzan]

Hi whisper.cpp community! 👋

We’ve recently open-sourced Knovum — a local-first desktop AI knowledge space built on an infinite 2D canvas with Tauri 2, Rust, and Vue 3.

One of our primary design goals was making multimodal speech ingestion truly seamless for long-form voice memos, interviews, and meetings without ever streaming private audio to third-party clouds.

How We Integrated Whisper with Tauri 2 and Rust

1. Sidecar Pipeline & Audio Preprocessing

   • We package Whisper sidecar binaries and models directly alongside the native desktop application bundle.
   • Long recordings dragged onto the canvas are normalized and converted into 16kHz mono WAV through an embedded FFmpeg pipeline.
   • Whisper decodes audio into timestamped segments which are mapped directly to spatial Markdown cards on the 2D infinite canvas.

2. Decoupled IPC & 60fps Canvas Performance

   • High-frequency transcription progress events are throttled and dispatched across the Tauri IPC boundary using dedicated Tauri Channels.
   • The UI thread (rendering hardware-accelerated Konva 2D canvas nodes) stays completely unblocked and holds steady at 60fps while background Whisper processes run at full speed on CPU/Metal.

3. Hybrid Local RAG on Transcribed Text

   • Once transcription completes, segments are dual-indexed:
     • SQLite FTS5: For exact word/keyword recall.
     • Embedded LanceDB: For vector semantic search.
   • Users can query their audio library using local LLMs (via Ollama) with sentence-level citation anchors that jump straight to the source timestamp on the canvas.

Source Code

The full implementation is open source under Apache-2.0:

• GitHub Repository: https://github.com/madebyzan/knovum
• Audio pipeline: Check src/features/audio/ and src-tauri/

Huge thanks to @ggerganov and the whisper.cpp contributors for making local, high-performance speech transcription possible on consumer machines! We would love any feedback or thoughts on our desktop audio pipeline.

[madebyzan]

🎙️ Update: Knovum v0.0.2 is released with Auditionable Voice Library & Active Speaker Visualization

In our v0.0.2 release today, local speech capabilities received a great UX boost:

• Auditionable Voice Library: You can now listen to and preview each local voice profile directly before choosing it for audio cards.
• Active Speaker Visualization: The UI actively visualizes and highlights the currently speaking profile during audio playback and transcription playback.

Release downloads and full notes:
https://github.com/madebyzan/knovum/releases/tag/v0.0.2