[FEATURE] VibeVoice-ASR for speaker diarization and domain-specific terminology

[TanaroSch]

Feature Description

Integrate Microsoft VibeVoice ASR as an optional transcription engine to provide unified "Rich Transcription," including native speaker diarization and domain-specific terminology support.

Problem Statement

Whisper and Parakeet don't support speaker diarization and customized context, which can lead to alignment errors and "speaker drift" in long meetings. Furthermore, transcribing niche technical or industry-specific terms accurately remains a challenge during meetings.

Proposed Solution

Leverage VibeVoice’s LLM-based architecture to:

1. Perform ASR and Diarization in a single pass for improved "who-said-what" accuracy.
2. Utilize its Context Prompting feature to inject meeting-specific keywords (project names, technical jargon) directly into the inference process.

User Story

As a meeting participant,
I want highly accurate transcripts that correctly identify speakers and technical terms,
So that my meeting minutes are actionable and require minimal manual correction.

Acceptance Criteria

• Option to select "VibeVoice" as the ASR provider in settings.
• Support for Context Prompts (allowing users to input meeting-specific terminology before processing).
• Speaker labels in the generated output (with manual override from speaker1 to Firstname Lastname).

Technical Considerations

• Hardware: VibeVoice uses a 7B parameter LLM backbone (Qwen 2.5), requiring more VRAM/compute than Whisper and Parakeet (though 4bit-quants already exist on Huggingface).
• Latency: It is slower than TDT-based models; it is best suited for asynchronous "post-meeting" processing rather than real-time captions.
• Language Support: Currently supports 50+ languages.

Alternatives Considered

• I don't know of any other ASR models that support both of those features.

Additional Context

Reference: Microsoft Research VibeVoice.

Checklist

• I have searched for similar feature requests
• I have provided all required information
• I have included any relevant screenshots/mockups
• I have described the problem and proposed solution clearly

[dpbattaglia-gps]

Microsoft VibeVoice-ASR is fantastic and a game changer for us. Definitely needs to be on a powerful GPU based server off the main meeting machine. So post transcription service (i.e. asynchronous) would be ideal and also requires the ability to choose an external transcription server.

[sujithatzackriya]

Would love to explore this but need more context. The model seems to be a 9B parameter model. Tried to run this locally on my 24GB Mac M2 and my system hung when the processing started. not ideal for on device deployment.

We are adding Openai compatible endpoint (Not ready with speaker We are adding an OpenAI-compatible endpoint (not yet ready with speaker diarization) to the new Pro version. yet) to the new pro version.

[dpbattaglia-gps]

VibeVoice-ASR transcript reply's with Speaker and timestamping built in. MLX-AUDIO allows VibeVoice-ASR through an openai like api however you can add a header to the api call under "Context" where you can add industry based words alongside the audio file for better transcription.

So in essence: adding a context header section (list of words) to add to the api call alongside the audio file and then formatting the response would probably be the only requirement you would need to add to meetily I think.

[KouissAbdessalam]

Anyone tried vibe voice? Is it better than whisper accuracy wise, is the delay trade-off worth it