Finalize the VS Code speech API

[joyceerhl]

There have been many advancements in the voice model space in the last few months—Cartesia, Deepgram, ElevenLabs and Inworld AI to name a few. BYOK for language model chat has been fantastic, and it would be great to also enable BYOK for VS Code Speech so that more capable models with higher quality translation and better multilingual support are available for use in GitHub Copilot.

[bpasero]

@joyceerhl I like the idea but that is really not supported by the VS Code speech extension which relies on https://github.com/Microsoft/node-speech which depends on the Azure Speech SDK. That module is a native binding to the underlying SDK.

I think what would be more possible is to eventually make the proposed speech API public so that an extension can implement it as well:

vscode/src/vscode-dts/vscode.proposed.speech.d.ts

Lines 6 to 74 in f873c6e

	declare module 'vscode' {
	export interface SpeechToTextOptions {
	readonly language?: string;
	}
	export enum SpeechToTextStatus {
	Started = 1,
	Recognizing = 2,
	Recognized = 3,
	Stopped = 4,
	Error = 5
	}
	export interface SpeechToTextEvent {
	readonly status: SpeechToTextStatus;
	readonly text?: string;
	}
	export interface SpeechToTextSession {
	readonly onDidChange: Event<SpeechToTextEvent>;
	}
	export interface TextToSpeechOptions {
	readonly language?: string;
	}
	export enum TextToSpeechStatus {
	Started = 1,
	Stopped = 2,
	Error = 3
	}
	export interface TextToSpeechEvent {
	readonly status: TextToSpeechStatus;
	readonly text?: string;
	}
	export interface TextToSpeechSession {
	readonly onDidChange: Event<TextToSpeechEvent>;
	synthesize(text: string): void;
	}
	export enum KeywordRecognitionStatus {
	Recognized = 1,
	Stopped = 2
	}
	export interface KeywordRecognitionEvent {
	readonly status: KeywordRecognitionStatus;
	readonly text?: string;
	}
	export interface KeywordRecognitionSession {
	readonly onDidChange: Event<KeywordRecognitionEvent>;
	}
	export interface SpeechProvider {
	provideSpeechToTextSession(token: CancellationToken, options?: SpeechToTextOptions): ProviderResult<SpeechToTextSession>;
	provideTextToSpeechSession(token: CancellationToken, options?: TextToSpeechOptions): ProviderResult<TextToSpeechSession>;
	provideKeywordRecognitionSession(token: CancellationToken): ProviderResult<KeywordRecognitionSession>;
	}
	export namespace speech {
	export function registerSpeechProvider(id: string, provider: SpeechProvider): Disposable;
	}
	}

[joyceerhl]

Ack, then please consider this a feature request to finalize the speech API 🙏

[adamhill]

Awesomesauce!

@joyceerhl Thank you, thank you, thank you for getting the teams attention too! 🙏

I have been waiting for the Speech API to get finalized for a while, I got the upvotes to move it to the Backlog back in March it got added then gets deffered every Roadmap and I have been in a holding pattern ever since.

We have been working on A11y issues in Kilo and getting VS Code Speech working in Kilo has been on my bucketlist.

There are many users that are gonna be very, very happy when this lands and we can ship this feature in Stable.

And thanks @bpasero for pushing this forward! ❤️