Realtime ASR widget

[mishig25]

Add realtime capability to ASR widget

(attempt number 3 😊)

asrdemo.mov

Testing:
wav2vec2-base-960h-english 🇬🇧
wav2vec2-large-xlsr-53-french 🇫🇷
wav2vec2-large-xlsr-mongolian 🇲🇳

Screenshots

initial state	success state
token error	mic access error
warning: mic input is empty	warning: realtime asr output is empty

[mishig25]

@Pierrci regarding this comment, i'm using js rather than ts because microsoft/TypeScript#28308

[Pierrci]

did you try adding https://www.npmjs.com/package/@types/audioworklet?

[mishig25]

I've added @types/audioworklet and the following issues I'm facing are:

1. when it becomes js/static/audioProcessor.ts, it does not get transpiled to js (looking into why, possible error in adapter-static)
2. to solve number 1, I've moved js/static/audioProcessor.ts -> js/src/lib/components/InferenceWidget/shared/WidgetRealtimeRecorder/audioProcessor.ts. In this case, I'm getting error ReferenceError: AudioWorkletProcessor is not defined (probably due to SSR)
3. to solve number 2, I've put the entire context of js/static/audioProcessor.js inside WidgetRealtimeRecorder.svelte onMount function. Still got ReferenceError: AudioWorkletProcessor is not defined inside the browser log.

I might be missing something here. Please let me know if you have any ideas

[Pierrci]

Yes ok, this is 💩

The ReferenceError you were getting was probably because you're were trying to import the audio processor outside of the audioWorklet.addModule fn (AudioWorkletProcessor is only available inside the AudioWorkletGlobalScope)

When audioProcessor.ts is moved in the /shared dir, using imports as URL works correctly w/ the static adapter, but it doesn't work in moon-landing (obviously).

So I think we can keep it in /static (and as .js), and for it to work in moon-landing we will need to add an additional express static handler pointing to this file (which I'm not a fan of, but I don't see any other alternative?)

[mishig25]

yes, indeed, we do need that /static handler on moon-landing

[Pierrci]

Works well! 👍

[mishig25]

@gary149 Hiding the other btns of AutomaticSpeechRecognitionWidget when realtime asr is live

Testing:
wav2vec2-base-960h-english 🇬🇧
wav2vec2-large-xlsr-53-french 🇫🇷
wav2vec2-large-xlsr-mongolian 🇲🇳

[osanseviero]

Is there a netlify link I can try out? https://app.netlify.com/sites/huggingface-widgets/deploys/6274de11534f4a451af9c3ac does not seem to work 😢

[mishig25]

@osanseviero

Testing:
wav2vec2-base-960h-english 🇬🇧
wav2vec2-large-xlsr-53-french 🇫🇷
wav2vec2-large-xlsr-mongolian 🇲🇳

[julien-c]

this is cool but it looks like it was a LOT of work, let's make sure we maximize the impact by having as many people as possible use it

Any idea on how to best promote this?

[mishig25]

cc: @patrickvonplaten @osanseviero