Result event not triggered on file upload

[Clement-mim]

Hello, I am working on a way to pass audio file to the recognizer all at once.

I took the react example and edited file-upload.tsx to send the whole file as buffer to the AudioStreamer "_write" method.
The problem reside on the "result" event of the recognizer not being fired after the process.
The "partialresult" event is called with every words but misses timestamps.

Here is the implementation of the "onChange" function in file-upload.tsx:

const onChange = useCallback(
    async ({ file }: UploadChangeParam<UploadFile<any>>) => {

      if (
        recognizer &&
        file.originFileObj &&
        file.percent === 100
      ) {
        const fileUrl = URL.createObjectURL(file.originFileObj);
        const _audioContext = audioContext ?? new AudioContext();
        const arr = await fetch(fileUrl).then((res) => res.arrayBuffer());

        _audioContext.decodeAudioData(arr, (buffer) => {
          let audioStreamer = new AudioStreamer(recognizer);
          audioStreamer._write(buffer, {
            objectMode: true,
          }, () => {
            console.log('done')
          });
        });
      }
    },
    [audioContext, recognizer]
  );

I have also noticed when uploading a second file it works well, the result event is triggered and includes both files data.

What I am missing? Is there a way to dispatch a "result" event?

[erikh2000]

This might be more of a workaround than directly addressing what you see, but...

You don't need AudioStreamer for your use case. You can call AcceptWaveForm() directly, and probably just once with a single AudioBuffer.

(unofficial response--I'm not the maintainer or involved with the project)

[Clement-mim]

Thanks for the fast reply! Indeed AudioStreamer is not usefull in my case. I can replace the function in decodeAudioData to just recognizer.acceptWaveform(buffer)

Unfortunately the error persist: partialresult is fired without result.
I also noticed something weird: by duplicating the line recognizer.acceptWaveform(buffer) the second call end by triggering result event (with both audios data).

This is what I have now:

_audioContext.decodeAudioData(arr, (buffer) => {
    recognizer.acceptWaveform(buffer);
    // Duplicate line to get results:
    recognizer.acceptWaveform(buffer);
});

[ccoreilly]

Hi @Clement-mim !

I assume your audio contains little silence at the end or the model does not have sufficient information to decide it has reached an endpoint (end of utterance; the probability of the recognized text ending a sentence is low). Due to this fact, the end result event is not emitted.

When you send the audio twice, the recognizer received double the audio and even if there is not much silence at the end of the audio, the length of the utterance is enough to trigger an endpoint detection.

Once you have finished sending audio you should finalize recognition by removing the recognizer with recognizer.remove() which will trigger the FinalResult method of the Vosk recognizer and emit the result event. After removing the recognizer you'll need to create a new one.

However, I see that this event will not be dispatched to the listeners as is it removed from the list of recognizers before the event is received. I will work on a fix.

Moreover, there should be no need to remove and recreate a recognizer to force a FinalResult. I will update the API so it can be triggered independently.

If you feel you can contribute these changes let me know! Otherwise I'll make time for them.

[Clement-mim]

Hello, thanks for pointing me in the right direction. I have just created a PR with a new function "sendResults" that trigger the "result" event.

In my code I use it in the "partialresult" callback function (in recognizer.tsx) since I only have one buffer: the whole file.

It looks like this:

fileRecog.on("partialresult", (message: any) => {
    setPartial(message.result.partial);
    fileRecog.sendResults();
});