proper partial s2t for less latency on iOS

[calvma]

for issue #29 . Currently, iOS speech to text waits until after audio is done recording to do the translation as opposed to while audio is being recorded. By simply moving around the speech recognizer code, getting text back in to unity happens way faster, thus improving the app experience in iOS

[Korbain]

Hi Michael,

Thanks very much. I got it to work! I did have to do 2 changes, however, to get to work.
1)
In SpeechRecorderViewController.mm, you need to add an else statement (or else, since you are waiting for the isFinal flag, the partial results never get sent to Unity):

            if (result.isFinal) {
                UnitySendMessage("SpeechToText", "onResults", [transcriptText UTF8String]);
            }
            else
            {
                UnitySendMessage("SpeechToText", "onPartialResults", [transcriptText UTF8String]);
            }

In TextSpeech.cs, you need to add the onPartialResults for iOS. Currently this code exists but only for IF_UNITY_ANDROID.
Just move code out of the define so that it becomes available for iOS as well.

I am talking about this code:

    public Action<string> onPartialResultsCallback;
    /** Called when partial recognition results are available. */
    public void onPartialResults(string _params)
    {
        if (onPartialResultsCallback != null)
            onPartialResultsCallback(_params);
    }

Thanks again,

Korbain

[Korbain]

Hi Michael,

Thanks very much. I got it to work! I did have to do 2 changes, however, to get to work.
1)
In SpeechRecorderViewController.mm, you need to add an else statement (or else, since you are waiting for the isFinal flag, the partial results never get sent to Unity):

            if (result.isFinal) {
                UnitySendMessage("SpeechToText", "onResults", [transcriptText UTF8String]);
            }
            else
            {
                UnitySendMessage("SpeechToText", "onPartialResults", [transcriptText UTF8String]);
            }

In TextSpeech.cs, you need to add the onPartialResults for iOS. Currently this code exists but only for IF_UNITY_ANDROID.
Just move code out of the define so that it becomes available for iOS as well.

I am talking about this code:

    public Action<string> onPartialResultsCallback;
    /** Called when partial recognition results are available. */
    public void onPartialResults(string _params)
    {
        if (onPartialResultsCallback != null)
            onPartialResultsCallback(_params);
    }

Thanks again,

Korbain

[Korbain]

In addition, you need to stop the recognizer task in StopRecording or else it will continuously spam you with the same results

• (void)stopRecording {
  if (audioEngine.isRunning) {
  [inputNode removeTapOnBus:0];
  [audioEngine stop];
  [recognitionRequest endAudio];
  if (recognitionTask) {
  [recognitionTask cancel];
  }
  NSLog(@"STOPRECORDING");
  }
  }

[anita-web]

Hi @Korbain, I am a design student studying AR and I've been using the plugin for my project and it's been super helpful! Very glad it exists. I've been following a few youtube tutorials but now would really love to get partial s2t to work in my project but can't quite seem to understand how to get the partial results to unity. I don't know if you have time for a random stranger but would really appreciate some help! - Ana

[Korbain]

Hi Anita, sure. I’ll see what I can do to help. Did you do/understand the 3 points I identified above?

[Znoleg]

@Korbain Hi, Korbain! Thanks for your input and information.
But i didn't understand where to put your listed changes. I've never worked in swift + I don't quite understand how this code works :) So, can you just send me "SpeechRecorderViewController.mm" file?
Thanks!

[Korbain]

Hi, I just posted my class on Github in Implement Partial Results for iOS #29 Regis

…

Sent from my iPhone

On Mar 4, 2021, at 3:30 PM, Miroshkin Oleg ***@***.***> wrote:  @Korbain Hi, Korbain! Thanks for your input and information. I wanna ask: so, to use partialcallback on ios you need to enable it for UNITY_IOS directive and all will work? Like this: #if UNITY_ANDROID || UNITY_IOS public Action onPartialResultsCallback; /* Called when partial recognition results are available. */ public void onPartialResults(string _params) { if (onPartialResultsCallback != null) onPartialResultsCallback(_params); } #endif — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.