C++ continuous speech recognition

[ubhatti]

Hello all,

I am trying to use Google cloud Speech for speech-to-text using streaming API. The samples work for me fine, however my usecase is a bit different: I would like to do continuous, streaming speech recognition (multiple requests). So I face two issues:

1. It looks like I cannot use grpc::ClientReaderWriter<StreamingRecognizeRequest, StreamingRecognizeResponse> for multiple requests because I get an exception: assertion failed: call_ == nullptr client_context.cc. So, I have to recreate complete config (credentials, channel, context, config request) for each request and I don't find it elegant because it adds unnecessary overhead.

2. To avoid any latency due to the config creation I create it beforehand and wait for the audio samples from mics. However, if the audio samples arrive a bit late I get Audio Timeout Error: Long duration elapsed without audio. Audio should be sent close to real time.

So my question(s): What is the correct way to use google streaming API in C++ for multiple requests: Due we have to renew objects for each recognition requests and Is there any way to avoid timeout?

I am working on Ubuntu 14.04 LTS. The speech API code was downloaded in April 2018.

Thanks.

[beccasaurus]

1. For clarification, do your multiple concurrent streaming requests work OK? So long as you have a new channel/context/* for each separate request?

2. Hmm I've seen this before but have not personally tried real time streaming in awhile

I believe we have at least one sample showing how to open up a stream and continuously get results, although not in C++

You may / may not find some of these helpful...

• transcribe_streaming_indefinite.py
• Performing Streaming Speech Recognition on an Audio Stream (C#, Go, Java, Node, Python)

@nirupa-kumar Have you run into this error when doing infinite streaming in Java?

Other potentially helpful /cc's... @gguuss @nnegrey @SurferJeffAtGoogle

[ubhatti]

Thanks for looking into it.

1. For clarification, do your multiple concurrent streaming requests work OK? So long as you have a new channel/context/* for each separate request?

I am developing a software for interaction so the user can do multiple, non-overlapping/non-concurrent speech recognition requests on a single device. According to my understanding I need a separate context/channel for each subsequent request. Although, I would have prefered to just renew the context for each request without needing to create channel/credentials, etc.

2. Hmm I've seen this before but have not personally tried real time streaming in awhile

There is a mention of this problem in another issue regarding node js:
googleapis/nodejs-speech#62 (comment)
Also, the issue can be reproduced in the sample file streaming_transcribe.cc by adding this line before launching microphone thread:
std::this_thread::sleep_for(std::chrono::seconds(12));

I believe we have at least one sample showing how to open up a stream and continuously get results, although not in C++

You may / may not find some of these helpful...

• transcribe_streaming_indefinite.py
• Performing Streaming Speech Recognition on an Audio Stream (C#, Go, Java, Node, Python)

Yeah, I have looked into these earlier but I was not sure what implementation of grpc/protobuf stack these were using. Does Python use the same source code for GRPC/protobuf as C++ does?

Thanks again.

@nirupa-kumar Have you run into this error when doing infinite streaming in Java?

Other potentially helpful /cc's... @gguuss @nnegrey @SurferJeffAtGoogle

[coryan]

According to my understanding I need a separate context/channel for each subsequent request. Although, I would have prefered to just renew the context for each request without needing to create channel/credentials, etc.

You should be able to reuse the channel and credentials and send a second streaming request over the previous channel. gRPC can multiplex multiple streams over a single channel.

Yeah, I have looked into these earlier but I was not sure what implementation of grpc/protobuf stack these were using. Does Python use the same source code for GRPC/protobuf as C++ does?

Different protobuf, same gRPC core. There is a thin C++ wrapper on top of the core gRPC library, and a separate thin Python wrapper on top of the same core gRPC library.

[nirupa-kumar]

While working with the Java streaming samples, have been able to send multiple streams over a single channel.
There was one instance where I had seen this error occur. Modifying the duration in Thread.sleep(duration), and the buffer size for reading the data fixed it for me.

[tmatsuo]

@nirupa-kumar I think we can close this issue, but let us know if you think differently.