Texttospeech, 429 Received message larger than max

[vinaysraghavan]

I get an error when trying to synthesize speech over ~1500 characters (4MiB) despite documentation stating the limit is 5000 characters. There is no problem synthesizing 5000 characters when using MP3 encoding (output file size: 1.2MB) on the same text, indicating that the problem is definitely just the larger file size when using LINEAR16 encoding (output file size: >13MB). This seems to be related to issues googleapis/google-cloud-python#5819, googleapis/google-cloud-python#5574

Environment details

API:

google-cloud-texttospeech 0.5.0

OS:

macOS 10.15.3

Python version:

Python 3.7.6

Google Cloud Service Info:

Name: google-cloud-texttospeech
Version: 0.5.0
Summary: Google Cloud Text-to-Speech API client library
Home-page: https://github.com/GoogleCloudPlatform/google-cloud-python
Author: Google LLC
Author-email: googleapis-packages@google.com
License: Apache 2.0
Location: /Users/vinayraghavan/anaconda3/envs/aad/lib/python3.7/site-packages
Requires: google-api-core
Required-by:

Steps to reproduce

1. Synthesize text longer than 1500 characters with en-US Wavenet voices and LINEAR16 encoding.

Code example

from google.cloud import texttospeech
import os
from tqdm import tqdm
import numpy as np

CORPUS = 'MothClean'
GENDER = 'Male1'
voice_name = 'en-US-Wavenet-D'
pitch = 0
            
dirname = 'Text/'+CORPUS+'/'
outdirname = 'Speech/'+CORPUS+'/'+GENDER+'/'

# Instantiates a client
client = texttospeech.TextToSpeechClient()
        
langCode = voice_name[:5]

# Build the voice request, select the language code
voice = texttospeech.types.VoiceSelectionParams(language_code=langCode,name=voice_name)

# Select the type of audio file you want returned
audio_config = texttospeech.types.AudioConfig(
    audio_encoding = texttospeech.enums.AudioEncoding.LINEAR16,
    speaking_rate = 1,
    pitch = pitch)

# Get items in directory with text files here
items = os.listdir(dirname)
filelist = sorted([fn for fn in items if '.txt' in fn])
        
# Iterate through text files, performing TTS and saving audio as we go
for file in tqdm(filelist):
    with open(dirname+file, 'r') as text_file:
        text = text_file.read()
        text = text[:3000]

    # Set the text input to be synthesized
    synthesis_input = texttospeech.types.SynthesisInput(text=text)

    # Perform the text-to-speech request on the text input with the selected
    # voice parameters and audio file type
    response = client.synthesize_speech(synthesis_input, voice, audio_config)

    name = file[0:-4]

    with open(outdirname+name+'.wav', 'wb') as out:
        # Write the response to the output file.
        out.write(response.audio_content)

Stack trace

Traceback (most recent call last):
  File "/Users/vinayraghavan/anaconda3/envs/aad/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/Users/vinayraghavan/anaconda3/envs/aad/lib/python3.7/site-packages/grpc/_channel.py", line 550, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Users/vinayraghavan/anaconda3/envs/aad/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.RESOURCE_EXHAUSTED
	details = "Received message larger than max (8351413 vs. 4194304)"
	debug_error_string = "{"created":"@1581435404.899284000","description":"Received message larger than max (8351413 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":174,"grpc_status":8}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "gcloud_tts.py", line 92, in <module>
    response = client.synthesize_speech(synthesis_input, voice, audio_config)
  File "/Users/vinayraghavan/anaconda3/envs/aad/lib/python3.7/site-packages/google/cloud/texttospeech_v1/gapic/text_to_speech_client.py", line 322, in synthesize_speech
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/Users/vinayraghavan/anaconda3/envs/aad/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/Users/vinayraghavan/anaconda3/envs/aad/lib/python3.7/site-packages/google/api_core/retry.py", line 273, in retry_wrapped_func
    on_error=on_error,
  File "/Users/vinayraghavan/anaconda3/envs/aad/lib/python3.7/site-packages/google/api_core/retry.py", line 182, in retry_target
    return target()
  File "/Users/vinayraghavan/anaconda3/envs/aad/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/Users/vinayraghavan/anaconda3/envs/aad/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.ResourceExhausted: 429 Received message larger than max (8351413 vs. 4194304)

[busunkim96]

Tentatively triaging as a bug to appease the SLO bots. @nnegrey Would you be able to take a look?

[nnegrey]

@alexander-fenster, could this be some GRPC size limit thing?

[vinaysraghavan]

That sounds right based on this googleapis/google-cloud-python#8860

[alexander-fenster]

@nnegrey Definitely a gRPC problem, not much we (as a client library) can do here. Could you please talk to an API team?

[nnegrey]

Yea, I'll reach out.

[nnegrey]

Howdy, mind trying again?

Was able to get this to work now, there were some backend changes made.

[nnegrey]

I'm going to close this out as it should be fixed, but if it isn't, please let me know and I'll reopen.

[vinaysraghavan]

I'm still getting the same error

Traceback (most recent call last):
  File "/Users/vinayraghavan/anaconda3/envs/aad/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/Users/vinayraghavan/anaconda3/envs/aad/lib/python3.7/site-packages/grpc/_channel.py", line 550, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Users/vinayraghavan/anaconda3/envs/aad/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.RESOURCE_EXHAUSTED
	details = "Received message larger than max (7020499 vs. 4194304)"
	debug_error_string = "{"created":"@1583773500.735648000","description":"Received message larger than max (7020499 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":174,"grpc_status":8}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "gcloud_tts.py", line 94, in <module>
    voice, audio_config)
  File "/Users/vinayraghavan/anaconda3/envs/aad/lib/python3.7/site-packages/google/cloud/texttospeech_v1/gapic/text_to_speech_client.py", line 322, in synthesize_speech
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/Users/vinayraghavan/anaconda3/envs/aad/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/Users/vinayraghavan/anaconda3/envs/aad/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    on_error=on_error,
  File "/Users/vinayraghavan/anaconda3/envs/aad/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
    return target()
  File "/Users/vinayraghavan/anaconda3/envs/aad/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/Users/vinayraghavan/anaconda3/envs/aad/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.ResourceExhausted: 429 Received message larger than max (7020499 vs. 4194304)

[nnegrey]

My bad, I forgot to switch my test over to LINEAR16. I've reopened the issue with the product team.

[busunkim96]

@nnegrey Do you have the bug number for this?

[busunkim96]

Internal issue was marked fixed on May 18. Googlers see 150287650.

Thanks Noah!

[FarnoodF]

I still have the same issue with google-cloud-texttospeech=2.1.0. Could you please look into it?

[busunkim96]

@telpirion Could you take another look?

[prohilla]

ANy update on this? Facing the same issue.

[telpirion]

I can't reproduce this issue with a text payload of ~1550 characters.

However, I can reproduce this with a text payload of ~4400 characters. I will escalate to the Text-to-Speech team.

[telpirion]

Googlers, see 170125537.

[prohilla]

@telpirion any update on this? Need to decide if we should keep using the python client. Will really appreciate it. Thanks.

[telpirion]

@prohilla Yes! There is a work around. Using the principle described here, you can change the options used to create the Text-to-Speech client so that it accepts a larger message size.

Steps

Assuming that you're using Text-to-Speech v1:

1. In your local copy of the python-texttospeech client library, open up google/cloud/texttospeech_v1/services/text_to_speech/transports/grpc.py.

2. Change this line to the following:

    options=[('grpc.max_receive_message_length', 24*1024*1024)]
   

I'm going to close this issue for now. Please re-open if you continue to see this problem.

[busunkim96]

@telpirion I believe the same max applies in all the grpc implementations. It might be worth tracking a FR to increase the defaults in the library somehow.

Programmatic way to do the above:

from google.cloud import texttospeech_v1
from google.cloud.texttospeech_v1.services.text_to_speech.transports.grpc import (
    TextToSpeechGrpcTransport,
)

channel = TextToSpeechGrpcTransport.create_channel(
    options=[("grpc.max_receive_message_length", 24 * 1024 * 1024)]
)
transport = TextToSpeechGrpcTransport(channel=channel)
client = texttospeech_v1.TextToSpeechClient(transport=transport)