TransportError: Unable to find the server at accounts.google.com

[araa47]

Hi I have developed a program to use google speech api to generate subtitle files from a large movie file. Currently, it converts a large file to 16kHz mono audio, performs VAD, then queues the list of chunks to a bunch of threads. The program errors out like below. I am using the python-docs-samples/speech/cloud-client/transcribe.py file to do the transcription using google speech api. The error comes up after a few minutes of running the program with 8 threads.

My code modification for transcribe.py looks like this:

import argparse
import io
from retrying import retry 

@retry(wait_exponential_multiplier=10, wait_exponential_max=1000)
def transcribe(speech_file, language):
  

    with io.open(speech_file, 'rb') as audio_file:
        content = audio_file.read()
        audio_sample = speech_client.sample(
            content=content,
            source_uri=None,
            encoding='LINEAR16',
            sample_rate_hertz=16000)
    try: 
        alternatives = audio_sample.recognize(language)
    except Exception as e:
        errormsg = "Exception in transcribe: " 
        errorout = errormsg + str(e)
        print errorout
        if str(e) == "No results returned from the Speech API.":
            print "**** Cleaned ****"
            return ""
        else:
            raise Exception("Retry!")
            return errorout

    for alternative in alternatives:
        answer = format(alternative.transcript)
        audio_file.close()
        print "Response: " + str(answer) 
        return answer 

Error Message 8 threads:

Exception in transcribe: GaxError(Exception occurred in retry method that was not classified as transient, caused by <_Rendezvous of RPC that terminated with (StatusCode.UNAUTHENTICATED, Traceback (most recent call last):
  File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 154, in grpc._cython.cygrpc.plugin_get_metadata (src/python/grpcio/grpc/_cython/cygrpc.c:6780)
  File "/Users/Akshay/Desktop/Work/a2t/v2/lib/python2.7/site-packages/grpc/_plugin_wrapping.py", line 106, in __call__
    AuthMetadataPluginCallback(wrapped_cygrpc_callback))
  File "/Users/Akshay/Desktop/Work/a2t/v2/lib/python2.7/site-packages/google/auth/transport/grpc.py", line 73, in __call__
    callback(self._get_authorization_headers(context), None)
  File "/Users/Akshay/Desktop/Work/a2t/v2/lib/python2.7/site-packages/google/auth/transport/grpc.py", line 61, in _get_authorization_headers
    headers)
  File "/Users/Akshay/Desktop/Work/a2t/v2/lib/python2.7/site-packages/google/auth/credentials.py", line 118, in before_request
    self.refresh(request)
  File "/Users/Akshay/Desktop/Work/a2t/v2/lib/python2.7/site-packages/google/oauth2/service_account.py", line 310, in refresh
    request, self._token_uri, assertion)
  File "/Users/Akshay/Desktop/Work/a2t/v2/lib/python2.7/site-packages/google/oauth2/_client.py", line 143, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/Users/Akshay/Desktop/Work/a2t/v2/lib/python2.7/site-packages/google/oauth2/_client.py", line 104, in _token_endpoint_request
    method='POST', url=token_uri, headers=headers, body=body)
  File "/Users/Akshay/Desktop/Work/a2t/v2/lib/python2.7/site-packages/google_auth_httplib2.py", line 119, in __call__
    raise exceptions.TransportError(exc)
TransportError: Unable to find the server at accounts.google.com
)>)

2nd Error Message 8 threads:

E0501 22:25:03.178845000 123145598861312 plugin_credentials.c:74]      Getting metadata from plugin failed with error: Traceback (most recent call last):
  File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 154, in grpc._cython.cygrpc.plugin_get_metadata (src/python/grpcio/grpc/_cython/cygrpc.c:6780)
  File "/Users/Akshay/Desktop/Work/a2t/v2/lib/python2.7/site-packages/grpc/_plugin_wrapping.py", line 106, in __call__
    AuthMetadataPluginCallback(wrapped_cygrpc_callback))
  File "/Users/Akshay/Desktop/Work/a2t/v2/lib/python2.7/site-packages/google/auth/transport/grpc.py", line 73, in __call__
    callback(self._get_authorization_headers(context), None)
  File "/Users/Akshay/Desktop/Work/a2t/v2/lib/python2.7/site-packages/google/auth/transport/grpc.py", line 61, in _get_authorization_headers
    headers)
  File "/Users/Akshay/Desktop/Work/a2t/v2/lib/python2.7/site-packages/google/auth/credentials.py", line 118, in before_request
    self.refresh(request)
  File "/Users/Akshay/Desktop/Work/a2t/v2/lib/python2.7/site-packages/google/oauth2/service_account.py", line 310, in refresh
    request, self._token_uri, assertion)
  File "/Users/Akshay/Desktop/Work/a2t/v2/lib/python2.7/site-packages/google/oauth2/_client.py", line 143, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/Users/Akshay/Desktop/Work/a2t/v2/lib/python2.7/site-packages/google/oauth2/_client.py", line 104, in _token_endpoint_request
    method='POST', url=token_uri, headers=headers, body=body)
  File "/Users/Akshay/Desktop/Work/a2t/v2/lib/python2.7/site-packages/google_auth_httplib2.py", line 119, in __call__
    raise exceptions.TransportError(exc)
TransportError: Unable to find the server at accounts.google.com

Error Message 12 threads:

E0501 22:42:36.493784000 123145542119424 wakeup_fd_pipe.c:52]          pipe creation failed (24): Too many open files
E0501 22:42:36.493804000 123145542119424 ev_poll_posix.c:898]          pollset_work: {"created":"@1493649756.493791000","description":"OS Error","errno":24,"file":"src/core/lib/iomgr/wakeup_fd_pipe.c","file_line":53,"os_error":"Too many open files","syscall":"pipe"}
E0501 22:42:36.493813000 123145542119424 completion_queue.c:433]       Completion queue next failed: {"created":"@1493649756.493791000","description":"OS Error","errno":24,"file":"src/core/lib/iomgr/wakeup_fd_pipe.c","file_line":53,"os_error":"Too many open files","syscall":"pipe"}
E0501 22:42:36.493827000 123145542119424 wakeup_fd_pipe.c:52]          pipe creation failed (24): Too many open files
E0501 22:42:36.493847000 123145542119424 ev_poll_posix.c:898]          pollset_work: {"created":"@1493649756.493834000","description":"OS Error","errno":24,"file":"src/core/lib/iomgr/wakeup_fd_pipe.c","file_line":53,"os_error":"Too many open files","syscall":"pipe"}
E0501 22:42:36.493855000 123145542119424 completion_queue.c:433]       Completion queue next failed: {"created":"@1493649756.493834000","description":"OS Error","errno":24,"file":"src/core/lib/iomgr/wakeup_fd_pipe.c","file_line":53,"os_error":"Too many open files","syscall":"pipe"}
E0501 22:42:36.493869000 123145542119424 wakeup_fd_pipe.c:52]          pipe creation failed (24): Too many open files
E0501 22:42:36.493889000 123145542119424 ev_poll_posix.c:898]          pollset_work: {"created":"@1493649756.493876000","description":"OS Error","errno":24,"file":"src/core/lib/iomgr/wakeup_fd_pipe.c","file_line":53,"os_error":"Too many open files","syscall":"pipe"}
E0501 22:42:36.493898000 123145542119424 completion_queue.c:433]       Completion queue next failed: {"created":"@1493649756.493876000","description":"OS Error","errno":24,"file":"src/core/lib/iomgr/wakeup_fd_pipe.c","file_line":53,"os_error":"Too many open files","syscall":"pipe"}
E0501 22:42:36.493911000 123145542119424 wakeup_fd_pipe.c:52]          pipe creation failed (24): Too many open files
E0501 22:42:36.493932000 123145542119424 ev_poll_posix.c:898]          pollset_work: {"created":"@1493649756.493918000","description":"OS Error","errno":24,"file":"src/core/lib/iomgr/wakeup_fd_pipe.c","file_line":53,"os_error":"Too many open files","syscall":"pipe"}
E0501 22:42:36.493941000 123145542119424 completion_queue.c:433]       Completion queue next failed: {"created":"@1493649756.493918000","description":"OS Error","errno":24,"file":"src/core/lib/iomgr/wakeup_fd_pipe.c","file_line":53,"os_error":"Too many open files","syscall":"pipe"}
E0501 22:42:36.493954000 123145542119424 wakeup_fd_pipe.c:52]          pipe creation failed (24): Too many open files
E0501 22:42:36.493975000 123145542119424 ev_poll_posix.c:898]          pollset_work: {"created":"@1493649756.493962000","description":"OS Error","errno":24,"file":"src/core/lib/iomgr/wakeup_fd_pipe.c","file_line":53,"os_error":"Too many open files","syscall":"pipe"}
E0501 22:42:36.493984000 123145542119424 completion_queue.c:433]       Completion queue next failed: {"created":"@1493649756.493962000","description":"OS Error","errno":24,"file":"src/core/lib/iomgr/wakeup_fd_pipe.c","file_line":53,"os_error":"Too many open files","syscall":"pipe"}
E0501 22:42:36.493997000 123145542119424 wakeup_fd_pipe.c:52]          pipe creation failed (24): Too many open files
E0501 22:42:36.494010000 123145542119424 ev_poll_posix.c:898]          pollset_work: {"created":"@1493649756.494002000","description":"OS Error","errno":24,"file":"src/core/lib/iomgr/wakeup_fd_pipe.c","file_line":53,"os_error":"Too many open files","syscall":"pipe"}
E0501 22:42:36.494015000 123145542119424 completion_queue.c:433]       Completion queue next failed: {"created":"@1493649756.494002000","description":"OS Error","errno":24,"file":"src/core/lib/iomgr/wakeup_fd_pipe.c","file_line":53,"os_error":"Too many open files","syscall":"pipe"}

[theacodes]

Looks like you're exhausting some resources (see the "too many open files") or having network trouble. I'm not sure this is an issue with the sample code, it seems to be more likely that your environment is causing some trouble. Can you tell us more about where you're running the code?

[araa47]

Hey, I am running on a Late 2015 iMac, with 16GB ram and a 3.3GHz Intel Core i7 processor, running macOS Sierra. I have tried to look for these file "src/core/lib/iomgr/wakeup_fd_pipe.c" on my mac, but it does not exist. I am wondering if this may be an error message from the google API server. All these error messages seem to be new messages that started coming up only in the last week. Before this, the program had no troubles running 12 threads.

[theacodes]

"Unable to find the server at accounts.google.com" is a DNS error, it means that we never even made the request to the server.

[datspike]

I'm having the same kind of errors as in the last example with a lot of threads and instances of recognitions in them running continiously.
I don't think there was such problem in v1beta, where i was able to close the recognititon with recognize_stream.cancel().

[theacodes]

/cc @lukesneeringer @dhermes

One thing you can try is using a new type of credential that doesn't involve network requests to refresh. You must be using service account credentials for this to work. See the code here.

[araa47]

@jonparrott

Thanks, will try out using the new type of credential and get back to you with my results.

[engelke]

Issue seems to be addressed above.