Max retries exceeded caused by SSLError(SSLZeroReturnError

[lautaropacellaDR]

Hi, Im getting this error since mid June when before it was working well.

Max retries exceeded with url: /entities/x?fields=y (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1129)'))

I'm getting this error for all the external sources I try to request from this slack bot. Being storage, bigquery, cloud runs and cloud functions. I want to emphasise that this had only started happening in mid June, and before it was working well.

When I run the same repo locally, it works perfectly. This is why I think it could be something related to auth?
Sorry if not, I'm pretty new to this.

Any help would be much appreciated!

Environment details

Running on Cloud Run. Fast API and Slack SDK packages.
Python Version: 3.11.4

Requirements.txt:

anyio==3.7.1
cachetools==5.3.1
certifi==2023.5.7
charset-normalizer==3.2.0
dealroom-firestore-connector
dealroom-urlextract
fastapi==0.78.0
filelock==3.12.2
google-api-core==2.11.1
google-api-core[grpc]==2.11.1
google-auth==2.22.0
google-cloud-bigquery==3.11.3
google-cloud-core==2.3.3
google-cloud-firestore==2.11.1
google-cloud-pubsub==2.18.0
google-cloud-storage==2.10.0
google-cloud==0.34.0
google-crc32c==1.5.0
google-resumable-media==2.5.0
googleapis-common-protos==1.59.1
googleapis-common-protos[grpc]==1.59.1
grpc-google-iam-v1==0.12.6
grpcio-status==1.56.0
grpcio==1.56.0
gunicorn==20.1.0
idna==3.4
numpy==1.25.1
packaging==23.1
pandas==2.0.3
proto-plus==1.22.3
protobuf==4.23.4
pyasn1-modules==0.3.0
pyasn1==0.5.0
pydantic==1.10.11
python-dateutil==2.8.2
pytz==2023.3
requests-file==1.5.1
requests==2.31.0
rsa==4.9
setuptools==68.0.0
six==1.16.0
slack-bolt-router==0.1.1
slack-bolt==1.18.0
slack-sdk==3.21.3
sniffio==1.3.0
starlette==0.19.1
tldextract==3.4.4
typing-extensions==4.7.1
tzdata==2023.3
urllib3==1.26.16

Full error report:

  File "/usr/local/lib/python3.9/site-packages/slack_bolt/listener/thread_runner.py", line 120, in run_ack_function_asynchronously
    listener.run_ack_function(request=request, response=response)
  File "/usr/local/lib/python3.9/site-packages/slack_bolt/listener/custom_listener.py", line 50, in run_ack_function
    return self.ack_function(
  File "/app/app/main.py", line 108, in verification_submission
    _verification_submission(ack, payload, body, say)
  File "/app/app/commands/modal_actions.py", line 169, in _verification_submission
    messages, not_found = process_verification_payload(USER, profiles, action)
  File "/app/app/utils.py", line 152, in process_verification_payload
    entity_uuid = get_entity(clean_profile_url)["uuid"]
  File "/app/app/services/db_connector.py", line 19, in get_entity
    res = _session.get(f"entities/{identifier}", params={"fields": "uuid"})
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "/app/app/services/cloud_run_auth.py", line 299, in request
    return super(AuthorizedBaseUrlSession, self).request(
  File "/usr/local/lib/python3.9/site-packages/google/auth/transport/requests.py", line 549, in request
    response = super(AuthorizedSession, self).request(
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='db-connector-get-prod-x3nzdls4ha-ez.a.run.app', port=443): Max retries exceeded with url: /entities/verschuren_centre?fields=uuid (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1129)')))```


**Other errors related to this, that I also started getting at the same time:**

1_
 Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 191, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded

2_
 Traceback (most recent call last):
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 330, in _invoke_callbacks
    callback(self)
  File "/app/app/publish.py", line 18, in callback
    print(publish_future.result(timeout=60))
  File "/usr/local/lib/python3.9/site-packages/google/cloud/pubsub_v1/publisher/futures.py", line 66, in result
    return super().result(timeout=timeout)
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/google/cloud/pubsub_v1/publisher/_batch/thread.py", line 274, in _commit
    response = self._client._gapic_publish(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/pubsub_v1/publisher/client.py", line 267, in _gapic_publish
    return super().publish(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/google/pubsub_v1/services/publisher/client.py", line 831, in publish
    response = rpc(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 113, in __call__
    return wrapped_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func
    return retry_target(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 207, in retry_target
    raise exceptions.RetryError(
google.api_core.exceptions.RetryError: Deadline of 600.0s exceeded while calling target function, last exception: 504 Deadline Exceeded 

3_
 Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /storage/v1/b/x/o/x.csv?fields=name&prettyPrint=false (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1129)'))) 

4_
 Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http/__init__.py", line 482, in api_request
    response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http/__init__.py", line 341, in _make_request
    return self._do_request(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http/__init__.py", line 379, in _do_request
    return self.http.request(
  File "/usr/local/lib/python3.9/site-packages/google/auth/transport/requests.py", line 549, in request
    response = super(AuthorizedSession, self).request(
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Max retries exceeded with url: /bigquery/v2/projects/x/jobs?prettyPrint=false (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1129)')))

Thanks for your time!

[clundin25]

This is not an auth issue, but possibly an SSL issue.

There is not enough data in this stack trace to determine what the issue might be.

It seems that the connection is timing out when trying to reach these endpoints. Did you notice this issue on multiple endpoints? Are the errors isolated to Google? Are you able to establish a TLS session to the URLs that are having issues with something like OpenSSL's tls client?

[acolavin]

Hi @lautaropacellaDR , I wanted to mention that we ran into what seems like a highly related issue. We think it stems from how the PubSub PublisherClient handles batched message publishing and authentication, and went to some trouble to demonstrate how to reproduce it here: googleapis/python-pubsub#977

Happily, we also found a couple of workarounds that we document in the reproduction script. I hope that they can be useful to you.

Warmly,
Alex

[lautaropacellaDR]

Hi @acolavin! thanks for that, I did do a quick check but I think that would solve issues specifically when using pubsub.
The issue I'm having is triggered also when calling directly to other google cloud run or storage API.

Hello @clundin25, thanks for taking the time to check on this.
It does seems that is it a timeout. I have tried adding a function of asynchronicity to slack and adding more timeout limit to the container and it did seem to help, but from time to time, I still get this timeout error, specially when calling cloud storage API and another cloud run we use internally.
The most confusing part is that when I run it locally, it works fine, but the one that is deployed on the cloud tends to take much longer and run into these issues only sometimes.
Related to this:

Are you able to establish a TLS session to the URLs that are having issues with something like OpenSSL's tls client?

I'm very sorry @clundin25, but I'm not sure how to do this

[yu-iskw]

I have a similar issue on the different context using dbt-osmosis with BigQuery. I am doubting the issue is caused by conflicts between multpile threads in applications and the Google Cloud client. I am keep looking into it.

z3z1ma/dbt-osmosis#110

[Hsgngr]

I am getting the same error is there any news on this ?

[clundin25]

Can you share your stack trace @Hsgngr ?

[Hsgngr]

Unfortunately only stack trace I have is this because its coming from the Cloud Run. @clundin25 although my problem is not about google-auth-library The problem happens on my cloud Run as well. Probably this issue is related to that.

{
insertId: "654d60fa000066c165ab5f61"
jsonPayload: {
filename: "serper_service.py"
lineno: 49
message: "Error occurred: HTTPSConnectionPool(host='google.serper.dev', port=443): Max retries exceeded with url: /search (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1129)')))"
}
labels: {
resource: {
labels: {
}
type: "cloud_run_revision"

requests==2.31.0
fastapi==0.103.1

[Strimboli]

Hi @clundin25 - I am having a similar SSL Error that I have a full stack trace for (just anonymized my project and dataset name, can provide privately if needed):

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1077, in _send_output
    self.send(chunk)
  File "/usr/local/lib/python3.10/http/client.py", line 999, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python3.10/ssl.py", line 1270, in sendall
    v = self.send(byte_view[count:])
  File "/usr/local/lib/python3.10/ssl.py", line 1239, in send
    return self._sslobj.write(data)
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2426)

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 799, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Max retries exceeded with url: /bigquery/v2/projects/<project>/datasets/<dataset>/tables/bq_usershifts_raw_1700143203/insertAll?prettyPrint=false (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2426)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/google/api_core/retry.py", line 191, in retry_target
    return target()
  File "/usr/local/lib/python3.10/site-packages/google/cloud/_http/__init__.py", line 482, in api_request
    response = self._make_request(
  File "/usr/local/lib/python3.10/site-packages/google/cloud/_http/__init__.py", line 341, in _make_request
    return self._do_request(
  File "/usr/local/lib/python3.10/site-packages/google/cloud/_http/__init__.py", line 379, in _do_request
    return self.http.request(
  File "/usr/local/lib/python3.10/site-packages/google/auth/transport/requests.py", line 549, in request
    response = super(AuthorizedSession, self).request(
  File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Max retries exceeded with url: /bigquery/v2/projects/<project>/datasets/<dateset>/tables/bq_usershifts_raw_1700143203/insertAll?prettyPrint=false (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2426)')))

The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/opt/prefect/flows/ingestion/mongodb.py", line 82, in load_mongodb_by_page
    errors = bigquery_client.insert_rows(table, chunk)
  File "/usr/local/lib/python3.10/site-packages/google/cloud/bigquery/client.py", line 3470, in insert_rows
    return self.insert_rows_json(table, json_rows, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/google/cloud/bigquery/client.py", line 3666, in insert_rows_json
    response = self._call_api(
  File "/usr/local/lib/python3.10/site-packages/google/cloud/bigquery/client.py", line 789, in _call_api
    return call()
  File "/usr/local/lib/python3.10/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func
    return retry_target(
  File "/usr/local/lib/python3.10/site-packages/google/api_core/retry.py", line 207, in retry_target
    raise exceptions.RetryError(
google.api_core.exceptions.RetryError: Deadline of 600.0s exceeded while calling target function, last exception: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Max retries exceeded with url: /bigquery/v2/projects/<project>/datasets/<dataset>/tables/bq_usershifts_raw_1700143203/insertAll?prettyPrint=false (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2426)')))

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1584, in orchestrate_task_run
    result = await call.aresult()
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 181, in aresult
    return await asyncio.wrap_future(self.future)
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 194, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/prefect/flows/ingestion/mongodb.py", line 105, in load_mongodb_by_page
    raise Exception("Error Loading Chunk")

Exception: Error Loading Chunk

When attempting to debug locally, this error trace is then shown:

Traceback (most recent call last):
  File "/Users//ingestion/mongodb.py", line 86, in load_mongodb_by_page
    errors = bigquery_client.insert_rows(table, chunk)
  File "/Users//.venv/lib/python3.10/site-packages/google/cloud/bigquery/client.py", line 3470, in insert_rows
    return self.insert_rows_json(table, json_rows, **kwargs)
  File "/Users//.venv/lib/python3.10/site-packages/google/cloud/bigquery/client.py", line 3666, in insert_rows_json
    response = self._call_api(
  File "/Users//.venv/lib/python3.10/site-packages/google/cloud/bigquery/client.py", line 789, in _call_api
    return call()
  File "/Users//.venv/lib/python3.10/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func
    return retry_target(
  File "/Users//.venv/lib/python3.10/site-packages/google/api_core/retry.py", line 191, in retry_target
    return target()
  File "/Users/.venv/lib/python3.10/site-packages/google/cloud/_http/__init__.py", line 494, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.GoogleAPICallError: 413 POST https://bigquery.googleapis.com/bigquery/v2/projects/<project>/datasets/<dataset>/tables/bq_usershifts_raw_1700155784/insertAll?prettyPrint=false: <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 413 (Request Entity Too Large)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>413.</b> <ins>That’s an error.</ins>
  <p>Your client issued a request that was too large.
 <script>
  (function() { var c=function(a,d,b){a=a+"=deleted; path="+d;null!=b&&(a+="; domain="+b);document.cookie=a+"; expires=Thu, 01 Jan 1970 00:00:00 GMT"};var g=function(a){var d=e,b=location.hostname;c(d,a,null);c(d,a,b);for(var f=0;;){f=b.indexOf(".",f+1);if(0>f)break;c(d,a,b.substring(f+1))}};var h;if(4E3<unescape(encodeURI(document.cookie)).length){for(var k=document.cookie.split(";"),l=[],m=0;m<k.length;m++){var n=k[m].match(/^\s*([^=]+)/);n&&l.push(n[1])}for(var p=0;p<l.length;p++){var e=l[p];g("/");for(var q=location.pathname,r=0;;){r=q.indexOf("/",r+1);if(0>r)break;var t=q.substring(0,r);g(t);g(t+"/")}"/"!=q.charAt(q.length-1)&&(g(q),g(q+"/"))}h=!0}else h=!1;
h&&setTimeout(function(){if(history.replaceState){var a=location.href;history.replaceState(null,"","/");location.replace(a)}},1E3); })();

</script>
 <ins>That’s all we know.</ins>

[Hsgngr]

Hi Strimboli, is this also Cloud Run that tries to connect with BigQuery or local ?

[Strimboli]

Hi @Hsgngr - it is running in a kubernetes job, not local.

[priye-1]

Any solutions to this? I've been on it for weeks, any help will be greatly appreciated pls!

I am also getting the same error on Cloud Run:
2024-01-03 10:07:47,668 - ERROR - Error occurred: Deadline of 600.0s exceeded while calling target function, last exception: HTTPSConnectionPool(host='oauth2.googleapis.com', port=443): Max retries exceeded with url: /token (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))

This happens when I call the Bigquery client for the second time on Cloud Run or even when I call Google Sheets for the second time and subsequently. It works perfectly when I run my app locally.

Traceback from google sheet call:

"Traceback (most recent call last):
    spreadsheet = gc.open_by_url(sheet_url)
  File "/usr/local/lib/python3.10/site-packages/gspread/client.py", line 227, in open_by_url
    return self.open_by_key(extract_id_from_url(url))
  File "/usr/local/lib/python3.10/site-packages/gspread/client.py", line 206, in open_by_key
    spreadsheet = Spreadsheet(self, {"id": key})
  File "/usr/local/lib/python3.10/site-packages/gspread/spreadsheet.py", line 37, in __init__
    metadata = self.fetch_sheet_metadata()
  File "/usr/local/lib/python3.10/site-packages/gspread/spreadsheet.py", line 245, in fetch_sheet_metadata
    r = self.client.request("get", url, params=params)
  File "/usr/local/lib/python3.10/site-packages/gspread/client.py", line 80, in request
    response = getattr(self.session, method)(
  File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/google/auth/transport/requests.py", line 542, in request
    response = super(AuthorizedSession, self).request(
  File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='sheets.googleapis.com', port=443): Max retries exceeded with url: /v4/spreadsheets/1qgqVLxWZAuZTnkcWSS7fjI0oJMWy1F7fvFyb3C_JgwI?includeGridData=false (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))"

During handling of the above exception, another exception occurred:
 File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 714, in urlopen
      httplib_response = self._make_request(
 File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 403, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.10/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.10/ssl.py", line 1375, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 798, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='sheets.googleapis.com', port=443): 
Max retries exceeded with url: /v4/spreadsheets/<spreadsheet_id>?includeGridData=false (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))

[GoooGu]

I have the same issue when I call gspread on Cloud Run. It runs fine on local.

[max-l-weaver]

Having this same issue now. Running the auth and storage Python libraries and getting the same error arround Max retries exceeded and SSL error. As other commentators have said it works fine on local and the issue is intermittent, i.e. works sometimes on Cloud Run and not others, with no indication or code changes between revisions.

[GoooGu]

So I have turned on no-cpu-throttling on my cloud run and the problem went away. I think it is due to the fact that it needs stable network environment to make external network calls. However turning on no-cpu-throttling incurred more cost.

I am still figuring out a way to mitigate this without spending too much.

[jamessystem]

I'm experiencing the same issue with pods inside our GKE cluster writing to bigquery. The strange part is it happens with some pods and not all.

[priye-1]

The only way I was able to solve this was to create a cloud function that specifically writes to Google sheet and big query, hence my app on cloud run calls cloud function using a httpx client.
Not sure if this is an optimized way but for now it works

[jamessystem]

I'm experiencing the same issue with pods inside our GKE cluster writing to bigquery. The strange part is it happens with some pods and not all.

We've made some code changes (so far, it's working; I'll know more next week). The issue was we were writing too much data to a big query. One of our JSON payloads contained megabytes of data.

[tanel3203]

Same issue here. Pretty much identical setup and trace as the OP.

[max-l-weaver]

Having this same issue now. Running the auth and storage Python libraries and getting the same error arround Max retries exceeded and SSL error. As other commentators have said it works fine on local and the issue is intermittent, i.e. works sometimes on Cloud Run and not others, with no indication or code changes between revisions.

As frustrating as this is, my issue just seems to have resolved itself with no changes on our end. It hasn't come back now for a couple of weeks so I believe it's stable, although it would be great to know what is causing this.

[GoooGu]

LOL same error still occurs to me.

[tuliocaviquioli]

I am having the same error while trying to connect to ClickHouse.

[alex-feel]

I've encountered the SSLZeroReturnError while executing code in Cloud Run, specifically when calling a function to send data to BigQuery as a background task. Locally, everything works without any issues. The error logs indicate retries after connection breaks by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1129)')), with the final attempt resulting in Max retries exceeded with url: /token (Caused by SSLError(SSLZeroReturnError....

The issue persists despite using the following library versions:

• google-auth-oauthlib==1.2.0
• google-cloud-bigquery==3.17.2
• fastapi==0.109.2
• uvicorn==0.27.1
• gunicorn==21.2.0

This seems to be a similar issue to what others have experienced, as discussed in this thread. Has anyone found a reliable solution or workaround for Cloud Run environments?

[NikOcaml]

Also getting this error - for both BigQuery and Cloud Storage calls from Cloud Run.

[djcarrillo]

I have the same error, I use this version:
google-cloud-bigquery==3.17.2
sqlalchemy_bigquery==1.9.0
google-cloud-bigquery-storage==2.24.0
cloud-sql-python-connector==1.2.3

[iftachAvital]

+1

[clundin25]

I'm going to close this, as I believe this issue is unrelated to the code maintained in this repository.

From what I can tell this is a networking issue for large payloads. If anyone can reproduce this consistently, please share and we can try to track down a solution.

Otherwise, the best bet is to open a support case (https://cloud.google.com/support/) and collaborate with the support team to collect a trace of what happened on the server side. The client side logs are opaque and I believe there is little we can do without an investigation on the server end.

[menghong-square]

This is because by default cloudrun only allocate CPU during requests, it means that in background task there is no CPU allocated, and no CPU == no egress network.

This can be change by adding --no-cpu-throttling to the service

ref: https://cloud.google.com/run/docs/configuring/cpu-allocation