SSL Certificate verification issue

[Spikey123]

Hello.

I've been using tswast's method to query BigQuery results and it's been purring along fine for a while now. Over the last several days, I've been getting an SSL certificate verification issue. Below is the full error message and below that is the code I use.

I'll note that I updated the python client, bigquery library, cloud storage library, and gcloud components.
pip install --upgrade google-api-python-client
pip install --upgrade google-cloud-bigquery
pip install --upgrade google-cloud-storage
gcloud components update

Service account permissions should be fine

Update: I'm getting the same exact error on other scripts I have running - like this one that deletes blobs in Google Cloud Storage:

Traceback (most recent call last):
  File "sfyExport/sfyExport.py", line 334, in <module>
    compression="GZIP" if args.gzip else "NONE")
  File "sfyExport/sfyExport.py", line 203, in main
    didDelete = delete_blobs_matching("mwg-bigqueryexport", deleteString)
  File "sfyExport/sfyExport.py", line 50, in delete_blobs_matching
    blobs = list_blobs(bucket_name)
  File "sfyExport/sfyExport.py", line 38, in list_blobs
    bucket = storage_client.get_bucket(bucket_name)
  File "/usr/lib/python2.7/site-packages/google/cloud/storage/client.py", line 173, in get_bucket
    bucket.reload(client=self)
  File "/usr/lib/python2.7/site-packages/google/cloud/storage/_helpers.py", line 99, in reload
    _target_object=self)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 290, in api_request
    headers=headers, target_object=_target_object)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 183, in _make_request
    return self._do_request(method, url, headers, data, target_object)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 212, in _do_request
    url=url, method=method, headers=headers, data=data)
  File "/usr/lib/python2.7/site-packages/google/auth/transport/requests.py", line 176, in request
    self._auth_request, method, url, request_headers)
  File "/usr/lib/python2.7/site-packages/google/auth/credentials.py", line 121, in before_request
    self.refresh(request)
  File "/usr/lib/python2.7/site-packages/google/oauth2/service_account.py", line 310, in refresh
    request, self._token_uri, assertion)
  File "/usr/lib/python2.7/site-packages/google/oauth2/_client.py", line 143, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/usr/lib/python2.7/site-packages/google/oauth2/_client.py", line 104, in _token_endpoint_request
    method='POST', url=token_uri, headers=headers, body=body)
  File "/usr/lib/python2.7/site-packages/google/auth/transport/requests.py", line 115, in __call__
    raise exceptions.TransportError(exc)
google.auth.exceptions.TransportError: HTTPSConnectionPool(host='accounts.google.com', port=443): Max retries exceeded with url: /o/oauth2/token (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)'),))

I checked the certs on accounts.google.com and they appeared to be fine. So I'm not sure why the SSL error would be thrown.

Thanks!

Traceback (most recent call last):
  File "bqTests.py", line 295, in <module>
    args.num_retries)
  File "bqTests.py", line 254, in main
    failCount = async_query("SELECT Count(*) from [Tests.{0}]".format(bqTest["tempTable"]))
  File "bqTests.py", line 104, in async_query
    query_job.begin()
  File "/usr/lib/python2.7/site-packages/google/cloud/bigquery/job.py", line 380, in begin
    method='POST', path=path, data=self._build_resource())
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 290, in api_request
    headers=headers, target_object=_target_object)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 183, in _make_request
    return self._do_request(method, url, headers, data, target_object)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 212, in _do_request
    url=url, method=method, headers=headers, data=data)
  File "/usr/lib/python2.7/site-packages/google/auth/transport/requests.py", line 176, in request
    self._auth_request, method, url, request_headers)
  File "/usr/lib/python2.7/site-packages/google/auth/credentials.py", line 121, in before_request
    self.refresh(request)
  File "/usr/lib/python2.7/site-packages/google/oauth2/service_account.py", line 310, in refresh
    request, self._token_uri, assertion)
  File "/usr/lib/python2.7/site-packages/google/oauth2/_client.py", line 143, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/usr/lib/python2.7/site-packages/google/oauth2/_client.py", line 104, in _token_endpoint_request
    method='POST', url=token_uri, headers=headers, body=body)
  File "/usr/lib/python2.7/site-packages/google/auth/transport/requests.py", line 115, in __call__
    raise exceptions.TransportError(exc)
google.auth.exceptions.TransportError: HTTPSConnectionPool(host='accounts.google.com', port=443): Max retries exceeded with url: /o/oauth2/token (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)'),))

The code is:

87- # [START wait_for_job]
88- def wait_for_job(job):
89-     while True:
90-        job.reload()  # Refreshes the state via a GET request.
91-        if job.state == 'DONE':
92-            if job.error_result:
93-                raise RuntimeError(job.errors)
94-            return
95-        time.sleep(1)
96- # [END wait_for_job]

99- # [START async_query]
100- def async_query(query):
101-    client = bigquery.Client()
102-    query_job = client.run_async_query(str(uuid.uuid4()), query)
103-    query_job.use_legacy_sql = True
104-    query_job.begin()
105-
106-    wait_for_job(query_job)
107-
108-    rows = query_job.results().fetch_data(max_results=10)
109-    
110-    for row in rows:
111-        return row[0]
112-    
113- # [END async_query]

[lukesneeringer]

Hi @Spikey123,
Thanks for reporting. Even though it will not change the audience for your issue much, I am going to move this over to google-auth-library-python, which is the actual source of the error you are getting.

[lukesneeringer]

This issue was moved to googleapis/google-auth-library-python#192