Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

discovery.build( ) raises ResourceWarning: unclosed <ssl.SSLSocket fd=13, #618

Closed
ramonmedeiros opened this issue Jan 30, 2019 · 13 comments
Closed

Comments

@ramonmedeiros
Copy link

@ramonmedeiros ramonmedeiros commented Jan 30, 2019

Environment details

  • OS:
    Linux hu3br 4.18.16-300.fc29.x86_64 #1 SMP Sat Oct 20 23:24:08 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

  • Python version:
    Python 3.6.7

  • pip version:
    pip 19.0.1 from /home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages/pip (python 3.6)

  • google-api-python-client version:

Name: google-api-python-client
Version: 1.7.8
Summary: Google API Client Library for Python
Home-page: http://github.com/google/google-api-python-client/
Author: Google Inc.
Author-email: None
License: Apache 2.0
Location: /home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages
Requires: google-auth-httplib2, six, uritemplate, google-auth, httplib2
Required-by:

Steps to reproduce

Instantiate a dataflow, v1b3 API with googleapis.discovery

> /home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages/bigtable_backups/dataflow.py(117)__init__()
-> self.api_service = build(API_COMPONENT, API_VERSION, cache_discovery=False).projects().locations()
(Pdb) n
/home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/.
  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
/home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages/googleapiclient/_helpers.py:130: ResourceWarning: unclosed <ssl.SSLSocket fd=13, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('10.43.33.41', 48878), raddr=('74.125.133.95', 443)>
  return wrapped(*args, **kwargs)
> /home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages/bigtable_backups/dataflow.py(120)__init__()

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

@berendo
Copy link

@berendo berendo commented Oct 2, 2019

We're also seeing this while using other API endpoints (other than dataflow) when -Wall is in effect for python.

@busunkim96
Copy link
Contributor

@busunkim96 busunkim96 commented Oct 2, 2019

@berendo Could you elaborate on what -Wall is?

@berendo
Copy link

@berendo berendo commented Oct 3, 2019

Could you elaborate on what -Wall is?

Sure, it’s running Python with python -Wall. See the -W option here.

@busunkim96
Copy link
Contributor

@busunkim96 busunkim96 commented Oct 14, 2019

@berendo Thank you!

This is straightforward to reproduce:

from googleapiclient import discovery

client = discovery.build("dataflow", "v1b3")
$ python3 -Wall make-client.py 
~/.local/lib/python3.6/site-packages/googleapiclient/_helpers.py:130: ResourceWarning: unclosed <ssl.SSLSocket fd=3, family=AddressFamily.AF_INET6, type=2049, proto=6, laddr=('2620:15c:10f:202:e603:43de:a156:cc2', 33710, 0, 0), raddr=('2607:f8b0:400a:809::200a', 443, 0, 0)>

@busunkim96
Copy link
Contributor

@busunkim96 busunkim96 commented Oct 14, 2019

The request to fetch the discovery doc results in the ResourceWarning.

from googleapiclient import discovery
import requests
import httplib2

# Doesn't raise socket resource warning
dataflow_discovery_doc = requests.get("https://dataflow.googleapis.com/$discovery/rest?version=v1b3").json()
client = discovery.build_from_document(dataflow_discovery_doc)

# Raises socket resource warning
client = discovery.build("dataflow", "v1b3")

# Function to retrieve public discovery doc
doc = discovery._retrieve_discovery_doc("https://www.googleapis.com/discovery/v1/apis", httplib2.Http(), cache_discovery=False)

httplib2 seems to leave sockets open when requests are successful.

import httplib2

http = httplib2.Http()
http.request("http://example.com")

I'm not sure if there's a good way to ask that the connection be closed in httplib2. I filed httplib2/httplib2#148 to ask the maintainers.

From similar discussions on requests (psf/requests#3912) it may be okay to let this be.

@busunkim96 busunkim96 self-assigned this Oct 14, 2019
@busunkim96 busunkim96 changed the title discovery dataflow v1b3 raises ResourceWarning: unclosed <ssl.SSLSocket fd=13, discovery.build( ) raises ResourceWarning: unclosed <ssl.SSLSocket fd=13, Oct 14, 2019
@busunkim96 busunkim96 added external and removed 🚨 labels Oct 14, 2019
@wintermelons
Copy link

@wintermelons wintermelons commented Dec 5, 2019

Since httplib2/httplib2#148 is now closed, any plans to add .close() to discovery?

@busunkim96
Copy link
Contributor

@busunkim96 busunkim96 commented Dec 5, 2019

@wintermelons Thank you for the ping! It looks like httplib2 hasn't yet made a release with the change.

@wintermelons
Copy link

@wintermelons wintermelons commented Dec 30, 2019

FYI: httplib2 has made a new release 🙂

@imjuanleonard
Copy link

@imjuanleonard imjuanleonard commented Jul 2, 2020

Bumping this up for the code sanity, any updates?

@mike9005
Copy link

@mike9005 mike9005 commented Aug 5, 2020

This is causing a lot of disruption for us - it causes containers running Airflow tasks against GCP to consistently run out of available sockets.

@busunkim96
Copy link
Contributor

@busunkim96 busunkim96 commented Aug 6, 2020

Hello folks,

Apologies for leaving this unattended for so long.

f this is actively causing pain you can manually call the httplib2.http.close() method when you are done with the client.

from googleapiclient import discovery

client = discovery.build("drive", "v3")

client._http.http.close()

Ideally the library should clean up httplib2.http it creates. httplib2 instances could be shared across clients so I think it will require some care to get the logic correct.

@haizaar
Copy link

@haizaar haizaar commented Aug 20, 2020

@ramonmedeiros Can you please elaborate why did you close this issue?

@danoscarmike danoscarmike reopened this Aug 27, 2020
busunkim96 added a commit that referenced this issue Sep 23, 2020
Fixes #618 🦕

httplib2 leaves sockets open by default. This can lead to situations where programs run out of available sockets. httplib2 added a method last year to clean up connections. https://github.com/httplib2/httplib2/blob/9bf300cdc372938f4237150d5b9b615879eb51a1/python3/httplib2/__init__.py#L1498-L1506

This PR adds two ways to close http connections. The interface is intentionally similar to the proposed design for GAPIC clients. googleapis/gapic-generator-python#575
@haizaar
Copy link

@haizaar haizaar commented Sep 23, 2020

@busunkim96 Thank you so much for fixing this! Looking forward for the next release!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

9 participants