Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multithreaded API clients connection pools should allow more connections #946

Closed
Yomguithereal opened this issue Mar 26, 2024 · 0 comments
Closed

Comments

@Yomguithereal
Copy link
Member

This happens with at least the Twitter and Facebook clients/scrapers.

WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: api.twitter.com. Connection pool size: 1
minet.crowdtangle.exceptions.CrowdTanglePostNotFound
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: m.facebook.com. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: m.facebook.com. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: m.facebook.com. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: m.facebook.com. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: m.facebook.com. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: m.facebook.com. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: m.facebook.com. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: m.facebook.com. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: m.facebook.com. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: m.facebook.com. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: m.facebook.com. Connection pool size: 1
ERROR:root:
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/minet/crowdtangle/client.py", line 96, in __request
    return data["result"]
           ~~~~^^^^^^^^^^
KeyError: 'result'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/minall/enrichment/crowdtangle/client.py", line 93, in __call__
    post = self.client.post(post_id=post_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/minet/crowdtangle/client.py", line 115, in post
    return crowdtangle_post(self.request, post_id, token=self.token, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/minet/crowdtangle/post.py", line 21, in crowdtangle_post
    data = request(api_url)
           ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/minet/rate_limiting.py", line 255, in decorated
    result = fn(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/minet/crowdtangle/client.py", line 102, in request
    return self.__request(url)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/minet/web.py", line 1345, in decorated
    return retryer(fn, self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/minet/web.py", line 1209, in __call__
    return super().__call__(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/tenacity/__init__.py", line 379, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/tenacity/__init__.py", line 314, in iter
    return fut.result()
           ^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/tenacity/__init__.py", line 382, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/minet/crowdtangle/client.py", line 98, in __request
    raise CrowdTanglePostNotFound(data=data, url=url)
minet.crowdtangle.exceptions.CrowdTanglePostNotFound
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: m.facebook.com. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: m.facebook.com. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: m.facebook.com. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: m.facebook.com. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: m.facebook.com. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: m.facebook.com. Connection pool size: 1
ERROR:root:<urllib3.connection.HTTPSConnection object at 0x7f07dccce490>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: 1scandal.com. Connection pool size: 1
ERROR:root:<urllib3.connection.HTTPSConnection object at 0x7f07dccccc90>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution
ERROR:root:<urllib3.connection.HTTPSConnection object at 0x7f07dce36f90>: Failed to establish a new connection: [Errno -2] Name or service not known
ERROR:root:<urllib3.connection.HTTPSConnection object at 0x7f07dca9b0d0>: Failed to establish a new connection: [Errno -2] Name or service not known
ERROR:root:<urllib3.connection.HTTPSConnection object at 0x7f07dca9a6d0>: Failed to establish a new connection: [Errno -2] Name or service not known
ERROR:root:Maximum number of redirects exceeded
ERROR:root:[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
ERROR:root:HTTPSConnectionPool(host='archive.ph', port=443): Read timed out. (read timeout=25)
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: planetes360.fr. Connection pool size: 1
ERROR:root:('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: archive.ph. Connection pool size: 1
ERROR:root:HTTPSConnectionPool(host='archive.ph', port=443): Read timed out. (read timeout=25)
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: reseauinternational.net. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: t.me. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: web.archive.org. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: www.bfmtv.com. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: www.bfmtv.com. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: www.france.tv. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: www.francetvinfo.fr. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: www.lemonde.fr. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: www.publicsenat.fr. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: www.radiofrance.fr. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: www.radiofrance.fr. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: www.tf1info.fr. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: x.com. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: www.tf1info.fr. Connection pool size: 1
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: twitter.com. Connection pool size: 1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant