NewConnectionError - urllib3.connection.HTTPConnection - Connection refused

[ezekieldas]

This is super low priority, not a blocker, easy to fix in most cases. I'm mainly posting for debate and maybe tips for newcomers to buildarr.

This is happening because one or more of the applications weren't fully initialized (in this case prowlarr) at the time buildarr was coming up. A subsequent buildarr run will almost always be successful.

I went round and round in looking for a means to avoid these failed connections. Container dependencies helped (eg, depends_on:) but brought other issues with it. There's also buildarr's request_timeout:.

At the core I think there could be some tweaks to Python requests/urllib3 that might help. I have at least one example I can pull up later in dealing with race conditions, flaky links, etc.

I can't recall if request_timeout: is passed to this but there are some additional parameters available. Here's one example: https://github.com/ingestbot/noisier/blob/main/noisier.py#L47

2024-04-27 16:24:27,073 buildarr:1 buildarr.cli.run [INFO] <sonarr> (default) Connection test successful
2024-04-27 16:24:27,073 buildarr:1 buildarr.cli.run [INFO] <prowlarr> (default) Fetching instance secrets
2024-04-27 16:24:27,075 buildarr:1 urllib3.connectionpool [WARNING] <prowlarr> (default) Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f298cdd8550>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/system/status
2024-04-27 16:24:27,075 buildarr:1 urllib3.connectionpool [WARNING] <prowlarr> (default) Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f298cdd93d0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/system/status
2024-04-27 16:24:27,076 buildarr:1 urllib3.connectionpool [WARNING] <prowlarr> (default) Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f298cbcd3d0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/system/status
2024-04-27 16:24:27,076 buildarr:1 buildarr.cli.daemon [ERROR] <prowlarr> (default) An error occurred while applying initial configuration: HTTPConnectionPool(host='localhost', port=9696): Max retries exceeded with url: /api/v1/system/status (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f298cbf2210>: Failed to establish a new connection: [Errno 111] Connection refused'))
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 400, in request
    self.endheaders()
  File "/usr/local/lib/python3.11/http/client.py", line 1298, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.11/http/client.py", line 1058, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.11/http/client.py", line 996, in send
    self.connect()
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 238, in connect
    self.sock = self._new_conn()
                ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 213, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f298cbf2210>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/buildarr/cli/daemon.py", line 215, in _initial_run
    run_apply()
  File "/usr/local/lib/python3.11/site-packages/buildarr/cli/run.py", line 208, in _run
    instance_secrets: SecretsPlugin = state.plugins[plugin_name].secrets.get(
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/buildarr_prowlarr/secrets.py", line 87, in get
    return cls.get_from_url(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/buildarr_prowlarr/secrets.py", line 145, in get_from_url
    system_status = prowlarr.SystemApi(api_client).get_system_status()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pydantic/decorator.py", line 40, in pydantic.decorator.validate_arguments.validate.wrapper_function
  File "pydantic/decorator.py", line 134, in pydantic.decorator.ValidatedFunction.call
  File "pydantic/decorator.py", line 206, in pydantic.decorator.ValidatedFunction.execute
  File "/usr/local/lib/python3.11/site-packages/prowlarr/api/system_api.py", line 579, in get_system_status
    return self.get_system_status_with_http_info(**kwargs)  # noqa: E501
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pydantic/decorator.py", line 40, in pydantic.decorator.validate_arguments.validate.wrapper_function
  File "pydantic/decorator.py", line 134, in pydantic.decorator.ValidatedFunction.call
  File "pydantic/decorator.py", line 206, in pydantic.decorator.ValidatedFunction.execute
  File "/usr/local/lib/python3.11/site-packages/prowlarr/api/system_api.py", line 670, in get_system_status_with_http_info
    return self.api_client.call_api(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prowlarr/api_client.py", line 413, in call_api
    return self.__call_api(resource_path, method,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prowlarr/api_client.py", line 218, in __call_api
    response_data = self.request(
                    ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prowlarr/api_client.py", line 439, in request
    return self.rest_client.get_request(url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-27 16:24:27,084 buildarr:1 buildarr.cli.daemon [INFO] Setting up signal handlers
  File "/usr/local/lib/python3.11/site-packages/prowlarr/rest.py", line 235, in get_request
    return self.request("GET", url,
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prowlarr/rest.py", line 201, in request
    r = self.pool_manager.request(method, url,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/_request_methods.py", line 136, in request
    return self.request_encode_url(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/_request_methods.py", line 183, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/poolmanager.py", line 444, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 877, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 877, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 877, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=9696): Max retries exceeded with url: /api/v1/system/status (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f298cbf2210>: Failed to establish a new connection: [Errno 111] Connection refused'))
2024-04-27 16:24:27,084 buildarr:1 buildarr.cli.daemon [WARNING] Aborted applying initial configuration
2024-04-27 16:24:27,085 buildarr:1 buildarr.cli.daemon [INFO] Finished setting up signal handlers
2024-04-27 16:24:27,085 buildarr:1 buildarr.cli.daemon [INFO] The next run will be at 2024-04-28 03:00
2024-04-27 16:24:27,085 buildarr:1 buildarr.cli.daemon [INFO] Buildarr ready.

[Callum027]

Thanks for the suggestion. Indeed, adding a retry mechanism for requests would make things more robust.

I'll make an issue where the details can be discussed (e.g. how it gets configured in Buildarr).

[Callum027]

Created issue: #209

[ezekieldas]

Partly inspired by this: https://stackoverflow.com/a/44459391

...and this: https://stackoverflow.com/a/35504626

The urllib3 docs befuddle me a bit though...