Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Connections not sustained and get constant reconnects. #245

Closed
Nornode opened this issue Jul 26, 2023 · 16 comments
Closed

Connections not sustained and get constant reconnects. #245

Nornode opened this issue Jul 26, 2023 · 16 comments

Comments

@Nornode
Copy link

Nornode commented Jul 26, 2023

I'm having issues keeping the connection stable.
Both instances on 2023.07.3 with updated remote HA addon installed through HACS.
Instances/nodes reachable locally (over VPN) on two different /24 subnets (.10 & .11)
I have regenerated the authentication tokens and ensured that the user is allowed to log in (tested dedicated & my own user).

On the "client node" I get constant:
2023-07-26 23:06:32.369 ERROR (MainThread) [custom_components.remote_homeassistant] websocket connection had an error

And at a few times I get:

This error originated from a custom integration.

Logger: custom_components.remote_homeassistant
Source: custom_components/remote_homeassistant/rest_api.py:47
Integration: Remote Home-Assistant (documentation, issues)
First occurred: 22:32:08 (3 occurrences)
Last logged: 22:32:28

failed to connect
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1085, in create_connection
    raise exceptions[0]
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1069, in create_connection
    sock = await self._connect_sock(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 973, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 634, in sock_connect
    return await fut
           ^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 674, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('192.168.10.14', 8123)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/config/custom_components/remote_homeassistant/__init__.py", line 373, in _async_instance_get_info
    return await async_get_discovery_info(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/remote_homeassistant/rest_api.py", line 47, in async_get_discovery_info
    async with session.get(url, headers=headers) as resp:
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1209, in _create_direct_connection
    raise last_exc
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1178, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 988, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host 192.168.10.14:8123 ssl:default [Connect call failed ('192.168.10.14', 8123)]

Loggbook is filling up with:
Screenshot 2023-07-26 at 23 13 39

On the "server node" I don't see any error in logs.

If not the following is related:

2023-07-26 22:58:55.909 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 358, in async_handle
    msg = await wsock.receive()
          ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_ws.py", line 440, in receive
    await self.close()
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_ws.py", line 355, in close
    await writer.drain()
  File "/usr/local/lib/python3.11/site-packages/aiohttp/http_writer.py", line 171, in drain
    await self._protocol._drain_helper()
  File "/usr/local/lib/python3.11/site-packages/aiohttp/base_protocol.py", line 90, in _drain_helper
    await asyncio.shield(waiter)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 85, in security_filter_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 100, in forwarded_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 236, in auth_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 148, in handle
    result = await handler(request, **request.match_info)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 51, in get
    return await WebSocketHandler(request.app["hass"], request).async_handle()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 422, in async_handle
    await self._writer_task
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 138, in _writer
    await send_str(message)
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_ws.py", line 308, in send_str
    await self._writer.send(data, binary=False, compress=compress)
  File "/usr/local/lib/python3.11/site-packages/aiohttp/http_websocket.py", line 690, in send
    await self._send_frame(message, WSMsgType.TEXT, compress)
  File "/usr/local/lib/python3.11/site-packages/aiohttp/http_websocket.py", line 659, in _send_frame
    await self.protocol._drain_helper()
  File "/usr/local/lib/python3.11/site-packages/aiohttp/base_protocol.py", line 90, in _drain_helper
    await asyncio.shield(waiter)
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 1082, in _write_ready
    n = self._sock.send(self._buffer)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
BrokenPipeError: [Errno 32] Broken pipe
2023-07-26 23:01:32.392 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 332, in data_received
    messages, upgraded, tail = self._request_parser.feed_data(data)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "aiohttp/_http_parser.pyx", line 557, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadStatusLine: 400, message:
  Bad status line "Invalid method encountered:\n\n  b'^\\r'\n    ^"
2023-07-26 23:10:41.250 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 358, in async_handle
    msg = await wsock.receive()
          ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_ws.py", line 440, in receive
    await self.close()
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_ws.py", line 355, in close
    await writer.drain()
  File "/usr/local/lib/python3.11/site-packages/aiohttp/http_writer.py", line 171, in drain
    await self._protocol._drain_helper()
  File "/usr/local/lib/python3.11/site-packages/aiohttp/base_protocol.py", line 90, in _drain_helper
    await asyncio.shield(waiter)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 85, in security_filter_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 100, in forwarded_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 236, in auth_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 148, in handle
    result = await handler(request, **request.match_info)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 51, in get
    return await WebSocketHandler(request.app["hass"], request).async_handle()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 422, in async_handle
    await self._writer_task
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 138, in _writer
    await send_str(message)
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_ws.py", line 308, in send_str
    await self._writer.send(data, binary=False, compress=compress)
  File "/usr/local/lib/python3.11/site-packages/aiohttp/http_websocket.py", line 690, in send
    await self._send_frame(message, WSMsgType.TEXT, compress)
  File "/usr/local/lib/python3.11/site-packages/aiohttp/http_websocket.py", line 659, in _send_frame
    await self.protocol._drain_helper()
  File "/usr/local/lib/python3.11/site-packages/aiohttp/base_protocol.py", line 90, in _drain_helper
    await asyncio.shield(waiter)
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 1082, in _write_ready
    n = self._sock.send(self._buffer)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
BrokenPipeError: [Errno 32] Broken pipe
@Nornode
Copy link
Author

Nornode commented Jul 26, 2023

It seems like the issue is with the server node from where the entities are fetched as when that nodes Homeassistant is restarted the client do get the entities for a few seconds which is enough for them to be visible to the client ha...

So issue from my perspective seems to be with the server node.

a note:
Both instances are running with independent (no supervisor) docker containers on Linux OSes.

@mycanaletto
Copy link

I have the same problem here after the 2023.7.3 update (it was OK with the 2023.7.2 with the fix or the 3.9 update).

image

@Nornode
Copy link
Author

Nornode commented Jul 27, 2023

Can this maybe be caused by the update of aiohttp version?
home-assistant/core#96945

@mycanaletto
Copy link

Can this maybe be caused by the update of aiohttp version? home-assistant/core#96945

I don't know how to act with this

@mycanaletto
Copy link

I have removed and reinstalled both parts but I still have the same symptom.

@Nornode
Copy link
Author

Nornode commented Aug 5, 2023

Version 2023.8.1 did not resolve the matter.

@Nornode Nornode mentioned this issue Aug 5, 2023
Open
@mycanaletto
Copy link

As far as I'm concerned, it started working again without me changing anything in 2023.7.3. I don't get it.

@Nornode
Copy link
Author

Nornode commented Aug 6, 2023

As far as I'm concerned, it started working again without me changing anything in 2023.7.3. I don't get it.

So, did you do anything in between to change any settings?
Also, do you run independent core containers or some other installation method of HA?

@mycanaletto
Copy link

No, juste Hass OS

@Nornode
Copy link
Author

Nornode commented Aug 19, 2023 •
edited
Loading

I still have the same error.
I enabled debug logging on the "client" HA instance:

2023-08-19 21:07:58.171 INFO (MainThread) [custom_components.remote_homeassistant] Connecting to ws://192.168.XX.YY:8123/api/websocket
2023-08-19 21:07:58.208 INFO (MainThread) [custom_components.remote_homeassistant] Connected to home-assistant websocket at ws://192.168.XX.YY:8123/api/websocket
2023-08-19 21:07:58.209 DEBUG (MainThread) [custom_components.remote_homeassistant] received: {'type': 'auth_required', 'ha_version': '2023.8.1'}
2023-08-19 21:07:58.248 DEBUG (MainThread) [custom_components.remote_homeassistant] received: {'type': 'auth_ok', 'ha_version': '2023.8.1'}
2023-08-19 21:07:58.280 DEBUG (MainThread) [custom_components.remote_homeassistant] received: {'id': 141165, 'type': 'result', 'success': True, 'result': None}
2023-08-19 21:07:59.974 ERROR (MainThread) [custom_components.remote_homeassistant] websocket connection had an error

Edit 1:
The client instance achieves connection to XX.YY or 10.14 (Server ha instance) for approximately 30 seconds immediately after restarting it's HA container (independent core Docker container). After which the client node gets stuck in a constant connecting, connected state.

Edit 2:
Same with 2023.8.3 on the server.

Ideas, anyone?

@Nornode
Copy link
Author

Nornode commented Aug 19, 2023 •
edited
Loading

I think I got something...

2023-08-19 22:22:00.692 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140066006615872] Robban from 192.168.11.15 (HomeAssistant/2023.8.3 aiohttp/3.8.5 Python/3.11): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {...}

So, slow connection? - The client node does not run on slow hardware...
Filters not working?

Edit 1:
My current settings:

Screenshot 2023-08-19 at 22 28 53 Screenshot 2023-08-19 at 22 29 24

(I can't click or edit anything in step 2. I tried Chrome & Safari and none responds to actions)

Screenshot 2023-08-19 at 22 29 35 Screenshot 2023-08-19 at 22 30 35

So really all I care about is the status of two persons and their current zone (home, away, work ...)

@Nornode
Copy link
Author

Nornode commented Aug 19, 2023

@Zen3515 - Could my issue be fixed with your PR #227 ?

@Zen3515
Copy link
Contributor

Zen3515 commented Aug 19, 2023

@Zen3515 - Could my issue be fixed with your PR #227 ?

The symptoms are exactly the same. Same error message, and flipping between connected and disconnected.

It may or may not help you, there's no harm in trying it out. But I do think that it would probably help.

But my fork only works up until 2023.7.x, if you'd like to try, make sure you remain using the old version.

If it does indeed help you, you'll have to wait for me to sync the fork, and keep using the fork, since I'm pretty sure that this repo doesn't open for contribution, despite being an open source itself.

@lukas-hetzenecker
Copy link
Member

Sorry it took so long to get this merged, please check whether 3.11 fixes the issue

@Nornode
Copy link
Author

Nornode commented Aug 23, 2023

Thanks, Yes!
Seems like this did the trick!

Thanks for all your help and excellent HA integration!
@Zen3515 && @lukas-hetzenecker
🙇

@Nornode Nornode closed this as completed Aug 23, 2023
@mathmaniac43
Copy link

Hello,

Just wanted to chime in and share that I am experiencing this problem over the past week or so (relatively unexpectedly, after I had already updated both HA installations). I am running HA 2023.10.5 on both machines, and Remote Home-Assistant 3.11 on both as well. I have tried re-downloading the extension via HACS on both.

Thanks,
Nick

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mathmaniac43 @lukas-hetzenecker @Zen3515 @Nornode @mycanaletto