Keep the client connected to receive data, even if the connection drops

[evmer]

I have a problem with my websocket client. I would like it to always listen for new data, even if there's a short connection loss.
This is the function I use to receive data:

async def __ws_recv(self):
    while self.ws.open:

        try:
            data = await asyncio.wait_for(self.ws.recv(), timeout=2)

        except asyncio.TimeoutError:
            continue
            log.critical('Websocket read timeout, restarting connection...')
            return

        except websockets.exceptions.ConnectionClosed:
            log.critical('Websocket connection is closed, restarting connection...')
            return

        except Exception as err_msg:
            log.critical(f'Unknown error __ws_recv: {err_msg}')
            os._exit(1)

        self.__process_ws_data(data)

The client works fine, the only problem is that if there is a short loss of connection, the function stops on ws.recv() until the ConnectionClosed exception is raised.

For example if I turn off/on wifi (which causes 1-2 seconds of connection loss), websocket stops and needs to be restarted.

How can I keep the websocket listen for new data, without having to reconnect to the server?

[lgrahl]

WebSocket is TCP-based and if your TCP connection has been lost, then there's nothing you can do other than to reconnect.

[evmer]

WebSocket is TCP-based and if your TCP connection has been lost, then there's nothing you can do other than to reconnect.

I don't think that's my case. The websocket client stays connected, it just doesn't receive data (nor pongs).

I want to elaborate the problem with a comparison. In my project I am also using asyncio Streams which does just what I would like websocket to do: if for a moment the connection is lost (for few seconds), once connected again it receives all the "pending" data. Websocket, on the other hand, hangs on recv() and disconnects.

[lgrahl]

Then we follow different terminology regarding "connection".

IIRC the recv method should ignore cancellation, so generally, I don't see a reason why your code shouldn't work but @aaugustin might have an idea. That being said, I don't understand why you do the wait_for dance in the first place if all you do is continue the loop once the 2s have been elapsed, which is essentially identical to await ws.recv() indefinitely. 🤔

[evmer]

Then we follow different terminology regarding "connection".

IIRC the recv method should ignore cancellation, so generally, I don't see a reason why your code shouldn't work but @aaugustin might have an idea. That being said, I don't understand why you do the wait_for dance in the first place if all you do is continue the loop once the 2s have been elapsed, which is essentially identical to await ws.recv(). 🤔

1. I mean "connection loss" in the sense of a couple of seconds of lack of wifi/internet connection, so this shouldn't cause the socket connection to drop.
2. My application needs to be sure of continuously receiving data. My client sends a ping every 1 second (in another task), so recv should receive the pong response every second (and so it works for most of the runtime). Thus, the only way to detect connection loss is to use the wait_for dance (with a 2 secs timeout tolerance, as written in the script above). If I remove it, the script takes a few tens of seconds before raising ConnectionClosed. It is too long. I'd really like to remove it, but unfortunately as I said a slight loss of connection causes the script to block on recv.

[aaugustin]

Instead of the custom ping/pong implementation you mention in (2) you could use the built-in functionality by configuring a suitable ping_interval and ping_timeout. Then recv() would raise ConnectionClosed after at most ping_interval + ping_timeout + 4 * close_timeout seconds when the connection lags. Unfortunately, it waits for the closing handshake to complete, hence the 4 * close_timeout factor which is less than ideal when the connection is timing out :-/ There's no real problem in setting a small close_timeout, so perhaps this could solve your issue.

I'm confused about why websockets would hang in circumstances where asyncio doesn't, because websockets runs on top of asyncio.

[aaugustin]

websockets interacts with the network only via asyncio; if there's a problem when the network disconnects and reconnects, that's an asyncio (or lower) issue, not a websockets issue.

Now the question is -- does websocket handle cancellation of recv incorrectly, when wait_for timeouts?

I ran the following experiment:

1. Start the server defined in example/show_time.py, which broadcasts messages at random intervals of up to 3 seconds
2. Start this client script, which wait for each message for up to two seconds, so it's expected to timeout on some messages

#!/usr/bin/env python

import asyncio
import websockets

async def test():
    uri = "ws://localhost:5678"
    while True:
        async with websockets.connect(uri) as ws:
            while True:
                try:
                    data = await asyncio.wait_for(ws.recv(), timeout=2)

                except asyncio.TimeoutError:
                    print("- recv timed out after 2 seconds")
                    continue

                except websockets.ConnectionClosed:
                    print("x connection closed, reconnecting")
                    break

                except Exception:
                    print("! unexpected problem")
                    raise

                else:
                    print("+ recv:", data)

asyncio.get_event_loop().run_until_complete(test())

I'm getting the expect results.

+ recv: 2021-05-02T19:13:28.691592Z
+ recv: 2021-05-02T19:13:30.229981Z
+ recv: 2021-05-02T19:13:31.993995Z
- recv timed out after 2 seconds
+ recv: 2021-05-02T19:13:34.766604Z
+ recv: 2021-05-02T19:13:35.205271Z
+ recv: 2021-05-02T19:13:36.623197Z
+ recv: 2021-05-02T19:13:37.554555Z
+ recv: 2021-05-02T19:13:39.425342Z
- recv timed out after 2 seconds
+ recv: 2021-05-02T19:13:42.162765Z
+ recv: 2021-05-02T19:13:42.532393Z
+ recv: 2021-05-02T19:13:43.130699Z
+ recv: 2021-05-02T19:13:44.635270Z
+ recv: 2021-05-02T19:13:46.165632Z
- recv timed out after 2 seconds
+ recv: 2021-05-02T19:13:48.513218Z
+ recv: 2021-05-02T19:13:49.104441Z
+ recv: 2021-05-02T19:13:50.026985Z
+ recv: 2021-05-02T19:13:51.022291Z
+ recv: 2021-05-02T19:13:51.624501Z
+ recv: 2021-05-02T19:13:53.102312Z
+ recv: 2021-05-02T19:13:54.003709Z
+ recv: 2021-05-02T19:13:55.977721Z
+ recv: 2021-05-02T19:13:57.648110Z
- recv timed out after 2 seconds
+ recv: 2021-05-02T19:14:00.339936Z
+ recv: 2021-05-02T19:14:01.354638Z
- recv timed out after 2 seconds
+ recv: 2021-05-02T19:14:03.960202Z
+ recv: 2021-05-02T19:14:04.084362Z
+ recv: 2021-05-02T19:14:04.858305Z
+ recv: 2021-05-02T19:14:06.288688Z
+ recv: 2021-05-02T19:14:06.947242Z
+ recv: 2021-05-02T19:14:07.805516Z
- recv timed out after 2 seconds
+ recv: 2021-05-02T19:14:10.189336Z
+ recv: 2021-05-02T19:14:11.252604Z
- recv timed out after 2 seconds

I don't see what else I could do to reproduce, so I'm going to close the issue.