Websocket connection issue

[Wozacosta]

Version: 5.2.2
Python: 3.6.8
OS: linux
Blockchain : Kaleido

What was wrong?

When interacting with contracts via a websocket provider, after a couple of successful requests, we get this error :

 Internal Server Error: 
 Traceback (most recent call last):
   File "/app/.local/lib/python3.7/site-packages/websockets/protocol.py", line 674, in transfer_data
     message = yield from self.read_message()
   File "/app/.local/lib/python3.7/site-packages/websockets/protocol.py", line 742, in read_message
     frame = yield from self.read_data_frame(max_size=self.max_size)
   File "/app/.local/lib/python3.7/site-packages/websockets/protocol.py", line 815, in read_data_frame
     frame = yield from self.read_frame(max_size)
   File "/app/.local/lib/python3.7/site-packages/websockets/protocol.py", line 884, in read_frame
     extensions=self.extensions,
   File "/app/.local/lib/python3.7/site-packages/websockets/framing.py", line 99, in read
     data = yield from reader(2)
   File "/usr/lib/python3.7/asyncio/streams.py", line 677, in readexactly
     raise IncompleteReadError(incomplete, n)
 asyncio.streams.IncompleteReadError: 0 bytes read on a total of 2 expected bytes
 
 The above exception was the direct cause of the following exception:
 
 Traceback (most recent call last):
   File "/app/.local/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
     response = get_response(request)
   File "/app/.local/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
     response = self.process_exception_by_middleware(e, request)
   File "/app/.local/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
     response = wrapped_callback(request, *callback_args, **callback_kwargs)
   File "/app/.local/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
     return view_func(*args, **kwargs)
   File "/app/.local/lib/python3.7/site-packages/rest_framework/viewsets.py", line 116, in view
     return self.dispatch(request, *args, **kwargs)
   File "/app/.local/lib/python3.7/site-packages/rest_framework/views.py", line 495, in dispatch
     response = self.handle_exception(exc)
   File "/app/.local/lib/python3.7/site-packages/rest_framework/views.py", line 455, in handle_exception
     self.raise_uncaught_exception(exc)
   File "/app/.local/lib/python3.7/site-packages/rest_framework/views.py", line 492, in dispatch
     response = handler(request, *args, **kwargs)
   File "/app/backend/eqs_transaction/views.py", line 143, in call
     if not w3.isConnected():
   File "/app/.local/lib/python3.7/site-packages/web3/main.py", line 227, in isConnected
     return self.provider.isConnected()
   File "/app/.local/lib/python3.7/site-packages/web3/providers/auto.py", line 76, in isConnected
     return provider is not None and provider.isConnected()
   File "/app/.local/lib/python3.7/site-packages/web3/providers/base.py", line 77, in isConnected
     response = self.make_request('web3_clientVersion', [])
   File "/app/.local/lib/python3.7/site-packages/web3/providers/websocket.py", line 119, in make_request
     return future.result()
   File "/usr/lib/python3.7/concurrent/futures/_base.py", line 432, in result
     return self.__get_result()
   File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
     raise self._exception
   File "/app/.local/lib/python3.7/site-packages/web3/providers/websocket.py", line 102, in coro_make_request
     timeout=self.websocket_timeout
   File "/usr/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
     return fut.result()
   File "/app/.local/lib/python3.7/site-packages/websockets/protocol.py", line 462, in send
     yield from self.ensure_open()
   File "/app/.local/lib/python3.7/site-packages/websockets/protocol.py", line 646, in ensure_open
     ) from self.transfer_data_exc
 websockets.exceptions.ConnectionClosed: WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), no reason

The connection drops during the interaction as web3.isConnected() returns true before that interaction.

[jpic]

So, do you think reconnection can be done at this lib's level ? ie. try to reconnect, and raise an error only if reconnecting failed twice in a row maybe ?

[Wozacosta]

Yeah that would work, maybe even with a parameter to control the number of retries.

[jpic]

For those who find this issue looking for a solution: we basically decorated our functions using web3py with tenacity. Not the best solution but unblocks the situation meanwhile core devs suggest an implementation that we can contribute.

[kclowes]

I ran into this yesterday when I was sent a malformed request via websocket, sometimes the websocket will just close. @Wozacosta or @jpic, are you seeing the exact same request go through after you reconnect? Or is the request somehow changing after reconnection?

[voith]

@kclowes Can you give an example of a malformed request?

[kclowes]

For example, if I attach to a local geth node with wscat and send a request like: {"jsonrpc":"2.0","method":"eth_chainId", params: []} (notice params doesn't have quotes around it), I get immediately disconnected with a 1006 code. This is a pretty contrived example, and it does sound to me like implementing reconnection is probably the right way to go for this issue, but I wanted to make sure we were treating the cause, not just the symptom.

[voith]

@kclowes What you're describing seems like a different issue from the one described in the title of this issue. My guess is that you got this error because you provided a malformed JSON to geth and geth failed to deserialize it giving 1006: Internal Error(Although, it would be nice if geth could give a better error message ). However, In @jpic's case, it seems like the error is random and the request is successful after several retries or maybe by creating a new connection.

I wanted to make sure we were treating the cause, not just the symptom.

Yeah, the case you've described is not possible using web3py and incase web3py is failing to sanitize user input that is causing such errors, then those need to be handled separately. Till the actual cause for the error is known, it's hard to say what would be the right fix.

@kclowes Btw, there's already some code that creates a new connection when the previous connection raises an error for some reason. Might be worth checking if that actually works.

web3.py/web3/providers/websocket.py

Lines 55 to 60 in 0aa00c8

	if exc_val is not None:
	try:
	await self.ws.close()
	except Exception:
	pass
	self.ws = None

[kclowes]

Yeah, the case you've described is not possible using web3py and incase web3py is failing to sanitize user input that is causing such errors, then those need to be handled separately.

I know. It was a bad example. I can't remember the exact request I made where I saw the error. I was also using wscat rather than web3, which of course makes a difference. I was just surprised that a connection would just drop when it encountered certain errors, but not others. What I was getting at was trying to narrow down whether this was a problem with a request that led to the provider closing the websocket or a problem with the provider.

In checking out the function you linked to @voith , I was able to reproduce something close to the error above by connecting to geth, issuing a request and then disconnecting from geth and issuing another request. Assuming that's representative of the original issue raised, I don't think this is a web3 problem per se, but I can see where it would be useful to retry a websocket request that comes back with a ConnectionClosedError, so I'm not opposed to adding that feature.

I'd also be curious to see your decorator functions @jpic, if possible, to see how you are getting around these errors.

[fselmo]

stale / WebsocketProviderV2 should help here