Websocket dies if received packet size from controller is over 1MB

[rahnarsson]

Error from the node:

2017-06-09 08:05:30 DEBUG juju-log Trying to connect to Juju API endpoint 10.17.20.29:17070
2017-06-09 08:05:40 INFO install ERROR:juju.model:Error in watcher
2017-06-09 08:05:40 INFO install Traceback (most recent call last):
2017-06-09 08:05:40 INFO install   File "/usr/local/lib/python3.5/dist-packages/juju/model.py", line 630, in _start_watch
2017-06-09 08:05:40 INFO install     self.loop)
2017-06-09 08:05:40 INFO install   File "/usr/local/lib/python3.5/dist-packages/juju/utils.py", line 94, in run_with_interrupt
2017-06-09 08:05:40 INFO install     result = [f.result() for f in done if f is not event_task]
2017-06-09 08:05:40 INFO install   File "/usr/local/lib/python3.5/dist-packages/juju/utils.py", line 94, in <listcomp>
2017-06-09 08:05:40 INFO install     result = [f.result() for f in done if f is not event_task]
2017-06-09 08:05:40 INFO install   File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
2017-06-09 08:05:40 INFO install     raise self._exception
2017-06-09 08:05:40 INFO install   File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
2017-06-09 08:05:40 INFO install     result = coro.send(None)
2017-06-09 08:05:40 INFO install   File "/usr/local/lib/python3.5/dist-packages/juju/client/facade.py", line 392, in wrapper
2017-06-09 08:05:40 INFO install     reply = await f(*args, **kwargs)
2017-06-09 08:05:40 INFO install   File "/usr/local/lib/python3.5/dist-packages/juju/client/_client1.py", line 59, in Next
2017-06-09 08:05:40 INFO install     reply = await self.rpc(msg)
2017-06-09 08:05:40 INFO install   File "/usr/local/lib/python3.5/dist-packages/juju/client/overrides.py", line 99, in rpc
2017-06-09 08:05:40 INFO install     result =  await self.connection.rpc(msg, encoder=TypeEncoder)
2017-06-09 08:05:40 INFO install   File "/usr/local/lib/python3.5/dist-packages/juju/client/connection.py", line 238, in rpc
2017-06-09 08:05:40 INFO install     result = await self.recv(msg['request-id'])
2017-06-09 08:05:40 INFO install   File "/usr/local/lib/python3.5/dist-packages/juju/client/connection.py", line 175, in recv
2017-06-09 08:05:40 INFO install     return await self.messages.get(request_id)
2017-06-09 08:05:40 INFO install   File "/usr/local/lib/python3.5/dist-packages/juju/utils.py", line 63, in get
2017-06-09 08:05:40 INFO install     raise value
2017-06-09 08:05:40 INFO install   File "/usr/local/lib/python3.5/dist-packages/juju/client/connection.py", line 183, in receiver
2017-06-09 08:05:40 INFO install     loop=self.loop)
2017-06-09 08:05:40 INFO install   File "/usr/local/lib/python3.5/dist-packages/juju/utils.py", line 94, in run_with_interrupt
2017-06-09 08:05:40 INFO install     result = [f.result() for f in done if f is not event_task]
2017-06-09 08:05:40 INFO install   File "/usr/local/lib/python3.5/dist-packages/juju/utils.py", line 94, in <listcomp>
2017-06-09 08:05:40 INFO install     result = [f.result() for f in done if f is not event_task]
2017-06-09 08:05:40 INFO install   File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
2017-06-09 08:05:40 INFO install     raise self._exception
2017-06-09 08:05:40 INFO install   File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
2017-06-09 08:05:40 INFO install     result = coro.send(None)
2017-06-09 08:05:40 INFO install   File "/usr/local/lib/python3.5/dist-packages/websockets/protocol.py", line 298, in recv
2017-06-09 08:05:40 INFO install     raise ConnectionClosed(self.close_code, self.close_reason)
2017-06-09 08:05:40 INFO install websockets.exceptions.ConnectionClosed: WebSocket connection is closed: code = 1009, no reason.
2017-06-09 08:05:40 INFO install ERROR:asyncio:Task exception was never retrieved
2017-06-09 08:05:40 INFO install future: <Task finished coro=<Model._watch.<locals>._start_watch() done, defined at /usr/local/lib/python3.5/dist-packages/juju/model.py:622> exception=ConnectionClosed('WebSocket connection is closed: code = 1009, no reason.',)>
2017-06-09 08:05:40 INFO install Traceback (most recent call last):
2017-06-09 08:05:40 INFO install   File "/usr/lib/python3.5/asyncio/tasks.p

Websocket connection error 1009 shows:

The max_size parameter enforces the maximum size for incoming messages in bytes. The default value is 1MB. None disables the limit. If a message larger than the maximum size is received, recv() will raise ConnectionClosed and the connection will be closed with status code 1009.

Changing juju/client/connection.py:
self.ws = await websockets.connect(url, **kw)
to
self.ws = await websockets.connect(url, max_size=None, **kw)

Fixes the connection issues.

[johnsca]

I'm a bit reluctant to completely remove any limit on the frame size, but I guess the initial AllWatcher frame contains the entire model description, so the size will be dependent on the model being connected to.

[rahnarsson]

Yes. Complete removal might be too harsh. But I think that size should definitely be increased or at least make possible to change when creating connection.

[rahnarsson]

I can see this causing problems when the model has been running for a while and there is lot of changes during time. Let's say the model has been running for a year or so, and some charm provides actions that users run on a regular basis.

[johnsca]

I'm asking on the juju-dev mailing list to find out if there is a maximum frame size on the controller side that we can match. Even if there's not currently, I think the initial AllWatcher frame on connect is likely to be the largest frame, and it will only depend on the size of the current state of the model (not changes over time), so we should be able to come up with a reasonable upper bound. But, in the end, the controller will need to ensure that large enough models have the initial AllWatcher frame chunked.