Why are the websockets closing with status 1006?

[abingham]

The browser is showing that our websockets to runner_repl are closing with status 1006, and I can't figure out why. This happens after 10 or 20 seconds if inactivity, it seems, though I'm not sure if there's a real relationship.

This happens across browsers: chrome, firefox, and safari at least.

[abingham]

I've gone so far as to replace the aiohttp servers with sanic, but this doesn't seem to fix the problem. My intuition was that perhaps the aiohttp websocket implementation was broken. Alas.

We'll need to add logic to the web client and runner_repl to deal with this, I suppose.

[abingham]

This is still happening, and I still don't know why, but our implementation seems to be robust against it now. I'm going to leave this open for now, but I'm not going to put any time into right now.

[abingham]

It could be this: https://stackoverflow.com/questions/28828332/gorilla-websocket-disconnects-after-a-minute

I ran some experiments to see if I could replicate these disconnects, and I wasn't able to. The experiment did not, however, include nginx, so perhaps we can configure nginx to stop killing the websocket proxies.

[abingham]

Some quick timing certainly seems to support the hypothesis that nginx is timing websocket proxies out after 1 minute.

[abingham]

I've verified to my satisfaction that nginx timeouts are the problem. In the default configuration, the proxy_read_timeout and proxy_send_timeout values are 60 seconds, and we see the 1006 disconnects after 60 seconds of inactivity. If I set them both to 120s, then we see the timeout after 120 seconds.

I'm going to consider the mystery solved, though we still need to figure out how to handle timeouts gracefully.

For now I'm going to set the timeouts to 10 hours.