Increased data use with websockets

[jimsiff]

I heard some users have hit data caps post-Cookie Monster updates. It appears that Cookie Monster uses ~5x Azure data out per idle browser remote monitor with Websockets enabled. (tested on Mac / Firefox 33, Windows 7 / IE10). This does not include uncached or cached browser session starts or manual browser refreshes.

I measured a websockets enabled CM site @ ~640 bytes/sec, websockets disabled CM site @ 143 bytes/sec, and a Brownie site @ 140 bytes/sec. Those figures are from Wireshark for at least 10 minutes per session, filtered with "ip.src == $AZURE_IP" to remove browser ACKs and other client > server overhead from the throughput figures.

Those figures translate to ~52 MB/day for Websockets, ~11.7 MB/day XHR polling w/ HTTPS, ~11.5 MB/day XHR polling w/ HTTP.

You can download my packet captures from http://bit.ly/1zFlllO

You can test yourselves using the following sites:

http://http-brownie.azurewebsites.net - Brownie 0.4.3
https://https-cookie.azurewebsites.net - CM 0.5.0 w/ Websockets
https://https-cookie-nows.azurewebsites.net - CM 0.5.0 w/o Websockets

All sites use the same Mongo store, my dev/test rig with live data. Data out for these sites is purely monitoring, no REST uploads.

[shanselman]

Odd. I will take a look.

[jimsiff]

FYI, the captures start after the browser sessions had stabilized and all objects had loaded so I would see a baseline throughput.

[shanselman]

Looks like because the client and server are very inefficient (they aren't sending deltas, but instead sending 2.5 days of data every minute!) then at the very least I think Free Azure Websites users should not use WebSockets. I propose we update the docs.

[scottleibrand]

That has always been true. Why is WebSockets so much more overhead than
sending the same data over the old method?

-Scott

On Sun, Nov 9, 2014 at 9:14 PM, Scott Hanselman notifications@github.com
wrote:

Looks like because the client and server are very inefficient (they aren't
sending deltas, but instead sending 2.5 days of data every minute!) then at
the very least I think Free Azure Websites users should not use WebSockets.
I propose we update the docs.

—
Reply to this email directly or view it on GitHub
#235 (comment)
.

[jimsiff]

I agree. I will update the docs, but we also need to see how we can shorten the negotiation timeout to something more acceptable.

[jimsiff]

I took a close look at the packet captures for CM with websockets compared with both Brownie and CM without websockets. The data use has the same pattern of keepalive chatter and site refresh data. The websocket chatter is more efficient by ~50% than the xhr chatter... less packets, less data.

The problem is that the site refresh data is literally scaled 5x. Is there something in the code that is sending the refresh data through a loop when using websockets? I find it hard to believe that the web socket protocol would add an order of magnitude in data overhead for the typical 5k/10k site refreshes.

Here's my data to back it up. http://bit.ly/1zFSYUB.

@bewest, @scottleibrand, @shanselman @jasoncalabrese, @rnpenguin, @ktind, what do you think?

[jimsiff]

@scottleibrand's PR for the mono sound file improves initial load times and reduces cached data, but the bulk of the data is still related to redrawing the website every minute. How big of a job is it to switch to sending deltas? Does a delta approach introduce new variability to the enduser experience due to browser preferences, default browser cache behavior, etc?

[jasoncalabrese]

Maybe we should turn websockets off, until we find a solution. A 5x data increase is going to push people over their limits.

[ELUTE]

That was our feeling too. Web sockets should remain off but we will have to explain to people why there is a 10-15 second load delay on their site. Wondering if we should wait on encouraging them to upgrade to CM.

On Nov 10, 2014, at 8:43 AM, Jason Calabrese notifications@github.com wrote:

Maybe we should turn websockets off, until we find a solution. A 5x data increase is going to push people over their limits.

—
Reply to this email directly or view it on GitHub #235 (comment).

[jasoncalabrese]

The websocket change is probably the biggest change on the web monitor side and it was added late in the release. My vote is to pull it out for now.

[jasoncalabrese]

I created #239 to disable websockets for now

[jimsiff]

0.5.1 has fixed the excessive data out usage as well as the websocket negotiation timeout when disabled in Azure.

Any idea what causes the increased payload size for websocket site refreshes?

[bewest]

Switching to 'deltas' is on roadmap, but is part of a big change.
FWIW, nothing about polling behavior has been changed in a long time. It could be changed from 1 minute to 4 minutes pretty easily, though.

[jimsiff]

After the connection is upgraded to websockets, the 2.5 days of BG data is sent clear-text rather than compressed using gzip/deflate. See this example: http://bit.ly/1xnDNMg

Based on the following link, websocket data compression is out there, but I'm not sure how universally it's supported beyond the newest browsers. If that's a question mark, maybe we shelve websockets until MQTT is ready.

https://www.igvita.com/2013/11/27/configuring-and-optimizing-websocket-compression/

[scottleibrand]

It looks like that's still an Internet-Draft, and likely won't be supported
in older browsers right away even once it's standardized. So I agree that
we probably don't want to have people on Azure free instances do websockets
for now.

On Mon, Nov 10, 2014 at 5:36 PM, Jim Sifferle notifications@github.com
wrote:

After the connection is upgraded to websockets, the 2.5 days of BG data is
sent clear-text rather than compressed using gzip/deflate. See this
example: http://bit.ly/1xnDNMg

Based on the following link, I'm not sure whether all current browsers
support websocket data compression.

https://www.igvita.com/2013/11/27/configuring-and-optimizing-websocket-compression/

If that's a question mark, maybe we shelve websockets until MQTT is ready.

—
Reply to this email directly or view it on GitHub
#235 (comment)
.

[bewest]

Interesting side effect, nice find @jimsiff !

[jasoncalabrese]

Nice research @jimsiff. One we can stopping sending all the data each update maybe we could send the initial data via xhr and the rest via websockets

[jasoncalabrese]

fixed way back with 0.5.1