when serving static content, sometimes it does not add a compressed chunk

[ktsaou]

Sometimes it gives an error that dygraph-combined.js cannot be loaded.

I tracked it down and found that the served file has a gap of 16383 bytes, at byte 98304. So, there is a condition (aparently timing related) that somehow makes netdata miss a whole compressed stream of data (each compression chunk is 16384 bytes).

It doesn't happen always, but it is there 1 out of 10.

It only happens on static content, so it has to be related to the FILECOPY operation and compression.

[ktsaou]

15-12-30 04:53:07: DEBUG (1008@web_client:web_client_proc): netdata: 21: Processing HTTP GET on url '/lib/dygraph-combined.js'.
15-12-30 04:53:07: DEBUG (1025@web_client:web_client_proc): netdata: 21: Processing command 'lib'.
15-12-30 04:53:07: DEBUG (0259@web_client:mysendfile     ): netdata: 21: Looking for file '/usr/share/netdata/web//lib/dygraph-combined.js'
15-12-30 04:53:07: DEBUG (1181@web_client:web_client_proc): netdata: 21: Generating HTTP header with response 200.
15-12-30 04:53:07: DEBUG (1366@web_client:web_client_proc): netdata: 21: Done preparing the response. Will be sending data file of 123832 bytes to client.
15-12-30 04:53:07: DEBUG (1677@web_client:web_client_main): netdata: 21: Waiting socket async I/O for INPUT
15-12-30 04:53:07: DEBUG (1599@web_client:web_client_rece): netdata: 21: Received 114687 bytes.
15-12-30 04:53:07: DEBUG (1677@web_client:web_client_main): netdata: 21: Waiting socket async I/O for INPUT OUTPUT
15-12-30 04:53:07: DEBUG (1438@web_client:web_client_send): netdata: 21: web_client_send_deflate(): w->response.data->len = 114687, w->response.sent = 0, w->response.zhave = 0, w->response.zsent = 0, w->response.zstream.avail_in = 0, w->response.zstream.avail_out = 0, w->response.zstream.total_in = 0, w->response.zstream.total_out = 0.
15-12-30 04:53:07: DEBUG (1478@web_client:web_client_send): netdata: 21: Compressing 114687 bytes starting from 0.
15-12-30 04:53:07: DEBUG (1498@web_client:web_client_send): netdata: 21: Requesting Z_SYNC_FLUSH.
15-12-30 04:53:07: DEBUG (1514@web_client:web_client_send): netdata: 21: Compression produced 16384 bytes.
15-12-30 04:53:07: DEBUG (1392@web_client:web_client_send): netdata: 21: OPEN CHUNK of 16384 bytes (hex: 4000).
15-12-30 04:53:07: DEBUG (1397@web_client:web_client_send): netdata: 21: Sent chunk header 6 bytes.
15-12-30 04:53:07: DEBUG (1520@web_client:web_client_send): netdata: 21: Sending 16384 bytes to sent.
15-12-30 04:53:07: DEBUG (1526@web_client:web_client_send): netdata: 21: Sent 16390 bytes.
15-12-30 04:53:07: DEBUG (1599@web_client:web_client_rece): netdata: 21: Received 9145 bytes.
15-12-30 04:53:07: DEBUG (1677@web_client:web_client_main): netdata: 21: Waiting socket async I/O for  OUTPUT
15-12-30 04:53:07: DEBUG (1438@web_client:web_client_send): netdata: 21: web_client_send_deflate(): w->response.data->len = 123832, w->response.sent = 114687, w->response.zhave = 16384, w->response.zsent = 16384, w->response.zstream.avail_in = 16383, w->response.zstream.avail_out = 0, w->response.zstream.total_in = 98304, w->response.zstream.total_out = 16384.
15-12-30 04:53:07: DEBUG (1410@web_client:web_client_send): netdata: 21: Sent chunk suffix 2 bytes.
15-12-30 04:53:07: DEBUG (1478@web_client:web_client_send): netdata: 21: Compressing 9145 bytes starting from 114687.
15-12-30 04:53:07: DEBUG (1495@web_client:web_client_send): netdata: 21: Requesting Z_FINISH.
15-12-30 04:53:07: DEBUG (1514@web_client:web_client_send): netdata: 21: Compression produced 16299 bytes.
15-12-30 04:53:07: DEBUG (1392@web_client:web_client_send): netdata: 21: OPEN CHUNK of 16299 bytes (hex: 3fab).
15-12-30 04:53:07: DEBUG (1397@web_client:web_client_send): netdata: 21: Sent chunk header 6 bytes.
15-12-30 04:53:07: DEBUG (1520@web_client:web_client_send): netdata: 21: Sending 16299 bytes to sent.
15-12-30 04:53:07: DEBUG (1526@web_client:web_client_send): netdata: 21: Sent 16307 bytes.
15-12-30 04:53:07: DEBUG (1677@web_client:web_client_main): netdata: 21: Waiting socket async I/O for  OUTPUT
15-12-30 04:53:07: DEBUG (1438@web_client:web_client_send): netdata: 21: web_client_send_deflate(): w->response.data->len = 123832, w->response.sent = 123832, w->response.zhave = 16299, w->response.zsent = 16299, w->response.zstream.avail_in = 0, w->response.zstream.avail_out = 85, w->response.zstream.total_in = 107449, w->response.zstream.total_out = 32683.
15-12-30 04:53:07: DEBUG (1443@web_client:web_client_send): netdata: 21: Out of output data.
15-12-30 04:53:07: DEBUG (1423@web_client:web_client_send): netdata: 21: Sent chunk suffix 7 bytes.
15-12-30 04:53:07: DEBUG (0164@web_client:web_client_rese): netdata: 21: Reseting client.
15-12-30 04:53:07: DEBUG (0167@web_client:web_client_rese): netdata: 21: Closing filecopy input file.
15-12-30 04:53:07: DEBUG (0191@web_client:web_client_rese): netdata: 21: Reseting compression.

1. Initially it reads 114687 bytes.
2. While it compresses it, it receives 9145 bytes more
3. The 1st compression leaves 16383 available for compression
4. The 2nd compression takes into account only the 9145 bytes from the second read

16383 bytes lost