Stream compression - Deactivate compression at runtime in case of a compressor buffer overflow #12037

odynik · 2022-01-25T15:07:08Z

Summary

This PR catches a stream compression buffer overflow and escapes from a stream corruption by deactivating stream compression at runtime. If the sender thread experiences a compressor buffer overflow will deactivate stream compression and re-establish a fresh link with stream version protocol 4.

Test Plan

To test this PR you are going to need,

At least a child <-> parent netdata agent set-up but I would recommend a grand-parent<->parent<->child set-up.
A collector/chart that can provide you with a chart/dimension definition or a data transmission greater than 16kB.

Steps

1 Enable stream compression in the stream.conf file for all the agents.

[stream]
enabled = yes
enable compression = yes

OR/AND

[API_KEY]
enabled = yes
enable compression = yes

In the child Netdata agent create a go example collector with 2 charts and 300 dim/chart,
2.1 cd /etc/netdata
2.2 sudo ./edit-config go.d.conf
Enable example go-plugin

#  dockerhub: yes
#  elasticsearch: yes
  example: yes
#  filecheck: yes
#  fluentd: yes

Create the chart with sudo ./edit-config go.d/example.conf

jobs:
  - name: stress
    charts:
      num: 2
      dimensions: 300

Restart all the agents
Look in the child error.log for the message sequence,

netdata INFO  : STREAM_SENDER[child01] : STREAM child01 [send to my.parent.IP]: connecting...
netdata INFO  : STREAM_SENDER[child01] : STREAM child01 [send to my.parent.IP]: initializing communication...
netdata INFO  : STREAM_SENDER[child01] : STREAM child01 [send to my.parent.IP]: waiting response from remote netdata...
netdata INFO  : STREAM_SENDER[child01] : STREAM_COMPRESSION: Compressor Reset
netdata INFO  : STREAM_SENDER[child01] : STREAM child01 [send to my.parent.IP]: established communication with a parent using protocol version 5 - ready to send metrics...
...
netdata ERROR : PLUGINSD[go.d] : STREAM_COMPRESSION: Compression Failed - Message size 27847 above compression buffer limit: 16384 (errno 9, Bad file descriptor)
netdata ERROR : PLUGINSD[go.d] : STREAM_COMPRESSION: Deactivating compression to avoid stream corruption
netdata ERROR : PLUGINSD[go.d] : STREAM_COMPRESSION child01 [send to my.parent.IP]: Restarting connection without compression
...
netdata INFO  : STREAM_SENDER[child01] : STREAM child01 [send to my.parent.IP]: connecting...
netdata INFO  : STREAM_SENDER[child01] : STREAM child01 [send to my.parent.IP]: initializing communication...
netdata INFO  : STREAM_SENDER[child01] : STREAM child01 [send to my.parent.IP]: waiting response from remote netdata...
netdata INFO  : STREAM_SENDER[child01] : Stream is uncompressed! One of the agents (my.parent.IP <-> child01) does not support compression OR compression is disabled.
netdata INFO  : STREAM_SENDER[child01] : STREAM child01 [send to my.parent.IP]: established communication with a parent using protocol version 4 - ready to send metrics...
netdata INFO  : WEB_SERVER[static4] : STREAM child01 [send]: sending metrics...

The stream between child <-> parent <-> gparent stream should not be corrupted which means,
8.1 See streaming data in parent and grandparent agent dashboards.
8.2 In parent agent, you should see a healthy sender thread shutdown sequence,

 netdata ERROR : STREAM_RECEIVER[child01,[my.child.ip]:43478] : STREAM child01 [receive from [my.child.ip]:43478]: disconnected (completed 665999 updates).
 netdata INFO  : STREAM_RECEIVER[child01,[my.child.ip]:43478] : Queuing status update for node=1423ff00-e22c-4f1f-ad51-e861b8cd7ce3, live=0, hops=1
 netdata INFO  : STREAM_RECEIVER[child01,[my.child.ip]:43478] : STREAM child01 [send]: signaling sending thread to stop...
 netdata INFO  : STREAM_RECEIVER[child01,[my.child.ip]:43478] : STREAM child01 [send]: waiting for the sending thread to stop...
 netdata INFO  : STREAM_SENDER[child01] : STREAM child01 [send]: sending thread cleans up...
 netdata INFO  : STREAM_SENDER[child01] : STREAM child01 [send]: sending thread now exits.
 netdata INFO  : STREAM_SENDER[child01] : thread with task id 3311408 finished
 netdata INFO  : STREAM_RECEIVER[child01,[my.child.ip]:43478] : STREAM child01 [send]: sending thread has exited.
 netdata INFO  : STREAM_RECEIVER[child01,[my.child.ip]:43478] : STREAM child01 [receive from [my.child.ip]:43478]: receive thread ended (task id 3311323)
 netdata INFO  : STREAM_RECEIVER[child01,[my.child.ip]:43478] : thread with task id 3311323 finished

8.3 Please report any sequence of messages containing

EOF found in spawn pipe
Shutting down spawn server event loop

Additional Information

This PR is a temporary work around for this issue #12020. I am not linking the issue here since it is not the permanent solution.

Check the netdata -W buildinfo for Stream Compression: YES
Check the runtime compression flag stream-compression: enabled/disabled/N/A at web api /api/v1/info

streaming/README.md

vlvkobal

Tested it with child <-> parent <-> gparent.

kickoke

Documentation: LGTM

thiagoftsm

I tested proposed scenario and I did not observe problems with SPAWN server!

thiagoftsm · 2022-01-27T11:31:10Z

@odynik we have conflicts with streaming/receiver.c, please, rebase your PR.

odynik · 2022-01-27T14:28:05Z

@odynik we have conflicts with streaming/receiver.c, please, rebase your PR.

Thanks @thiagoftsm. For now, we decided to keep this PR on hold.
I will rebase it if/when we are about to merge it.

…erflows

… loop

…erflows

… loop

odynik · 2022-03-11T10:30:18Z

PR rebased on latest master

streaming/README.md

kickoke

Docs: LGTM

…o stream_compression_buff_overflow_fix

thiagoftsm

I tested all the possible combinations for the proposed scenario, and I did not observe any crash in spawn server. I also observed a downgrade when buffer overflow happens. LGTM!

stelfrag · 2022-03-22T17:56:33Z

streaming/sender.c

+    error("STREAM_COMPRESSION: Deactivating compression to avoid stream corruption");
+    default_compression_enabled = 0;
+    s->rrdpush_compression = 0;
+    s->version = STREAM_VERSION_CLABELS;


So currently CLABELS is 4 and compression is 5. What will happen with version 6 ? It will support compression but when it fails it will fallback to 4 (losing the functionally added by version 6).

I would prefer to merge this PR in the master and respect the stream versions (4 - clabels, 5 - compression, 6 - gap filling) in the gap filling PR.

The stream version checks need to be gathered in a function because if we keep adding features in streaming the if/elif statements are going to increase the code complexity.

odynik requested review from DShreve2, kickoke, thiagoftsm, vkalintiris and vlvkobal as code owners January 25, 2022 15:07

odynik added the area/streaming label Jan 25, 2022

github-actions bot added area/docs area/web labels Jan 25, 2022

odynik self-assigned this Jan 25, 2022

odynik requested a review from stelfrag January 25, 2022 15:11

MrZammler reviewed Jan 25, 2022

View reviewed changes

streaming/README.md Outdated Show resolved Hide resolved

thiagoftsm requested a review from MrZammler January 25, 2022 20:55

erdem2000 previously approved these changes Jan 26, 2022

View reviewed changes

vlvkobal previously approved these changes Jan 26, 2022

View reviewed changes

kickoke previously approved these changes Jan 26, 2022

View reviewed changes

thiagoftsm previously approved these changes Jan 27, 2022

View reviewed changes

odynik added 12 commits March 11, 2022 11:04

[Stream compression] Downgrade stream version if compressor buffer ov…

1842f16

…erflows

[Stream compression] More user friendly compression messages

f4c70e7

[Stream compression] - clabel actions were omitted

f319490

[Stream compression] - Fix mutex starvation - Remove this ugly while…

c30feeb

… loop

[Stream compression] - Remove db msgs

a7be189

[Stream compression] Update Readme.md with reconnection msgs

4705561

[Stream compression] Downgrade stream version if compressor buffer ov…

ebc3d71

…erflows

[Stream compression] More user friendly compression messages

b179672

[Stream compression] - clabel actions were omitted

028e8c3

[Stream compression] - Fix mutex starvation - Remove this ugly while…

3a5871e

… loop

[Stream compression] - Remove db msgs

21d133f

[Stream compression] Update Readme.md with reconnection msgs

7689c9b

[Stream compression] fix typo

673ef77

odynik dismissed stale reviews from thiagoftsm, kickoke, vlvkobal, and erdem2000 via 4705561 March 11, 2022 10:28

odynik force-pushed the stream_compression_buff_overflow_fix branch from 0851eec to 4705561 Compare March 11, 2022 10:28

thiagoftsm requested review from thiagoftsm, vlvkobal, kickoke and erdem2000 March 11, 2022 13:47

kickoke reviewed Mar 11, 2022

View reviewed changes

streaming/README.md Outdated Show resolved Hide resolved

Update streaming/README.md

659761d

kickoke previously approved these changes Mar 11, 2022

View reviewed changes

odynik added 2 commits March 16, 2022 15:54

[Stream compression] enable compression by default

605e3f8

Merge commit 'refs/pull/12037/head' of github.com:netdata/netdata int…

bf8f0b6

…o stream_compression_buff_overflow_fix

odynik dismissed kickoke’s stale review via bf8f0b6 March 16, 2022 14:09

odynik requested a review from kickoke March 16, 2022 14:20

kickoke approved these changes Mar 16, 2022

View reviewed changes

thiagoftsm approved these changes Mar 17, 2022

View reviewed changes

vlvkobal approved these changes Mar 18, 2022

View reviewed changes

stelfrag reviewed Mar 22, 2022

View reviewed changes

odynik merged commit 57d6c17 into netdata:master Mar 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stream compression - Deactivate compression at runtime in case of a compressor buffer overflow #12037

Stream compression - Deactivate compression at runtime in case of a compressor buffer overflow #12037

odynik commented Jan 25, 2022

vlvkobal left a comment

kickoke left a comment

thiagoftsm left a comment

thiagoftsm commented Jan 27, 2022

odynik commented Jan 27, 2022

odynik commented Mar 11, 2022

kickoke left a comment

thiagoftsm left a comment

stelfrag Mar 22, 2022

odynik Mar 23, 2022 •

edited

Stream compression - Deactivate compression at runtime in case of a compressor buffer overflow #12037

Stream compression - Deactivate compression at runtime in case of a compressor buffer overflow #12037

Conversation

odynik commented Jan 25, 2022

Summary

Test Plan

Steps

Additional Information

vlvkobal left a comment

Choose a reason for hiding this comment

kickoke left a comment

Choose a reason for hiding this comment

thiagoftsm left a comment

Choose a reason for hiding this comment

thiagoftsm commented Jan 27, 2022

odynik commented Jan 27, 2022

odynik commented Mar 11, 2022

kickoke left a comment

Choose a reason for hiding this comment

thiagoftsm left a comment

Choose a reason for hiding this comment

stelfrag Mar 22, 2022

Choose a reason for hiding this comment

odynik Mar 23, 2022 • edited

Choose a reason for hiding this comment

odynik Mar 23, 2022 •

edited