Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZSTD and GZIP/DEFLATE streaming support #16268

Merged
merged 46 commits into from Oct 27, 2023
Merged

Conversation

ktsaou
Copy link
Member

@ktsaou ktsaou commented Oct 25, 2023

This PR adds:

  • ZSTD compression support to streaming
  • GZIP/DEFLATE compression support to streaming
  • user configurable compression algorithms on parents
  • compression level per algorithm on children
  • add more error messages to detect disconnect issues
  • unittest for all compression algorithms

@netdata-community-bot
Copy link

This pull request has been mentioned on Netdata Community Forums. There might be relevant details there:

https://community.netdata.cloud/t/update-frequency-of-cascaded-parents/4796/2

@ktsaou ktsaou changed the title ZSTD streaming support ZSTD and GZIP/DEFLATE streaming support Oct 26, 2023
@vkalintiris
Copy link
Contributor

vkalintiris commented Oct 26, 2023

@ktsaou check vkalintiris@15743be
From the logs of a child:

~/opt/child/netdata$ grep tainted var/log/netdata/error.log | grep GVD | grep '[01]$' | awk '{ print $NF }' | sort | uniq -c
157533 0
143786 1

Aka, 52% of the charts do not have changed dimensions between consecutive collection intervals. The buffer of the charts can be flushed and its contents can be replaced with a new command that will let the parent know that the chart's dimensions did not change state.

I believe we can reduce traffic ~50% and improve compression CPU consumption by an equal or even larger amount.

@stelfrag
Copy link
Collaborator

stelfrag commented Oct 26, 2023

New parent (zstd-streaming) with old version child connected gets frequent disconnections due to

2023-10-26 23:15:12: netdata ERROR : RCVR[destiny] : RRDPUSH_DECOMPRESS: decompressed data is 16384 bytes, which is bigger than the max msg size 16255
2023-10-26 23:15:12: netdata INFO  : RCVR[destiny] : STREAM 'destiny' [receive from [192.168.1.138]:49754]: disconnected (completed 2 updates). STATUS: DISCONNECTED (DISCONNECTED UNKNOWN SOCKET READ ERROR)

@ktsaou
Copy link
Member Author

ktsaou commented Oct 27, 2023

New parent (zstd-streaming) with old version child connected gets frequent disconnections due to

Fixed.

@ilyam8 ilyam8 mentioned this pull request Oct 27, 2023
4 tasks
@ktsaou ktsaou merged commit cd584e0 into netdata:master Oct 27, 2023
148 of 149 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants