compression between carbon-c-relays?

[azhiltsov]

Hi.
I am wondering could you implement a compression between carbon-c-relays?
It is a highly compressible protocol and could give significant saving on the bandwidth.

[grobian]

Just wondering, do you actually see high bandwidth usage? Compression complicates things a bit, an interesting test would be to run an xz tunnel (or alike) and let the relay use that tunnel.

[azhiltsov]

Yes we do. This is a traffic between relays and stores, as both of them are running carbon-c-relay it looks like an obvious solution to have a compression in between and keep a graphite compatible protocol for input and output flows

[dgryski]

As a comparison, enabling compression between carbonserver<->carbonzipper gave:

[grobian]

is that context aware compression, or generic "zipping" compression?

[dgryski]

This was just gzipping the protobufs.

[grobian]

that looks promising then

[xneo64]

+1 this could save many $$ in ISP VPN bandwidth costs when transmitting data across data centres

[tehlers320]

If your multi-regional and have the budget you could have (carbon-c-relay -> telegraf -> kafka (with compression) -> telegraf elsewhere -> carbon-c-relay -> carbon-cache)

It's a bit of pipe but it does give you a big buffer for metrics that you could hang other things off of such as alarming/other graph systems. I've never seen a pipe like this in action so i'd wonder about the cpu use of telegraf/kafka.

[grobian]

My problem (design-wise) with this thing, is that I need anchorpoints both send and receive to specify the traffic is compressed or not. Since the protocol isn't really having any wire version or something, it needs to use another port, but for that it needs configuration too.
Perhaps a separate program /is/ the easiest solution, like stunnel or something, or a different invocation of the c-relay so it can use metric-level compression or something. It could be a memory drain, but with some communication (huffman-trees?) between relays a high compression level could be reached. Or maybe that's too far-fetched, and is regular gzip compression already strong enough because producers typically generate trees of metrics that all share prefixes. Meh.

[azhiltsov]

Separate invocation on the same port with enabled gzip compression should be sufficient for receiving part
For output streams this could be declared in config file on per-recipient basis.

[grobian]

This is on my todo list, it will come with a way to configure listeners instead of the current "single port is everything" approach.

[grobian]

in master there is support for sending and receiving gzipped and bzip2-ed streams.