We should consider gzipping `datum_t` objects when we serialize them

[mlucy]

This results in space savings for even smallish documents, for several reasons:

• Englishy language compresses by a factor of 4.
• We use a whole double to store every number, so arrays of small numbers compress by a factor of 10 (counting protobuf overhead).
• Our protobuf format has to use multiple type tags for every datum term -- we have a type for the term, then a type for the datum. Compressing away protobuf overhead alone gives pretty good savings (although this is harder to quantify).

[coffeemug]

That would be awesome! (Unless we saturate the CPU, in which case that would not be awesome).

[srh]

Before that we should add variable length encoding for string lengths (edit: that was done btw) and find some way to avoid writing keys in all the places.

[srh]

We don't use protocol buffers to serialize datum_t objects.

[srh]

Gzipping datum_t's and other forms of wanton compression could also expose some systems to BREACH-style attacks.

[mlucy]

Before that we should add variable length encoding for string lengths and find some way to avoid writing keys in all the places.

The string length thing would be good. What do you mean by "avoid writing keys in all the places"?

We don't use protocol buffers to serialize datum_t objects.

We may want to consider offering gzip compression to/from the clients, too. (Also, our own serialization function isn't that much better; we still serialize a whole double for every number, for example.)

Gzipping datum_t's and other forms of wanton compression could also expose some systems to BREACH-style attacks.

Duly noted; we should document that we do this. (This isn't as high on my list of concerns as our performance problems.)

[coffeemug]

What do you mean by "avoid writing keys in all the places"?

I think he means that if you have a billion documents, each with a field "first_name", we'll end up writing the string "first_name" a billion times.

[underrun]

please don't use gz as it is a CPU drain and slow (relatively speaking).

compression can gain you both space and speed performance if you choose an algorithm that has a higher bandwidth than the device from/to which you are reading/writing (either network or storage), and luckily these algorithms are typically very easy on the CPU as well.

look at lzo if you want something commonly available and well suited to high speed / low cpu. maybe take a peak at google's snappy as well.

[mlucy]

I'd like to do this sooner rather than later, because I think it will help with Benchmark X (both speed and actually fitting the goddamn thing on disk).

@jdoliner also recommended snappy, so I'm going to look into that.

@AtnNn -- snappy doesn't seem to have pre-precise packages, so we'd probably have to include it like with re2. Would that be plausible?

[AtnNn]

If snappy is as portable and as easy to build as re2, it is plausible.

[mlucy]

I've looked into this more.

Snappy is really fast (4-5x zlib on zlib's fastest compression speed), but it has really poor compression on some workloads. It can barely compress English text at all (it takes the size down by about 20%, compared to about 50% for zlib). My hypothesis is that zlib is getting good compression on english text by compressing more common characters (like e) to fewer than 8 bits; snappy doesn't do entropy encoding, so it can't do very well on those inputs (about all it can do is eliminate common subsequences of letters).

Snappy does much better on more redundant data; it takes the size of a random github issue I picked through the JSON API down by about 50%, probably because of all the repeated URLs and stuff.

[mlucy]

I think we should just use snappy for now. Our original test data for benchmark X was highly redundant because it wasn't written expecting to be compressed; we'll have to revise it with a target snappy compression ration of about .5 (GitHub seems like as good a standard as anything for how well JSON documents will compress).

[mlucy]

Also, snappy would perform much better if it were compressing multiple rows at a time, since it could then de-dup field names.

[srh]

Duly noted; we should document that we do this. (This isn't as high on my list of concerns as our performance problems.)

We should document that our database shouldn't be used for web applications?

[jdoliner]

Maybe I don't understand breach style attacks but all the data we'd be compressing is data we've previously been passing around as clear text. How is this going to make things less secure?