Decompression uses too much memory

[jarifibrahim]

Badger uses compression to reduce disk space. When the block are decompressed, it takes up too much memory.

badger/table/table.go

Lines 625 to 629 in 5b4c0a6

	case options.Snappy:
	return snappy.Decode(nil, data)
	case options.ZSTD:
	return y.ZSTDDecompress(nil, data)
	}

The crux of the problem is that the decompression call allocates new block every time we want to decompress an existing block. An ideal fix here would be to find a way to reuses the blocks of memory.

Taken from chat: https://dgraph.slack.com/archives/C13LH03RR/p1583223673220200?thread_ts=1583216894.212400&cid=C13LH03RR

[jarifibrahim]

I've tried two different approaches to reduce the memory but both of them aren't working.

Attempt 1 - PR #1247 Reuse the original byte slice (the one read from the table). The problem with this approach is that the sst is mmap and if the slice of bytes read from the table is put into the sync pool, we might end up modifying the original contents of the file. This was seen in #1247 .

Attempt 2 - PR #1308 . Attempt to reuse the byte slice after the block iterator is done processing it. This also has some strange issue which I haven't been able to figure out. In simple terms, the contents of block change if the slices are reused. This is definitely an issue with the implementation in #1308 but I haven't been able to fix it. I've spent significant time trying to debug this but haven't found the issue yet.