Compaction multi-threading

[mdisibio]

What this PR does:
Compaction is single-threaded and solidly limited to using a single core. This PR introduces some basic multi-threading by moving iteration to a separate goroutine, which improves performance.

Benchmark was added and shows ~12% improvement:

pkg: github.com/grafana/tempo/tempodb
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkCompaction-12     	    3594	    329848 ns/op
BenchmarkCompaction2-12    	    4042	    290434 ns/op

Testing in a real env is better:
objects and bytes written up ~20%

Some notes:

• New tunable: the channel buffer size. Unfortunately it is in traces (not bytes). Defaulted to 1000 based on testing.
• ID and data slices must be copied to escape the iterators which increases GC and memory pressure slightly. Interested in trying pooling, but the current Iterator interface doesn't expose the ability return slices from Next(). Will explore this in a later PR unless there is interest in trying it now.

Which issue(s) this PR fixes:
n/a

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[joe-elliott]

Ha! you deleted compactor bookmarks. This is my new favorite PR.

[mdisibio]

Ha! you deleted compactor bookmarks. This is my new favorite PR.

Sorry then to say they were actually just moved into iterator_multiblock.go. Although there was a slight change to retain currentError.

[joe-elliott]

Comments included.

I'm finding the logic of the iterators and channels to be a bit confusing to follow. I think it's mostly b/c you directly translated the gross bookmark logic into a goroutine. The error path (actual error vs EOF) is I think the part I find most confusing.

[joe-elliott]

Nice!

[annanay25]

Nice! Couldn't give this a thorough review but the decoupling of reads and writes using a buffered message queue looks great