Carry over pre-allocation sizes across TSDB blocks and targets

[fabxc]

We see in #2774 that new TSDB blocks being cut as well as pods being scaled cause significant memory spikes. (Seems we regressed a bit in the former since alpha.0 – not exactly sure why.)

One possible reason is that we allocate maps and slices, which then basically immediately converge to their max size. On the way, we grow these structures, which may make a lot of allocations that go to garbage shortly after.
For slices, this is easy to measure (see below). For maps, which account for most of those structures, I'm not entirely sure whether in how far it behaves the same or just dynamically adds new buckets.

For slices however, the growth is only by power of two until 1KB and slows down after, causing more garbage to be generated. Until 200k elements are created, 860k elements of garbage are created: https://play.golang.org/p/KcvqVdZjR1

We should try to pre-allocate on instantiation based on size info we have from previous blocks and targets and see whether that reduces the spikes.

@gouthamve

[gouthamve]

Closing as this no longer happens in 2.0 as we don't cut blocks anymore.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.