Fix loading huge series into RAM when points are overwritten

[jwilder]

Required for all non-trivial PRs

• Rebased/mergable
• Tests pass
• CHANGELOG.md updated

In some query scenarios, if there are a lot of points on disk for the same series spread across many blocks in TSM files and a point is overwritten near the beginning of the shard's time range, the full series could be loaded into RAM triggering OOMs and huge allocations. I believe this same problem exists in the compactor, but this PR only fixes the query path. A separate fix will be needed for compactions.

The issue was that the KeyCursor code that handles overwriting points had a simple implementation that just deduped the whole series in this case. This falls over when the series is quite large.

Instead, the KeyCursor has been changed to only decode blocks with updated points. It then keeps track of what section of the blocks have been read so they are not re-read when the later points are decoded.

Since the points in a block are always sorted, the code was also changed to remove the Deduplicate calls since they end up reallocating the slice (as well as creating an equally sized map). Instead, we do a sorted merge and re-use the slice as much as we can.

To reproduce this issue, I wrote 50M points to a single series and overwrote the first point a few times until the TSM file blocks were in the state that this issue surfaced. Then I ran a select count(value) from cpu query to could every point in the series.

Some before and after system stats:

Before

$time influx -database stress -execute "select count(value) from cpu"
name: cpu
---------
time    count
0   50000000


real    0m39.994s
user    0m0.004s
sys 0m0.006s

$ ps -o rss,vsz,pid $(pgrep influxd)
   RSS      VSZ   PID
5960276 573563196 20644

After

$ time influx -database stress -execute "select count(value) from cpu"
name: cpu
---------
time    count
0   50000000

real    0m11.352s
user    0m0.004s
sys 0m0.006s

$ ps -o rss,vsz,pid $(pgrep influxd)
   RSS      VSZ   PID
137428 573564744 20907

Query time reduced from 40s to 11.3s and RSS from 6GB to 137MB.

[mention-bot]

By analyzing the blame information on this pull request, we identified @mark-rushakoff and @joelegasse to be potential reviewers

[jwilder]

@benbjohnson

[benbjohnson]

Minor nit but we should probably save UnixNano() since we're calling it twice. Maybe atime & btime?

[benbjohnson]

Oh, wait, I was thinking those were time.Time values. Ok, it probably doesn't make a big difference if they're just grabbing int64 values underneath.

[jwilder]

Might be a small savings to save the slice indexing call.

[e-dard]

minor nit: you could save a few lines with:

if a[i].UnixNano() > b[j].UnixNano() {
    a[i], b[j] = b[j], a[i]
} else if a[i].UnixNano() == b[j].UnixNano(){
    a[i] = b[j]
    j++
}
i++

[jwilder]

@e-dard That's nicer. I'll update it to use:

    var i, j int
    for ; i < len(a) && j < len(b); i++ {
        av, bv := a[i].UnixNano(), b[j].UnixNano()
        if av > bv {
            a[i], b[j] = b[j], a[i]
        } else if av == bv {
            a[i] = b[j]
            j++
        }
    }

[benbjohnson]

Overall it lgtm. Just a few minor comments.

[e-dard]

LGTM 👍