Compaction memory requirements are too high

[SuperQ]

Proposal

Reduce the default of the --storage.tsdb.max-block-duration from 10% to 2% or 1%.

Bug Report

When using a longer retention time on a high traffic Prometheus server, the 10% compaction level causes a number of users to get into crash loops. This seems to be due to the amount of memory used during the compaction.

For example, a server with a 365d retention time. This server is normally operating at ~20G RSS and needs an additional 6G of memory to perform a 2% compaction of ~60G of TSDB. If this were a full 10% compaction, it could need far more memory for this compaction, causing it to OOM.

There's also no need to generate such large blocks, with a 2% max block duration ratio, we would have under 60 TSDB blocks over a year.

[SuperQ]

Some additional information on an existing TSDB compaction.

This is for a level 5 compaction, 6.75 days. (--storage.tsdb.retention=365d, --storage.tsdb.max-block-duration=7d).

"stats": {
  "numSamples": 57293773416,
  "numSeries": 3522226,
  "numChunks": 485794168
},

The churn on this server is somewhat low, but it has a large ingestion rate and a lot of metrics.

This compaction has a 3.6GB index and 62GB of chunks.

[brian-brazil]

Did you ever get the heap profiles when the compaction was happening?

[SuperQ]

No, I haven't gotten to that yet, thanks for the reminder.

[brian-brazil]

In addition, could I get all the go memory metrics during the compaction spike?

[tcolgate]

Does an msync actually reclaim space allocated to dirty pages? Could periodic msyncs reduce the usage? (tested it, it doesn't seem to, RSS drops once you unmap)

On a more practical note, can storage.tsdb.max-block-duration be set on a pre-existing prom instance? It seems like reducing the block size could reduce the memory used here?

[krasi-georgiev]

Maybe some streaming during compaction could solve this.

Instead of expanding all blocks at once do it in stages or something.

[brian-brazil]

With the PRs I've sent out, it's already as efficient as it's going to get. It seems that @SuperQ's issue was likely cardinality, which my PRs should address.

[krasi-georgiev]

looking at the bench results for #4816 worse memory jump I found is from 30 to 40gb for a week run with the default settings which doesn't seem so bad.

http://prombench.prometheus.io/grafana/d/7gmLoNDmz/prombench?orgId=1&var-RuleGroup=All&var-pr-number=4816&from=1541709439128&to=1541714031412

@SuperQ do you see any bigger jumps?

[SuperQ]

@krasi-georgiev The issues we saw were with the default 10% max block size when compacting long retention times (365d). We tun our max block size to 7d, which helps. Once we have Brian's memory use optimizations in a release, I can do some re-testing.

[spjspjspj]

Why would entire block need to be loaded into memory? We observe high memory use (130GB) too upon (probably unclean) restart.

Edit: this is actually unrelated to this issue. What we experienced I believe is actually #4842

On our dashboard, compaction seemingly happens once every hour and doesn't take much time or memory:

retention 180d
min-block 1h
max-block 1d

[SuperQ]

Here's a comparison of 2.5.0 to 2.6.0:

Looking at the data, it seems we still see about a 20-25% spike in RSS size when two compactions are run closely together.

Oddly, the go_memstats_alloc_bytes doesn't seem to notice this use.

Perhaps we need to add some sleep time to the scheduling of compactions to avoid this?

[SuperQ]

So, after some discussion with @brian-brazil, I think this is solved. Figuring out the RSS issue when compactions run is a separate problem.