store: Store gateway consuming lots of memory / OOMing

[felipejfc]

Thanos, Prometheus and Golang version used
thanos v0.1.0rc2

What happened
Thanos-store is consuming 50gb of memory during startup

What you expected to happen
Thanos-store does not consume so much memory for starting up

Full logs to relevant components
store:

level=debug ts=2018-07-27T15:51:21.415788856Z caller=cluster.go:132 component=cluster msg="resolved peers to following addresses" peers=100.96.232.51:10900,100.99.70.149:10900,100.110.182.241:10900,100.126.12.148:10900
level=debug ts=2018-07-27T15:51:21.416254389Z caller=store.go:112 msg="initializing bucket store"
level=warn ts=2018-07-27T15:52:05.28837034Z caller=bucket.go:240 msg="loading block failed" id=01CKE41VDSJMSAJMN6N6K8SABE err="new bucket block: load index cache: download index file: copy object to file: write /var/thanos/store/01CKE41VDSJMSAJMN6N6K8SABE/index: cannot allocate memory"
level=warn ts=2018-07-27T15:52:05.293692332Z caller=bucket.go:240 msg="loading block failed" id=01CKE41VE4XXTN9N55YPCJSPP2 err="new bucket block: load index cache: download index file: copy object to file: write /var/thanos/store/01CKE41VE4XXTN9N55YPCJSPP2/index: cannot allocate memory"

Anything else we need to know
Some time after initialization the ram usage goes down to normal levels, something around 8Gb

Another thing that's happening is that my thanos-compactor consumer way too much ram memory as well, the last time it ran, it used up to 60Gb of memory.

I run store with this args:

      containers:
      - args:
        - store
        - --log.level=debug
        - --tsdb.path=/var/thanos/store
        - --s3.endpoint=s3.amazonaws.com
        - --s3.access-key=xxx
        - --s3.bucket=xxx
        - --cluster.peers=thanos-peers.monitoring.svc.cluster.local:10900
        - --index-cache-size=2GB
        - --chunk-pool-size=8GB

Environment:

• OS (e.g. from /etc/os-release): kubernetes running on debian

[bwplotka]

Do you have compactor running? If not that is expected as you might have millions of small blocks in your bucket stored in super inefficient way pt., 27 lip 2018, 17:57 użytkownik Felipe Cavalcanti < notifications@github.com> napisał:

…

*Thanos, Prometheus and Golang version used* thanos v0.1.0rc2 *What happened* Thanos-store is consuming 50gb of memory during startup *What you expected to happen* Thanos-store does not consume so much memory for starting up *Full logs to relevant components* store: level=debug ts=2018-07-27T15:51:21.415788856Z caller=cluster.go:132 component=cluster msg="resolved peers to following addresses" peers=100.96.232.51:10900,100.99.70.149:10900,100.110.182.241:10900,100.126.12.148:10900 level=debug ts=2018-07-27T15:51:21.416254389Z caller=store.go:112 msg="initializing bucket store" level=warn ts=2018-07-27T15:52:05.28837034Z caller=bucket.go:240 msg="loading block failed" id=01CKE41VDSJMSAJMN6N6K8SABE err="new bucket block: load index cache: download index file: copy object to file: write /var/thanos/store/01CKE41VDSJMSAJMN6N6K8SABE/index: cannot allocate memory" level=warn ts=2018-07-27T15:52:05.293692332Z caller=bucket.go:240 msg="loading block failed" id=01CKE41VE4XXTN9N55YPCJSPP2 err="new bucket block: load index cache: download index file: copy object to file: write /var/thanos/store/01CKE41VE4XXTN9N55YPCJSPP2/index: cannot allocate memory" *Anything else we need to know* One thing that's happening is that my thanos-compactor consumer way too much ram memory as well, the last time it ran, it used up to 60Gb of memory. *I run store with this args:* containers: - args: - store - --log.level=debug - --tsdb.path=/var/thanos/store - --s3.endpoint=s3.amazonaws.com - --s3.access-key=xxx - --s3.bucket=xxx - --cluster.peers=thanos-peers.monitoring.svc.cluster.local:10900 - --index-cache-size=2GB - --chunk-pool-size=8GB *Environment*: - OS (e.g. from /etc/os-release): kubernetes running on debian — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#448>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGoNuygVNcs4idveMfvyH5JaitCZ7Mjzks5uKzhxgaJpZM4VjvTD> .

[felipejfc]

hi @Bplotka
I do actually, it also uses tons of memory (~60Gb) in the last run, is this normal?

[bwplotka]

it also uses tons of memory (~60Gb) in the last run, is this normal?

Yes, fix is in review: #529

[bwplotka]

But huge mem usage on startup for store gateway is mainly because fine greained blocks - I think your compactor did not compact all things yet. Might that be the case?

[xjewer]

that's a different problem, #529 fixes the downsampling issue #297

[bwplotka]

@xjewer no:

it also uses tons of memory (~60Gb) in the last run, is this normal?

It = means compactor in #448 (comment) (: so #529 is actually fix for this

[xjewer]

oh, I missed the details, ok then 😀

[awprice]

TL;DR - We are currently seeing thanos-store consuming incredibly large amounts of memory during initial sync and then being OOM killed. It is not releasing any memory as it is performing the initial sync and there is very likely a memory leak. Memory leak is likely to be occurring in https://github.com/improbable-eng/thanos/blob/v0.1.0/pkg/block/index.go#L105-L154

Thanos, Prometheus and Golang version used
thanos-store 0.1.0, Golang 1.11 (built with quay.io/prometheus/golang-builder:1.11-base)

What happened
thanos-store is consuming 32gb of memory during initial sync, then being OOM (out of memory) killed

What you expected to happen
thanos-store not to use this much memory on initial sync and to progress past the initial sync

Full logs to relevant components
No logs are emitted whilst the initial sync is occurring, see graphs below

Anything else we need to know
Here is a graph of the total memory usage (cache + rss), rss memory usage and cache memory usage:

We have Kubernetes memory limits on the thanos-store container set to 32Gb, which is why it is eventually killed when it reaches this point.

Our thanos S3 bucket is currently 488.54404481872916G, 15078 objects in size.

We've noticed that thanos-store doesn't progress past the InitialSync function - https://github.com/improbable-eng/thanos/blob/v0.1.0/cmd/thanos/store.go#L113 and exceed the memory limits of the container before finishing.

We've modified the goroutine count for how many blocks are being processed concurrently. It is currently hardcoded 20, but by changing it to a much lower number, e.g. 1, we can have thanos-store last longer before being OOM killed. Although it does take longer to do the InitialSync - https://github.com/improbable-eng/thanos/blob/v0.1.0/pkg/store/bucket.go#L231

The goroutine count for SyncBlocksshould really be a configurable option as well, hard coding it to 20 is not ideal.

Through some debugging, we've identified the loading of the index cache as the location of the memory leak - https://github.com/improbable-eng/thanos/blob/v0.1.0/pkg/store/bucket.go#L1070

By commenting out that function from the newBucketBlock function, thanos-store is able to progress past the InitialSync, (albeit without any index caches) and consumes very little memory.

We then ran some pprof heap analysis on the thanos-store as the memory leak was occurring and it identified block.ReadIndexCache as consuming alot of memory, see image below of the pprof heap graph

The function in question - https://github.com/improbable-eng/thanos/blob/v0.1.0/pkg/block/index.go#L105-L154. The heap graph above suggests that the leak is in the json encoding/decoding of the index file and for some reason is not releasing memory.

[jdfalk]

Any update on this? We have blocks that after compaction but before downsampling are 400GB which means either we run a massively expensive AWS instance or just add a massive swap file.

[mwhittington21]

Any update on this? We can't use Thanos at the scale that we want to because of this.

[pathikritmodak]

Any update on this? Can we have a timeline for this fix?

[bwplotka]

Thanks guys for this, especially @awprice for detailed work. This is interesting as our heap profiles were totally different - suggesting proper place -> fetching bytes into buffer for actual series in the query.

Maybe you have bigger indexes and not much traffic on the query side? You said:

Our thanos S3 bucket is currently 488.54404481872916G, 15078 objects in size.

This is totally reasonable number. Can you check you biggest blocks? How larger they are and notably what is the index size? (:

[bwplotka]

Also sorry for delay, I totally missed this issue

[awprice]

@bwplotka Apologies for the late reply, here is some info on our largest blocks/index sizes:

Largest block - 14 GiB
Index size for that block - 6 GiB

Our largest index is 7 GiB