Mimir queriers OOMKilling

[LasseHels]

Hello!

Our Mimir queriers regularly spike in memory usage, and we occasionally get periods of 30-60 minutes where queriers will continuously OOMKill:

Our platform is used by a wide variety of users, and we initially believed the spiky memory usage was due to certain user query patterns. However, this past Sunday (February 11th), we saw another instance of spiky memory usage. We would expect no or few users to be running queries outside of work hours.

Under normal operations, our queriers consume ~1GB of memory:

We're having a hard time finding metrics and/or log lines that correlate with the memory spikes. We would appreciate any help in narrowing down where the excessive memory usage is coming from.

This is our querier configuration:

      - args:
        - -auth.multitenancy-enabled=false
        - -auth.no-auth-tenant=fake
        - -blocks-storage.azure.container-name=mimir
        - -blocks-storage.bucket-store.metadata-cache.backend=memcached
        - -blocks-storage.bucket-store.metadata-cache.memcached.addresses=dnssrvnoa+memcached-metadata.mimir.svc.cluster.local.:11211
        - -blocks-storage.bucket-store.metadata-cache.memcached.max-async-concurrency=50
        - -blocks-storage.bucket-store.metadata-cache.memcached.max-item-size=1048576
        - -blocks-storage.bucket-store.sync-dir=/data/tsdb
        - -blocks-storage.bucket-store.sync-interval=15m
        - -common.storage.azure.account-name=pensievemimirprodwe01
        - -common.storage.backend=azure
        - -config.expand-env=true
        - -config.file=/etc/mimir/config/mimir.yaml
        - -distributor.health-check-ingesters=true
        - -ingester.ring.heartbeat-timeout=10m
        - -ingester.ring.prefix=
        - -ingester.ring.replication-factor=3
        - -ingester.ring.store=memberlist
        - -ingester.ring.zone-awareness-enabled=true
        - -mem-ballast-size-bytes=268435456
        - -memberlist.bind-port=7947
        - -memberlist.join=gossip-ring.mimir.svc.cluster.local:7947
        - -querier.cardinality-analysis-enabled=true
        - -querier.frontend-client.grpc-max-send-msg-size=419430400
        - -querier.max-concurrent=100
        - -querier.max-outstanding-requests-per-tenant=300
        - -querier.max-partial-query-length=816h
        - -querier.max-query-lookback=64w
        - -querier.query-ingesters-within=13h
        - -querier.query-store-after=12h
        - -query-scheduler.ring.prefix=
        - -query-scheduler.ring.store=memberlist
        - -query-scheduler.service-discovery-mode=ring
        - -runtime-config.file=/etc/mimir/overrides.yaml
        - -server.grpc.keepalive.min-time-between-pings=10s
        - -server.grpc.keepalive.ping-without-stream-allowed=true
        - -server.http-listen-port=8080
        - -server.http-write-timeout=2m
        - -store-gateway.sharding-ring.prefix=multi-zone/
        - -store-gateway.sharding-ring.replication-factor=3
        - -store-gateway.sharding-ring.store=memberlist
        - -store-gateway.sharding-ring.zone-awareness-enabled=true
        - -target=querier

[pstibrany]

Hello. If your queriers are configured with activity tracker enabled (it is enabled by default, but requires emptyDir volume to store data -- if you use Helm or our Jsonnet, this should be the case), then on next restart queriers will log all ongoing queries that were running when querier crashed. This can be used to find out which queries caused OOM.

We would expect no or few users to be running queries outside of work hours.

It can be a single query causing OOMs. Once you understand which query it is and what exactly is happening, you can protect queriers by placing limits on the queries.

[LasseHels]

@dimitarvdimitrov Thanks for your reply! We will take a look at the suggested items.

[LasseHels]

We've lowered querier.max-concurrent from 100 to 8. This has had some positive effect, but we still see regular OOMKills.

We are looking at different Mimir configuration options, and were wondering whether modifying our sharding configuration could have some effect on memory usage?

This is our current sharding configuration

        - -query-frontend.parallelize-shardable-queries=true
        - -query-frontend.query-sharding-max-sharded-queries=128
        - -query-frontend.query-sharding-target-series-per-shard=2500
        - -query-frontend.query-sharding-total-shards=16

Specifically, we are thinking about lowering -query-frontend.query-sharding-target-series-per-shard, but our understanding of how this configuration option works is fairly limited.

We are also looking at the -querier.max-samples configuration option. Right now, we have it set to the default of 50000000. We would expect there to be a correlation between the amount of samples that a given query fetches and the memory usage incurred by that query. Is that true? If so, lowering -querier.max-samples could prevent costly queries from being ran at all.

[dimitarvdimitrov]

Specifically, we are thinking about lowering -query-frontend.query-sharding-target-series-per-shard, but our understanding of how this configuration option works is fairly limited.

this flag helps when the same query has been ran previously over a different time range. The feature uses the actually processed series in the previous request and use them to estimate the shards for the current query.

We would expect there to be a correlation between the amount of samples that a given query fetches and the memory usage incurred by that query. Is that true?

that's true. Also consider lowering the limits -querier.max-fetched-series-per-query and -querier.max-fetched-chunks-per-query, which can also be configured on a per-tenant basis. Those limits are applied earlier in the query path and can be hit before the querier has to load everything in memory to run the promQL engine. So they have somewhat better protection against OOMs.

[LasseHels]

For the record, we were able to mostly solve this issue by changing only -querier.max-concurrent back down to 8 for both our regular queriers and ruler queriers. We still occasionally see OOMKills, but the rate has been reduced to the point where it's not a problem in practice.

Thanks for the help @dimitarvdimitrov and @pstibrany ❤️.

[pstibrany]

That's great to hear!