Mutex contention in vmselect

[misutoth]

Describe the bug

Using VictoriaMetrics cluster version, and I choose to use vmselect v1.93.3-cluster, some queries execute fast but some takes orders of magnitude longer. See
We are using HPA but apparently the system does not want to scale up. It seems there is some problem preventing parallelization. There are "only" 6 vmselect pods so it seems there is not a loack of CPU resources. And indeed, when I execute the following on vmselect:

$ go tool pprof -web http://localhost:53808/debug/pprof/goroutine

We can see that the goroutines are waiting on the mutex to access items InternString cache. Please see
Mutex contention.pdf

When I disable caching with internStringDisableCache=true the contention disappears, vmselect instance count increases to 9, response time drops and stabilizes:
No contention.pdf

To Reproduce

We have a performance test environment to simulate the production load. We simulate about 430 million unique time series, stored with replication factor of 2. The ingestion speed is 6.5 million samples per second. We have about 700 recording rules generating queries quite frequently.

Version

/ # /vmselect-prod --version
vmselect-20230902-002725-tags-v1.93.3-cluster-0-gf78d8b994d

/ # /vmstorage-prod --version
vmstorage-20230902-002932-tags-v1.93.3-cluster-0-gf78d8b994d

/ # /vminsert-prod --version
vminsert-20230902-002549-tags-v1.93.3-cluster-0-gf78d8b994d

Logs

may not be needed

Screenshots

Mutex contention.pdf

No contention.pdf

Used command-line flags

vmslect:
dedup.minScrapeInterval=1ms
envflag.enable=true
envflag.prefix=VM_
http.connTimeout=15s
http.maxGracefulShutdownDuration=2m
loggerFormat=json
memory.allowedBytes=40GiB
replicationFactor=2
search.maxConcurrentRequests=150
search.maxLookback=6m
search.maxQueryDuration=300s
search.maxSamplesPerQuery=5000000000
search.maxSeries=500000
search.maxStalenessInterval=6m
search.maxUniqueTimeseries=100000000
search.minStalenessInterval=5m
vmalert.proxyURL=http://alert-rules:8880
vmstorageDialTimeout=1s

vmstorage:
retentionPeriod=1
storageDataPath=/storage
envflag.enable=true
envflag.prefix=VM_
http.connTimeout=15s
loggerFormat=json
memory.allowedBytes=220GiB
search.maxUniqueTimeseries=100000000
smallMergeConcurrency=4

vminsert:
envflag.enable=true
envflag.prefix=VM_
http.connTimeout=15s
insert.maxQueueDuration=30s
loggerFormat=json
maxConcurrentInserts=50
maxInsertRequestSize=1GiB
maxLabelsPerTimeseries=70
memory.allowedBytes=20GiB
opentsdbhttp.maxInsertRequestSize=512MiB
replicationFactor=2
vmstorageDialTimeout=2s

Additional information

I have just realized as I was writing my report and was looking at the source code that I can disable this kind of caching using the internStringDisableCache flag. Originally I wanted to report it as a regression and compare it with v1.81.2.

Disabling this cache is a good workaround but probably the motivation was that it tolerates high concurrency. So this bug report may be useful still.

[zekker6]

Hello @misutoth , thank you for the detailed report!

Currently, VictoriaMetrics uses Golang sync.Map which is optimized for this use-case(1):

The Map type is optimized for two common use cases: (1) when the entry for a given key is only ever written once but read many times, as in caches that only grow, or (2) when multiple goroutines read, write, and overwrite entries for disjoint sets of keys.

So there is not much we can optimize on our side without removing sync.Map.
Is it possible that during alert / recording rules calculation a lot of unique series is fetched?
In case it is true, there will be a lot of unique strings fetched at every evaluation, which makes it almost harmful to try to intern and cache all these values as values will not be reused later on.

@valyala Could you advise if it would make sense to try replacing sync.Map with fastcache here?

[misutoth]

Hello all. Thanks for the quick response.

Is it possible that during alert / recording rules calculation a lot of unique series is fetched?

Yes, a lot of time series are used. However, there are not that many new time series on each execution of the queries.

I am thinking that the internStringCacheExpireDuration default of 6 minutes may be too short in our use case. We have around 8-12 instances of vmselect which are load balanced. Also, many of the rules have 3 minutes interval. All these increases the chances that the interned strings get evicted. That could lead to such inefficiency, couldnt it?

[hagen1778]

@misutoth if I'm not mistaken, it is a bug that interning is still used in vmselect. It shouldn't be there.
AFAIK, @f41gh7 is going to remove it as soon as he has time.

[misutoth]

@misutoth if I'm not mistaken, it is a bug that interning is still used in vmselect. It shouldn't be there. AFAIK, @f41gh7 is going to remove it as soon as he has time.

I havent reached that conclusion yet but I believe you. 😄 I was just thinking that maybe with a more generous eviction time it might be still useful even in an environment with high cpu cores and multiple vmselects. Because I guess in most deployments and in most of the times churn is supposed to be low. So the interned strings may be read multiple times throughout their lifecycle. If you give me a reasonable eviction time I can run the performance test in our environment with that.

[valyala]

@misutoth , could you build vmselect from the commit 4278b00 according to these instructions and verify whether it resolves the mutex contention during querying under your workload?

[valyala]

FYI, the fix, which reduces mutex contention at vmselect on machines with big number of CPU cores has been included in VictoriaMetrics v1.93.6. @misutoth , could you verify whether this release really fixes the issue with high mutex contention at vmselect under your workload?

P.S. The fix will be also included in the upcoming v1.95.0 release.

[misutoth]

All looks good with v1.93.6. Thank you all! ❤️

[valyala]

@misutoth , thanks for the confirmation that the fix resolves scalability issues for vmselect running on systems with big number of CPU cores. FYI, there is yet another pull request, which can improve vmselect scalability on systems with big number of CPU cores further - #5036 .

[valyala]

Closing the issue as fixed

[valyala]

FYI, the next release of VictoriaMetrics will support -search.maxWorkersPerQuery command-line flag, which can be used for fine-tuning query performance on systems with more than 16 CPU cores for particular workloads. See the description of this flag in the Troubleshooting docs for more details.

[valyala]

The -search.maxWorkersPerQuery command-line flag is available starting from v1.95.0 release.