How to scale the read path?

[Sinketsu]

Is your question request related to a specific component?

vmselect, vmstorage

Describe the question in detail

Hello!
We have a relatively big cluster with a lot of timeseries. Basically, these are some kind of system metrics of microservices, for example, response codes or query timings in them.

We use a lot of small load balancers (something like a service-mesh) to do requests from one microservice to another. Each of the load balancers provides metrics about each service. Next, we want to show aggregated metrics for a service/set of services.

For example, we want to make a summary graph of all 5xx response codes of all microservices. To do this, we need to take a lot of unique timeseries from the storage and combine them. It takes a very long time. Now such a request takes about 20-25 seconds, which is a very long time.

My question is - what is the main idea for scaling the reading pipeline? We plan to further increase the number of load balancers in our cluster.

Our ideas:

• Increase CPU/RAM for vmselect. We are currently using the maximum flavor available to us. Large values for CPU/RAM will have a worse effect on the possibility of settling them in the orchestrator (Kubernetes/Nomad/etc)
• Use recording rules. It's a good idea if we split the query into some sequential component parts (for example, aggregation by services). We will try it
• Use stream aggregation. An analogue of recording rolls, we will also try, even most likely in conjunction with recording rules. But in total, this will introduce an additional delay in the data

Do I understand correctly that if it is not possible to vertically scale vmselects, then it remains only to introduce sequential processing through recording rules /stream aggregation, which will cause its inconveniences (the names of the metrics will not match their original names) and delays?

Or maybe there are other ways to improve the cluster's read performance? I will be glad for any advice/thoughts.

Thank you very much!

Troubleshooting docs

• General - https://docs.victoriametrics.com/troubleshooting/
• vmagent - https://docs.victoriametrics.com/vmagent/#troubleshooting
• vmalert - https://docs.victoriametrics.com/vmalert/#troubleshooting

[hagen1778]

Do I understand correctly that if it is not possible to vertically scale vmselects, then it remains only to introduce sequential processing through recording rules /stream aggregation, which will cause its inconveniences (the names of the metrics will not match their original names) and delays?

It is correct. vmselect can't shard query execution among other vmselects. Hence, it can be scaled only vertically for resource-intensive queries.
The scaling logic is the following:

1. If you have intensive read queries - prefer a smaller number of vmselects with high resource capacity
2. If you have many light-weight queries - prefer a bigger number of vmselects with light resource capacity

These two approaches could be mixed. For example, alerting and recording rules are usually considered as lightweight load, since they usually select short time intervals. So the ruler (vmalert) could be configured to "talk to" a fleet of many lightweight queries.
If you have some heavy Grafana dashboards or other queriers, configure them to "talk to" a fleet of smaller number of heavy vmselects which would crunch big amounts of data faster.

Or maybe there are other ways to improve the cluster's read performance? I will be glad for any advice/thoughts.

I'd recommend starting with https://docs.victoriametrics.com/troubleshooting/#slow-queries.
Recommendations from the doc and query tracing could help to understand where the bottleneck is. It could be vmselect's compute resource, but it also could be vmstorage disk IO, vmstorage CPU, network between vmstorage and vmselect. In this case, scaling vmselects will have no effect on query latency.

[Sinketsu]

1. If you have intensive read queries - prefer a smaller number of vmselects with high resource capacity
2. If you have many light-weight queries - prefer a bigger number of vmselects with light resource capacity

These two approaches could be mixed. For example, alerting and recording rules are usually considered as lightweight load, since they usually select short time intervals. So the ruler (vmalert) could be configured to "talk to" a fleet of many lightweight queries. If you have some heavy Grafana dashboards or other queriers, configure them to "talk to" a fleet of smaller number of heavy vmselects which would crunch big amounts of data faster.

An interesting idea, we will try to divide them, thank you!

Or maybe there are other ways to improve the cluster's read performance? I will be glad for any advice/thoughts.

I'd recommend starting with https://docs.victoriametrics.com/troubleshooting/#slow-queries. Recommendations from the doc and query tracing could help to understand where the bottleneck is. It could be vmselect's compute resource, but it also could be vmstorage disk IO, vmstorage CPU, network between vmstorage and vmselect. In this case, scaling vmselects will have no effect on query latency.

We looked at it. In our case (judging by the traces), it takes most of the work time to receive data from the vmstorage-s (since there are many of them) and aggregate data on vmselect-s.

[hagen1778]

it takes most of the work time to receive data from the vmstorage-s (since there are many of them)

vmselect receives data from vmstorages in parallel, so many vmstorages shouldn't delay data retrieval. It should be opposite, the more vmstorages you have the faster they process the search query and respond with data.

and aggregate data on vmselect-s.

If aggregation on vmselect takes significant amount of time this is where vertical scaling should help. Please note, vmselect does query processing in parallel fashion, running a separate worker per each available CPU. These workers are responsinble for unpacking data received from vmstorages and performing concurrent calculations before the final merge.

[Sinketsu]

it takes most of the work time to receive data from the vmstorage-s (since there are many of them)

vmselect receives data from vmstorages in parallel, so many vmstorages shouldn't delay data retrieval. It should be opposite, the more vmstorages you have the faster they process the search query and respond with data.

I mean a lot of data blocks to retreive for query. Now we have 5 vmstorage servers. Do you have any recommendations on how many hosts it is optimal to have, depending on the amount of data? Maybe we should increase their number for better performance...

and aggregate data on vmselect-s.

If aggregation on vmselect takes significant amount of time this is where vertical scaling should help. Please note, vmselect does query processing in parallel fashion, running a separate worker per each available CPU. These workers are responsinble for unpacking data received from vmstorages and performing concurrent calculations before the final merge.

Yes, I understand that, but we can no longer scale vmselect vertically(

[hagen1778]

Do you have any recommendations on how many hosts it is optimal to have, depending on the amount of data?

It doesn't depend on amount of data, but rather on the bottlenecks during query processing. For example, you have 5 storage nodes, each has local disk with 100MB/s read throughput. Then one query (in the best case) can utilize 500MB/s read speed. If you add 5 more vmstorage nodes, then one query would be able to utilize 1000MB/s.

In short, scaling number for vmstorage nodes is similar to scaling the number of concurrent workers - it is always beneficial. Unless the vmselect is the bottleneck and can't process the received data fast enough.

Scaling vmstorage nodes makes sense up to 30-50 nodes. After that, the fragility of network starts to kick-in increasing query latency because of higher probability of network lag.

[Sinketsu]

Ok, we will try to increase our vmstorage nodes count too. Thanks!

[hagen1778]

Closing question as answered. Feel free to re-open or create a new issue if you still have questions.