vmui throws "Request failed with status 422" error

[surekha3]

Is your feature request related to a problem? Please describe

We've deployed victoriametrics cluster into AWS EKS. We're experiencing high disk usage by storage pods. When dig deeper, indexdb is consuming almost 2x-3x more space than data folder. According to Victoriametrics docs, this problem might be due to high churn rate. So inorder to identify root cause, I would like to view cardinality explorer but in our VMUI, it's throwing Error: Request failed with status 422 when I try to explore cardinality. Please help

Describe the solution you'd like

VMUI should display cardinality

Describe alternatives you've considered

How to fix high churn rate

Additional information

No response

[zekker6]

Hello @surekha3
Could you provide the following information:

• which version of VictoriaMetrics are you running?
• are there any errors in VictoriaMetrics logs?
• could you check the network tab in browser to get the full error text and share the result?

[surekha3]

Hi @zekker6 ,

1. We're running victoriametrics app version 1.95.1
2. In victoriametrics select logs, I have seen attached error log. Also after sometime, I observed there is drastic consumption of CPU and RAM by victoriametrics storage pods. Is this because of reason of VMUI?
3. It's working sometimes but when I changed the date on top right it's throwing attached error page

[zekker6]

@surekha3 Based on the vmselect error messages it seems like vmstorage is not able to keep up with the requests rate and thus vmselect returns an error.
I would suggest to set up a monitoring for VictoriaMetrics cluster and a dashboard. By using this dashboard you will be able to see if vmstorage hits the resources limits.

[surekha3]

@zekker6 You mean vmstorage pod resource limits or -search.maxconcurrentRequests parametes in victoriametrics config?

[zekker6]

@surekha3 In this case it is search.maxconcurrentRequests. Default value for this flag is calculated as number of CPUs * 2 in this case. Usually this default allows to effectively utilize resources available to vmstorage node so it is likely that vmstorage is hitting resource limits as well.

[surekha3]

okay when I load VMUI explore cardinality for today's date. it's working fine. But when I load for yesterday's date it's throwing above attached 422 error. So in this case, when I increase maxconcurrentRequests parameter, will it work fine?
Also after using this VMUI, cpu and memory utilisation increased drastically. Is it because of this parameter only?

[zekker6]

okay when I load VMUI explore cardinality for today's date. it's working fine. But when I load for yesterday's date it's throwing above attached 422 error. So in this case, when I increase maxconcurrentRequests parameter, will it work fine?

Based on the info above it should help. However, screenshot of browser requests lacks an actual request which returned 422 error. It would be great if you could open "network" tab and run a request once again just to confirm that request does return this specific error.

Also after using this VMUI, cpu and memory utilisation increased drastically. Is it because of this parameter only?

It's hard to tell without cluster monitoring as mentioned above. Using Grafana dashboard for the cluster monitoring will allow to see which requests were taking place at this time and correlate these requests with resource usage spikes.

[surekha3]

okay
Can you please guide why there are so many yellow coloured vertical lines in the attached screenshot of one cluster. But I'm not seeing them in other cluster. Is this some issue with first cluster

[zekker6]

@surekha3 You can hover the cursor over an arrow in the bottom of the graph and see the title of this vertical line. Yellow ones are usually representing a component restart. It's better to investigate the root cause of the restart in order to make sure the cluster is stable.

[surekha3]

@zekker6 you're correct. It states getting restarted. Is that because of leaving default value for -search.maxconcurrentRequests value as 2. Will heavy load causes these restarts?

[zekker6]

@surekha3 Heavy load can cause restarts only in case OOM is involved. -search.maxConcurrentRequests limits amount of requests vmstorage node will process at a time in order to prevent running out of resources, so this should not cause restarts.
Could you check the reason of pods restart in k8s events?

[surekha3]

@zekker6 Though it's showing as pod restarts in Grafana dashboard, when I ran kubectl events command, it's not showing as pod get restarted which seems strange

[zekker6]

@surekha3 Restarts annotation uses the following query: sum(changes(vm_app_start_timestamp)) by(job)
Could you run this query and check the value of vm_app_start_timestamp?