vmselect pod restarting

[rambabubolla]

Describe the bug

vmselect pod restarting continuously

To Reproduce

events logs
kubectl get event -n mon --field-selector involvedObject.name=vmselect-vms-0

4m5s Normal Scheduled pod/vmselect-vms-0 Successfully assigned sysmgmt-health/vmselect-vms-0 to ncn-w006
3m58s Normal AddedInterface pod/vmselect-vms-0 Add eth0 [10.34.128.19/12] from weave
3m58s Normal Pulled pod/vmselect-vms-0 Container image "victoriametrics/vmselect:v1.93.1-cluster" already present on machine
3m58s Normal Created pod/vmselect-vms-0 Created container vmselect
3m57s Normal Started pod/vmselect-vms-0 Started container vmselect
70s Normal Killing pod/vmselect-vms-0 Stopping container vmselect
69s Normal Scheduled pod/vmselect-vms-0 Successfully assigned sysmgmt-health/vmselect-vms-0 to ncn-w006
67s Normal AddedInterface pod/vmselect-vms-0 Add eth0 [10.34.128.19/12] from weave
67s Normal Pulled pod/vmselect-vms-0 Container image "victoriametrics/vmselect:v1.93.1-cluster" already present on machine
67s Normal Created pod/vmselect-vms-0 Created container vmselect
67s Normal Started pod/vmselect-vms-0 Started container vmselect
64s Normal Killing pod/vmselect-vms-0 Stopping container vmselect
62s Normal Scheduled pod/vmselect-vms-0 Successfully assigned sysmgmt-health/vmselect-vms-0 to ncn-w006
57s Normal AddedInterface pod/vmselect-vms-0 Add eth0 [10.34.128.19/12] from weave
57s Normal Pulled pod/vmselect-vms-0 Container image "victoriametrics/vmselect:v1.93.1-cluster" already present on machine
57s Normal Created pod/vmselect-vms-0 Created container vmselect
57s Normal Started pod/vmselect-vms-0 Started container vmselect

pod logs

2024-04-16T07:41:33.943Z info VictoriaMetrics/lib/logger/flag.go:12 build version: vmselect-20230824-081515-tags-v1.93.1-cluster-0-g54a67d439c
2024-04-16T07:41:33.943Z info VictoriaMetrics/lib/logger/flag.go:13 command-line flags
2024-04-16T07:41:33.943Z info VictoriaMetrics/lib/logger/flag.go:20 -cacheDataPath="/select-cache"
2024-04-16T07:41:33.943Z info VictoriaMetrics/lib/logger/flag.go:20 -dedup.minScrapeInterval="30s"
2024-04-16T07:41:33.943Z info VictoriaMetrics/lib/logger/flag.go:20 -httpListenAddr=":8481"
2024-04-16T07:41:33.943Z info VictoriaMetrics/lib/logger/flag.go:20 -selectNode="vmselect-vms-0.vmselect-vms.sysmgmt-health:8481,vmselect-vms-1.vmselect-vms.sysmgmt-health:8481"
2024-04-16T07:41:33.943Z info VictoriaMetrics/lib/logger/flag.go:20 -storageNode="vmstorage-vms-0.vmstorage-vms.sysmgmt-health:8401,vmstorage-vms-1.vmstorage-vms.sysmgmt-health:8401"
2024-04-16T07:41:33.943Z info VictoriaMetrics/lib/logger/flag.go:20 -vmalert.proxyURL="http://vmalert-vms.sysmgmt-health.svc:8080"
2024-04-16T07:41:33.943Z info VictoriaMetrics/app/vmselect/main.go:96 starting netstorage at storageNodes [vmstorage-vms-0.vmstorage-vms.sysmgmt-health:8401 vmstorage-vms-1.vmstorage-vms.sysmgmt-health:8401]
2024-04-16T07:41:34.982Z info VictoriaMetrics/app/vmselect/main.go:110 started netstorage in 1.039 seconds
2024-04-16T07:41:34.994Z info VictoriaMetrics/lib/memory/memory.go:42 limiting caches to 6442450944 bytes, leaving 4294967296 bytes to the OS according to -memory.allowedPercent=60
2024-04-16T07:41:34.994Z info VictoriaMetrics/app/vmselect/promql/rollup_result_cache.go:66 loading rollupResult cache from "/select-cache/rollupResult"...
2024-04-16T07:41:35.007Z info VictoriaMetrics/app/vmselect/promql/rollup_result_cache.go:94 loaded rollupResult cache from "/select-cache/rollupResult" in 0.013 seconds; entriesCount: 72, sizeBytes: 4325376
2024-04-16T07:41:35.007Z info VictoriaMetrics/lib/httpserver/httpserver.go:96 starting http server at http://127.0.0.1:8481/
2024-04-16T07:41:35.007Z info VictoriaMetrics/lib/httpserver/httpserver.go:97 pprof handlers are exposed at http://127.0.0.1:8481/debug/pprof/
2024-04-16T07:43:34.106Z info VictoriaMetrics/app/vmselect/querystats/querystats.go:87 enabled query stats tracking at /api/v1/status/top_queries with -search.queryStats.lastQueriesCount=20000, -search.queryStats.minQueryDuration=1ms

Version

• https://github.com/VictoriaMetrics/helm-charts
  type: application
  version: 0.20.0

Logs

No response

Screenshots

No response

Used command-line flags

No response

Additional information

No response

[dmitryk-dk]

Hi @rambabubolla ! In the logs you can find that the

57s Normal Pulled pod/vmselect-vms-0 Container image "victoriametrics/vmselect:v1.93.1-cluster" already present on machine

Can you uninstall your cluster via the helm uninstall command and run install again? Or can you run helm upgrade?

[dmitryk-dk]

Hi @rambabubolla ! Could you check also OOM events ? Could you check the pod events?

[rambabubolla]

This issue is related to chart requires kubeVersion: >=1.23.0-0 which is incompatible with Kubernetes v1.22.13

we are using older version