Bug: vmselect deployment should use selectNode argument to support metrics deletion

[fsperling]

Hi,

deleting a metric via:
https://fqdn/delete/0/prometheus/api/v1/admin/tsdb/delete_series?match[]=metric-name

results in the following error in the vmselect log:

2020-11-09T08:23:49.299Z panic VictoriaMetrics/app/vmselect/prometheus/prometheus.go:304 BUG: missing -selectNode flag 2020-11-09T08:23:49.300Z error net/http/server.go:3093 http: panic serving 10.28.1.3:36772: BUG: missing -selectNode flag goroutine 20556321 [running]: net/http.(*conn).serve.func1(0xc0252c83c0) net/http/server.go:1801 +0x147 panic(0x9d2860, 0xc025739c80) runtime/panic.go:975 +0x3e9 github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logMessage(0xa4d4fa, 0x5, 0xc02574b480, 0x1d, 0x4) github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:191 +0xa0c github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logLevelSkipframes(0x1, 0xa4d4fa, 0x5, 0xa5c44a, 0x1d, 0x0, 0x0, 0x0) github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:124 +0xd1 github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logLevel(...) github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:116 github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.Panicf(...) github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:112 github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/prometheus.resetRollupResultCaches() github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/prometheus/prometheus.go:304 +0x3ef github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/prometheus.DeleteHandler(0xbfe25d054d2d8760, 0x174d1191d6a4e, 0xda5540, 0xc02c041638, 0xc01fa24a00, 0x2a, 0xc025739ba0) github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/prometheus/prometheus.go:294 +0x3eb main.deleteHandler(0xbfe25d054d2d8760, 0x174d1191d6a4e, 0xda5540, 0xaebbc0, 0xc025743a10, 0xc01fa24a00, 0xc025743a40, 0xc02c041638, 0x34) github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/main.go:311 +0xc5 main.requestHandler(0xaebbc0, 0xc025743a10, 0xc01fa24a00, 0x0) github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/main.go:178 +0x44b github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.handlerWrapper(0xc0003103b0, 0xaebbc0, 0xc025743a10, 0xc01fa24a00, 0xa78ea0) github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver/httpserver.go:221 +0x267 github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.gzipHandler.func1(0xaebbc0, 0xc025743a10, 0xc01fa24a00) github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver/httpserver.go:156 +0xa7 net/http.HandlerFunc.ServeHTTP(0xc000302220, 0xaebec0, 0xc038840700, 0xc01fa24a00) net/http/server.go:2042 +0x44 net/http.serverHandler.ServeHTTP(0xc00033c000, 0xaebec0, 0xc038840700, 0xc01fa24a00) net/http/server.go:2843 +0xa3 net/http.(*conn).serve(0xc0252c83c0, 0xaed480, 0xc13d824380) net/http/server.go:1925 +0x8ad created by net/http.(*Server).Serve net/http/server.go:2969 +0x36c

This is arguments is needed to that the vmselect receiving the delete can tell the other vmselect pods to update/invalidate their cache.

From the man page:
-selectNode array Addresses of vmselect nodes; usage: -selectNode=vmselect-host1:8481 -selectNode=vmselect-host2:8481 Supports array of values separated by comma or specified via multiple flags.

[tenmozes]

@valyala could you please take a look?

[f41gh7]

Helm part was fixed, but you have to use statefulset deployment for vmselect.

Fix at vmselect side will be included into the next release.