[cluster/vminsert]:add reload -relabelConfig on the request to /-/reload

[z-anshun]

When I use vminsert's relabelConfig, I found that now there is no reloaded api. However, vminsert under VM-Single has it. So, I hope to add it to the cluster/vminster.

[hagen1778]

LGTM

[f41gh7]

LGTM

[z-anshun]

@valyala ,please help review the code.

[valyala]

The code looks good except one thing - usually the cluster contains multiple vminsert nodes hidden behind a load balancer. So the load balancer sends every request to /-/reload to different vminsert node. This may lead to inconsistent state at vminsert nodes, where different vmagent nodes have different relabeling configs. This adds confusion on how to properly reload relabeling configs at vminsert nodes. That's why the /-/reload endpoint wasn't added to vminsert. Right now it is possible to reload relabeling configs at vminsert nodes by sending SIGHUP signal to them.

So I'm not sure this is a good enhancement. Probably, it would be better adding an option such as -relabelConfigCheckInterval in the way similar to -promscrape.configCheckInterval at vmagent.

[z-anshun]

The code looks good except one thing - usually the cluster contains multiple vminsert nodes hidden behind a load balancer. So the load balancer sends every request to /-/reload to different vminsert node. This may lead to inconsistent state at vminsert nodes, where different vmagent nodes have different relabeling configs. This adds confusion on how to properly reload relabeling configs at vminsert nodes. That's why the /-/reload endpoint wasn't added to vminsert. Right now it is possible to reload relabeling configs at vminsert nodes by sending SIGHUP signal to them.

So I'm not sure this is a good enhancement. Probably, it would be better adding an option such as -relabelConfigCheckInterval in the way similar to -promscrape.configCheckInterval at vmagent.

@valyala very good. I found that vmagent has -promscrape.configCheckInterval and /-/reload, so I thought it would be better to add both of them to vminert.

[z-anshun]

@valyala ,please review

[Haleygo]

This may lead to inconsistent state at vminsert nodes, where different vmagent nodes have different relabeling configs.

@valyala I'm not quite understand your point here. As you said, component like vmagent or vmalert already have this /-/reload endpoint, all of them can get inconsistent state if user put load balancer in front of them and use it to perform reloading. So it's a generic wrong operation for reloading multiple instances.
The right move here should be putting a sidecar for each instance, like in k8s, it's common to having a config-reloader container as sidecar along with the service, so each instance will be updated when the config changed.
I think we should expose this /-/reload for vminsert, since it gives user an option to do so, instead of waiting for configCheckInterval.