scrape: Add global jitter for HA server

[xjewer]

Covers issue in #4926 (comment)
where the HA setup become a problem for targets unable to be scraped simultaneously.
The new jitter per server relies on a hostname and external labels which necessarily to be uniq.

As before, scrape offset will be calculated with regard the absolute time, so even
restart/reload doesn't change scrape time per scrape target + prometheus instance.

Jitter interval was chosen within the boundaries 0-59 seconds, which should be enough
to bring entropy for the scrapes across HA setup.

[brian-brazil]

It's be good to also include hostname, as that'll also catch test machines using the same config.

[xjewer]

It's be good to also include hostname, as that'll also catch test machines using the same config.

could you elaborate on it? what machines?

[brian-brazil]

could you elaborate on it? what machines?

Any other machine on which someone copies over a production config verbatim.

[xjewer]

someone copies over a production config

what about docker setup, for those who doesn't specify hostname it will move around the offset on re-runs. 🤔

[brian-brazil]

That's an acceptable tradeoff.

[xjewer]

updated with hostname, but smth went wrong with an import:

go: gopkg.in/vmihailenco/msgpack.v2@v2.9.1: unrecognized import path "gopkg.in/vmihailenco/msgpack.v2" (parse https://gopkg.in/vmihailenco/msgpack.v2?go-get=1: no go-import meta tags ())

could you restart the CI test build?

[xjewer]

I applied this patch to our 2.6.1 HA setup and it seems to work as expected:

[bwplotka]

@xjewer Nice! This graph you provide is for what?

[xjewer]

@xjewer Nice! This graph you provide is for what?

this one is for http probes

[xjewer]

fixed merge conflicts and remarks, ready for review

[xjewer]

any chance to be merged into next release? cc @brian-brazil

[brian-brazil]

2.8 is in progress, so it'll be 2.9 at this stage.

[brian-brazil]

We're trying to get rid of usage of this library.

[xjewer]

Any suggestions?

[xjewer]

should I use github.com/prometheus/prometheus/pkg/labels instead?

I can add this to GlobalConfig:

// GetExternalLabels wraps external labels as a labels.Labels.
func (c *GlobalConfig) GetExternalLabels() labels.Labels {
	labelSet := make(map[string]string, len(c.ExternalLabels))
	for n, v := range c.ExternalLabels {
		labelSet[string(n)] = string(v)
	}

	return labels.FromMap(labelSet)
}

but as this isn't related to the Jitter, let's leave it as is and do clean-up in follow up PRs, I can help you out with it.

Didn't find any issue regarding liquidation the github.com/prometheus/common/model library.

[brian-brazil]

model.Labelset is already a map[string]string, so you can use FromMap directly. You can also re-use its Hash method.

[xjewer]

the problem is, the model.Labelset isn't a map[string]string, it's map[LabelName]LabelValue even if LabelName and LabelValue are strings, so it's not the same and golang can't do type conversion.

[brian-brazil]

You can cast it

[xjewer]

what am I doing wrong?
labelSet := map[string]string(cfg.GlobalConfig.ExternalLabels)

cannot convert cfg.GlobalConfig.ExternalLabels (type model.LabelSet) to type map[string]string

https://play.golang.org/p/EwQjDtB00dJ

prog.go:13:23: cannot convert ls2 (type LabelSet2) to type map[string]string

where as LabelSet works as mentioned before.

[brian-brazil]

Right you are.

[brian-brazil]

Thanks!