Remote Write: Add max samples per metadata send

[LeviHarrison]

This PR adds the max_samples_per_send option under the metadata_config.

Fixes #8870

[cstyan]

Looking good, just a few small comments.

[cstyan]

I think you could just assign i := math.Ceil(len(metadata)/t.mcfg.MaxSamplesPerSend)) and then keep track of the start and end indexes into the slice for deciding what samples to send? Or something like that. Then we don't need the if statement you have below to check if there were remaining samples.

[LeviHarrison]

keep track of the start and end indexes into the slice for deciding what samples to send

I'm not sure what you mean here.

Also, math.Ceil isn't needed because the decimals are implicitly truncated when assigning to an int (and we don't want to round, just truncate).

[cstyan]

I'm saying we should maintain the decimal places (cast one of the two ints to a float) so that we can round up if len(metadata) is not evenly divisible by MaxSamplesPerSend. Either that or you could use len(metadata) % MaxSamplesPerSend and increment i if the remainder was not 0.

keep track of the start and end indexes into the slice for deciding what samples to send

I meant you'd need to a way to track what indexes within the metadata slice are a part of the next metadata send, just so that on the last send we index slice[beginning of last send:end of slice] instead of slice[beginning of last send:beginning plus MaxSamplesPerSend]

Either way, this would get rid of the need for the if condition at the end that checks if we need to do one last send.

[LeviHarrison]

Hows this:

for i = 0; i < int(math.Ceil(float64(len(metadata))/float64(t.mcfg.MaxSamplesPerSend)); i++ {
                last := (i+1)*t.mcfg.MaxSamplesPerSend
                if last > len(metadata) {
                        last = len(metadata)
                }
		err := t.sendMetadataWithBackoff(ctx, mm[i*t.mcfg.MaxSamplesPerSend:last])
		if err != nil {
			t.metrics.failedMetadataTotal.Add(float64(t.mcfg.MaxSamplesPerSend))
			level.Error(t.logger).Log("msg", "non-recoverable error while sending metadata", "count", t.mcfg.MaxSamplesPerSend, "err", err)
		}
}

[cstyan]

Personally I would put the math.Ceil calculation on it's own line with something like numSends := math.Ceil(...) but this looks good 👍

[cstyan]

can we give this a better name? expectedNumWrites perhaps?

[LeviHarrison]

I don't think expectedNumWrites is the proper term, because it's not a specified expectation, it's how many are counted.

How about writesReceived?

[cstyan]

yep, sounds good 👍

[csmarchbanks]

👍

[cstyan]

Thanks @LeviHarrison

[LeviHarrison]

Thanks for your time reviewing!

[bboreham]

Why is it named "max_samples" when it's sending metadata, not samples?

[LeviHarrison]

I guess because it's a subfield of metadata_config, so samples refer to samples of metadata. But I didn't come up with the name so...