Remote storage queue's concurrent sends cause out-of-order samples

[juliusv]

The remote storage queue manager fires off concurrent requests to send sample batches to the remote storage:

prometheus/storage/remote/queue_manager.go

Lines 183 to 204 in e980913

	go func() {
	defer func() {
	<-t.sendSemaphore
	}()
	// Samples are sent to the remote storage on a best-effort basis. If a
	// sample isn't sent correctly the first time, it's simply dropped on the
	// floor.
	begin := time.Now()
	err := t.tsdb.Store(s)
	duration := time.Since(begin).Seconds()
	labelValue := success
	if err != nil {
	log.Warnf("error sending %d samples to remote storage: %s", len(s), err)
	labelValue = failure
	t.failedBatches.Inc()
	t.failedSamples.Add(float64(len(s)))
	}
	t.samplesCount.WithLabelValues(labelValue).Add(float64(len(s)))
	t.sendLatency.Observe(duration)
	}()

This causes samples to be sent out-of-order, which the receiving side might not be equipped to handle. Not sure what's going to happen with this interface in general, but if it's generally going to stay, we should consider removing the parallelism (or make it configurable in case people have a backend that doesn't care?).

[brian-brazil]

I think the other end will need to be able to deal with this. I'm expecting parallelism to be required to have the required throughput.

[tomwilkie]

We could have both parallelism and in order delivery. For instance, we could shard by metric name (or fingerprint...) and then deliver in parallel.

[brian-brazil]

To get to the point where you could consider not having parallelism would require major design first (as in we'd need to completely flesh out this interface), so I think that's very much out of scope for the medium term anyway.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.