Samples delivered out of order with remote write in Prometheus 2.6.0

[tomwilkie]

Just updated to prometheus v2.6.0 and getting a lot of out-of-order samples.

When I revert #4772 they go away...

[gouthamve]

*2.6.0 :)

[tomwilkie]

I've spent last hour trying to figure this out and I'm completely stuck...

[brian-brazil]

Now multiple enqueues can happen in parallel, whereas previously they were serialised?

[tomwilkie]

enqueue picks a queue (based on fingerprint of metric) and adds the sample to the queue, serialising requests.

It assumes that all the samples for a given timeseries are Appended, in order, by a single goroutine. Which I think is true as a single go routine scrapes each target, and blocks on calls to Append.

The queue is serviced by a single goroutine, which guarantees samples on the queue are delivered in order.

[brian-brazil]

Yeah, the code looks okay. Might you have multiple targets ingesting the same time series? I'm wondering if #4926 might be related.

[tomwilkie]

Might you have multiple targets ingesting the same time series?

Don't know yet, my prometheus is pretty unstable. Will get back to you.

[jojohappy]

@tomwilkie Any update?

[tomwilkie]

Fraid not; it looks like we might have some OOO samples pre 2.6 that I can't explain, and that time timing changed with the locking changes made them more visible. But not sure.

[tomwilkie]

Its looks like this was caused by our config: multiple exporters in a single pod all got the same target labels, and all exposed the go_runtime_* metrics.