Fingerprint collision handling is O(n^2) in the number of mappings

[brian-brazil]

The fingerprint mappings write out a checkpoint at each mapping, syncing it to disk (

prometheus/storage/local/persistence.go

Line 1352 in e83f05f

func (p *persistence) checkpointFPMappings(fpm fpMappings) (err error) {

). This is O(n^2).

This is done while the raw fingerprint is locked (

prometheus/storage/local/storage.go

Line 589 in e83f05f

s.fpLocker.Lock(rawFP)

), so a scrape of an existing timeseries which is now colliding is now blocked waiting for the checkpoint to be written. Each sample from a scrape is processed sequentially, so this can block many scrapes.

If you've many new collisions, this can block scraping for quite a while. The example I'm looking at seems to have tens of thousands of collisions which took an hour to work through before scraping worked again.

I recommend switching to appending new mappings rather than writing out a whole new file. We should also look at coalescing the write() and syncs() together, otherwise we'll be bottlenecked there too.

[beorn7]

oops
the code is definitely designed assuming collisions are rare.

[brian-brazil]

This case was with synthetic data for a loadtest, so more collisions and at a higher rate than usual.

We might away with just the coalescing, which'd also avoid a data format change.

[brian-brazil]

Ah, I'd missed how the fingerprintlocker is implemented. This will hold up all scraping as that works with locks for 1024 buckets rather than one per timeseries.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.