Prom2: crash on opening WAL block

[mwitkow]

What did you do?
Ran latest dev-2.0 branch.
What did you expect to see?

Crash looping restart:

2017-06-01T19:19:18.768587000Z time="2017-06-01T19:19:18Z" level=error msg="Opening storage failed: open block /prometheus-data/01BGVHR6TMNFQHTSPK67E9K62W: open head block /prometheus-data/01BGVHR6TMNFQHTSPK67E9K62W: unknown series reference 40611 (max 40591); abort WAL restore" source="main.go:89" 

@fabxc for reference :)

[fabxc]

Seen this a couple of times myself. Not sure where it's coming from yet.

[gouthamve]

@mwitkow Do happen to have the offending WAL file / data dir with you. I haven't been able to reproduce this for sometime now.

[gouthamve]

One cause for this is a race-condition, but I am not sure how the usage inside Prometheus could ever cause it:

Suppose we have 100 existing series inside a HeadBlock. Now we open two appenders in two routines A1, A2 and append 30 new series and 60 new series respectively with some common series.

Both try to commit at the same time and the following happens in the given order:

A2 executes createSeries()
A1 executes createSeries() (with its common series referencing the ids from A2's creations)
A1 persists its newlabels, samples
A2 persists its newlabels, samples

Now when reading it back, we read A1's samples which reference A2's id and thereby fail.

But we never have 2 appenders with the same series in Prometheus. Could this be due to staleness changes @fabxc?

[brian-brazil]

But we never have 2 appenders with the same series in Prometheus.

This can and does happen.

[fabxc]

This hasn't been spotted again in a while. There's a good chance this was fixed along the way. Should we close until it reappears?

…

On Fri, Jul 14, 2017, 3:46 PM Brian Brazil ***@***.***> wrote: But we never have 2 appenders with the same series in Prometheus. This can and does happen. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2795 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEuA8vhB3zWjY3QoX4ZdElPr6G_cLkJXks5sN3EagaJpZM4NtbYj> .

[brian-brazil]

sgtm

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.