Prometheus not deleting data several hours passed retention time

[kinghrothgar]

What did you do?
Set Prometheus --storage.tsdb.retention=24h

What did you expect to see?
Data be deleted after 24h

What did you see instead? Under which circumstances?
Data over 31 hours old is being retained. I have restarted Prometheus several times as this was the only thing I could think of to try and trigger a cleanup.

• Prometheus version:

2.0.0

The reason this is problematic is I'm transitioning to Prometheus 2. I started recording with Prometheus 2 several days before stopping Prometheus 1.8.2. This resulted in duplicate data that messes up my queries. I set the retention shorter and monitored it hoping to restart Prometheus 2 with a longer retention soon as it deletes just past what is in 1.8.2. I don't know how else I could have fixed this problem as I am not aware of any way to manipulate data that is already ingested into Prometheus.

[kinghrothgar]

It finally caught up suddenly but data was around 12 hours past the retention before it finally deleted it. I'm not sure what triggers cleanup and if this is expected.

[kinghrothgar]

What info is needed? I should be able to replicate this on a test box if specific data is needed.

[gouthamve]

Was this when Prometheus was started with 24hr from the beginning itself? Or was prometheus started with 15days and was restarted with 24hrs?

We delete one block at a time and the max block duration is usually 10% of the retention. So if prometheus was initially started with 15days, then the blocks could get large enough.

[kinghrothgar]

I'm pretty sure it was previously set higher than 24hrs so that's probably what happened. It sucks that there wasn't a better solution for this but it also seems like an edge case.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.