Loss of data on restart

[kolesnikovae]

A restart may result in losing the head block:

Log from one of the ingesters (cluster of 4 ingesters with the default replication factor): apparently, shutdown didn’t finish gracefully:

I find it interesting that none of the ingesters has finished shutdown as well: no panics or explicit errors in the log – however, most likely this indicates a problem with log collection.

It seems that under some circumstances a block is not picked up after the restart, and the head is getting polluted with chunks:

/ $ ls -la /data/1218/head | wc -l
276
/ $ du -sh /data/1218/head
49.9G   /data/1218/head

[kolesnikovae]

Also see grafana/pyroscope#2076

[cyriltovena]

Possible we need to increase the grace period

    deployment.mixin.spec.template.spec.withTerminationGracePeriodSeconds(4800)

[kolesnikovae]

Termination grace period added in #837. Closing for now