Persist watermark updated prematurely

[beorn7]

In https://github.com/prometheus/prometheus/blob/master/storage/local/storage.go#L1008 , the call to series.chunksToPersist() already updates the persist watermark. However, in the following, the series file write may fail (in case of a rewrite it will be all or nothing, in case of appending it might even partially fail), in which case the persist watermark needed to be adjusted (or better set to the appropriate value after successful write of the new series file).

Also, consider how to recover from a partial append.

[juliusv]

Regarding partial appends, we currently error when we see a file with wrong length (not a multiple of the expected chunk size), but we don't attempt to ever repair it.

[beorn7]

That might be because in that case, we can't be sure what happened anyway. But I think we should do something when encountering an error appending to a file.

Whatever, I will investigate.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.