Mangled positions results in promtail crash loop (k8s daemonset)

[owen-d]

What

If the positions file is corrupted/mangled somehow, it causes a promtail daemon to continually fail. Because it's mounted as a hostpath, it persists across both restarts and pod deletions/recreations on the same host and requires manual intervention to rectify. Perhaps this is desirable?

Ideas

• delete a corrupted file, forcing all positions to be recreated/reset? We'd see out of order errors, but would these eventually catch up?
• Leave as it is?
• Flag to determine which strategy to use?

credit @pracucci :
An easy solution to avoid truncated files could be writing to a temporary file and then use a move syscall to move the file to the expected location (the move is atomic). This also allows you to verify the file written in tmp location before moving it.

/cc @slim-bean @rfratto

Origin

#1457

[owen-d]

It looks like we made writes to the positions file atomic on November 21 2019: fd25e6d.