File checkpointing

[binarylogic]

The ability to persist a file's current read position and resume where left off between restarts.

One approach, which is what Timber used in it's old agent, is to take the first 256 bytes of a file and produce a checksum / finger print as well as the position. The checksum is used to identify the file if it is renamed.

https://github.com/timberio/agent/blob/004e495f9e6b21c3ca2b9cd516e3e498ddec40dc/tailer.go#L144-L162

Open to alternative solutions. Requirements are:

• Vector can be restarted and resumed, starting where it left off
• Vector can be restarted and resumed, even if the file has been rotated / renamed. We can use the checksum to identify the file. This, of course, requires the user to configure a glob pattern that would fine the renamed file.
• Vector can be restarted, the file can be truncated, and Vector will start reading from the beginning.

[binarylogic]

Just noting, this is not something we should do for the 0.1 release, just adding this issue for future reference.