[Filebeat] Update outdated info in the "Log rotation results in lost or duplicate events”

[vishaangelova]

The Log rotation results in lost or duplicate events doc is outdated:

Log rotation strategies that copy and truncate the input log file can result in Filebeat sending duplicate events. This happens because Filebeat identifies files by inode and device name. During log rotation, lines that Filebeat has already processed are moved to a new file. When Filebeat encounters the new file, it reads from the beginning because the previous state information (the offset and read timestamp) is associated with the inode and device name of the old file.

Starting from version 9.x, the default file_identity is fingerprint, so this statement is true only for older versions of Filebeat, or if the file_identity used is not the default one.

The document needs to be updated to reflect that the described common problem can affect versions of Filebeat <9.0.0 or versions 9.0.0+ if file_identity is not set to the default.

Resources

• Doc to be updated: Log rotation results in lost or duplicate events

• Related breaking change in 9.0.0:

  Filestream inputs now starts ingesting files only if they are 1024 bytes or larger because the default file identity has changed from native to fingerprint.

• file_identity documentation