Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
Filebeat 1.1.2 does not record correct inode values of rotated logs on exit #1281
I am seeing filebeat re-ship rotated log files when the filebeat service restarts.
We are experiencing the problem on a local ext4 file system on Ubuntu 14.04 running in AWS EC2.
While filebeat remains running, it is correctly processing the files. The problem is that when it persists its state to the registry file, it is persisting the older inode values of the files before they were rotated, not the inodes of the files at those paths at the time filebeat exited.
I have created a basic configuration and bash script which starts filebeat, writes to a log file, rotates it a couple of times (with pauses in between to allow filebeat to notice) and then stops filebeat.
Full repro script, conf, and output log available as a gist:
After filebeat stops, the registry has persisted the wrong inodes for the log files present on disk. As such, when filebeat starts again, an existing log file will appear to be a new file without an existing offset and its contents will be shipped a second time.
My repro script was run on a local ext4 file system on Ubuntu 14.04 running in a Virtualbox VM.
This might be an edge case not handled in #1010
referenced this issue
Apr 12, 2016
@jstangroome I think I found a solution (see #1375). The problem occured when multiple files were rotated at the same time and it was written to the first file before the rotation was detected. Can you test if master works now for you?
@tsg I tried to find a fix that is minimal and backportable. Should we backport it?