Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

filebeat 1.2.2 reships rotated log files upon restart due to lost offset #1528

Closed
jstangroome opened this issue Apr 29, 2016 · 4 comments

Comments

Projects
None yet
2 participants
@jstangroome
Copy link

commented Apr 29, 2016

We just updated from filebeat 1.1.2 to 1.2.2 to resolve #1281 in our environment. This has revealed a new issue.

We are experiencing the problem on a local ext4 file system on Ubuntu 14.04 running in AWS EC2 and also running in Virtualbox 5.

I have created a bash script which reproduces the issue and published it as a gist, including a log of the output:
https://gist.github.com/jstangroome/0774df94b46aa288b0855f533c67d152

Essentially when filebeat exits, the registry is recorded with the entry for the fully-shipped and recently-rotated log file not having an offset specified.

When filebeat is started again, the missing offset value is treated effectively the same as "offset": 0 and the log file's contents are reshipped. Because the file was rotated recently, it is still new enough that ignore_older directives won't exclude it.

/cc @ruflin

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Apr 29, 2016

@jstangroome Thanks for the report. I will have a look and hopefully be able to provide a fix.

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Apr 29, 2016

I created a test case for the issue here: #1532 Hopefully I can provide a fix soon.

ruflin added a commit to ruflin/beats that referenced this issue Apr 29, 2016

Fix rotation issue with ignore_older elastic#1528
Following changes were made
* All state information is fetched in advance for later comparison to prevent race conditions
* State is also written to registrar file if offset is 0, same for path
* Test case to validate fix added
@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Apr 29, 2016

@jstangroome I think I found a fix in #1532

ruflin added a commit to ruflin/beats that referenced this issue May 2, 2016

Fix rotation issue with ignore_older elastic#1528
Following changes were made
* All state information is fetched in advance for later comparison to prevent race conditions
* State is also written to registrar file if offset is 0, same for path
* Test case to validate fix added

ruflin added a commit to ruflin/beats that referenced this issue May 2, 2016

Fix rotation issue with ignore_older elastic#1528
Following changes were made
* All state information is fetched in advance for later comparison to prevent race conditions
* State is also written to registrar file if offset is 0, same for path
* Test case to validate fix added

tsg added a commit that referenced this issue May 2, 2016

Fix rotation issue with ignore_older #1528 (#1532)
Following changes were made
* All state information is fetched in advance for later comparison to prevent race conditions
* State is also written to registrar file if offset is 0, same for path
* Test case to validate fix added
@ruflin

This comment has been minimized.

Copy link
Collaborator

commented May 3, 2016

Closing as #1532 was merged. This fix will also be forwardported to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.