Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filebeat file rotation problem #1010

Closed
smaniero opened this issue Feb 19, 2016 · 4 comments

Comments

Projects
None yet
3 participants
@smaniero
Copy link

commented Feb 19, 2016

I test file rotation feature of filebeat, and I can't understand how it work.

My config file is:

filebeat:
  prospectors:
    -
      paths:
        - "/var/log/test/*.log"
  registry_file: "registry"

output:
  console:
    pretty: true

Create log file with commands:

echo START > logfile.log
echo ROW 1 >> logfile.log
echo ROW 2 >> logfile.log

Registry file content:
{"/var/log/test/logfile.log":{"source":"/var/log/test/logfile.log","offset":18,"FileStateOS":{"inode":1230350,"device":2065}}}

Rotate log file with command:
mv /var/log/test/logfile.log /var/log/test/logfile.1.log

Repeat command:

echo START > logfile.log
echo ROW 3 >> logfile.log
echo ROW 4 >> logfile.log
echo ROW 5 >> logfile.log

Registry file content:
{"/var/log/test/logfile.log":{"source":"/var/log/test/logfile.log","offset":24,"FileStateOS":{"inode":1230350,"device":2065}}}

After rotation of log file with command "mv", in registry file there's no entry for file "logfile.1.log", and when append lines to logfile.log or logfile1.log in registry single entry offset info change in both cases.

When restart filebeat command, in stdout replay all entries of logfile.1.log and I have duplicated log entry in my repository.

How to solve the problem?
My configuration or use of filebeat is wrong?

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Feb 22, 2016

Which version of filebeat are you using?

@smaniero

This comment has been minimized.

Copy link
Author

commented Feb 22, 2016

Beats 1.1 build on Linux 64bit

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Feb 29, 2016

@smaniero Thanks a lot for the detailed report and sort for the late answer. Unfortunately I was able to reproduce it and it is indeed a bug. Here is the fix for it: #1060

ruflin added a commit to ruflin/beats that referenced this issue Feb 29, 2016

ruflin added a commit to ruflin/beats that referenced this issue Feb 29, 2016

ruflin added a commit to ruflin/beats that referenced this issue Feb 29, 2016

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Feb 29, 2016

Backport fix for 1.1 can be found here: #1061

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.