Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

log rotation silently fails at random intervals #828

Open
klehigh opened this issue Feb 27, 2020 · 1 comment
Open

log rotation silently fails at random intervals #828

klehigh opened this issue Feb 27, 2020 · 1 comment

Comments

@klehigh
Copy link

@klehigh klehigh commented Feb 27, 2020

I see logs get renamed to -YYYY-MM-DD-HH-MM-SS.log , but they don't get compressed and moved to the archive directory. This happens at random intervals and is reasonably well distributed across all the logs we're writing. There are no log rotation related errors in reporter.log or std*.log .

I'm running on FreeBSD and this problem is present in 3.0.1. It has, admittedly, been around for a very long time.

This issue was brought up in a Zeeksperts call recently, so it isn't specific to our site. Pinging @sethhall as he and I have discussed this and he had some thoughts on a possible cause.

@JustinAzoff

This comment has been minimized.

Copy link
Contributor

@JustinAzoff JustinAzoff commented Mar 1, 2020

I think part of the reason there's no errors logged is because they are ignored

Log::run_rotation_postprocessor_cmd does

    system(fmt("%s %s %s %s %s %d %s",
               pp_cmd, npath, info$path,
               strftime("%y-%m-%d_%H.%M.%S", info$open),
               strftime("%y-%m-%d_%H.%M.%S", info$close),
               info$terminating, writer));

    return T;

and throws away the result of system.

LogAscii::default_rotation_postprocessor_func does the same thing.

We should be able to replace

system(fmt("/bin/mv %s %s", info$fname, dst));

with

rename(info$fname, dst);

In general the approach https://github.com/ncsa/bro-atomic-rotate uses is the right one and fixes almost all of the log rotation issues, but if the system() call is failing it won't help there. It would be possible to avoid the system() part entirely using a new default_rotation_postprocessors.

Integrating the atomic rotation ideas might be finally doable with the new supervisor framework.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.