Race Condition May Result in Data Loss During Log Rotation

[zwass]

This is a theorized issue that has not been demonstrated in the real-world. I am reporting this because it seems like a potential problem when reading the code.

There is a potential race condition in which result logs may be lost when log rotation is performed by an external tool (the recommended strategy). See #302 for some context.

Take the following ordering of events:

1. Osquery opens the log file to write results.

2. An external log rotation process rotates the log file.

3. Osquery writes results to the fd of the old log file (now defunct due to the log rotation performed externally).

In this scenario, I believe that the final set of logs may be permanently lost. This could depend on the exact behavior of the log rotation tool.

[directionless]

AFAIK, under posix that is the standard pattern, and it does not result in stale filehandle. It is, specifically:

1. Program opens a log file
2. Program happily writes data
3. Log rotation tool moves file aside
   a. file handles remain valid
   b. This is why rm won't recover disk space
4. log rotation tool sents a some signal to program, so it re-opens the logfile
5. time passes
6. old log file is compressed

Reading #302, the comment is that osquery does a continue close/open cycle. I'm not sure how you'd get to data loss. Maybe if logrotate did a mv, osquery created a new file, then logrotate did a rm. But that seems weird.

[zwass]

This would all work fine if logrotate essentially mvs the file; the file descriptor will remain valid. How does a tool like logrotate work if compression is turned on? Is it possible that osquery would be writing to the old fd after the existing contents have been compressed?

[directionless]

That's the purpose of the time in step 5 above. Common settings are to keep 2ish days uncompressed, and 7-30 days compressed.

This pattern hinges on the ability of logrotate to signal the daemon to re-open the filehandle. (Superfluous, if osquery re-opens every write)

[zwass]

Nice, okay, thank you for filling me in on this. I think it is a non-issue.