Strategy for building up stats database as logs rotate

[hraftery]

I've been sifting through the various use cases and am trying to plan one with the following features:

• Report served from file by nginx on the same server as is being reported on.
• Not real-time because the extra complexity and item count limitations aren't worth having more frequent updates.
• Periodically updates the report with new log data.

This seems simple enough with a combination of cron, --restore and --persist. But the specifics of that behaviour are subtle, so I want to make sure I'm thinking about this correctly. Here's my plan:

1. One time goaccess all-the-historical-log-files --persist to seed the database with existing logs.
2. Periodic cron job that restores from the db and appends anything new in the current access.log. Eg: goaccess access.log --restore
3. Daily cron job that runs after logrotate, that adds yesterday's log to the db. Eg: goaccess access.log.1 --persist

I'm confused about the interaction of these restore/persist operations. I figure I can't just periodically goaccess access.log --persist --restore, because when the file rotates the entries since the last periodic run will be lost. But am I taking the wrong path by having one job that restores and one that persists?

[allinurl]

Hey Heath,

Great question. You should be able to run it exactly as you described with goaccess access.log --persist --restore. It will continue building on the existing database and shouldn't lose any previously recorded data. The --persist flag simply tells goaccess to save the processed data to disk, it doesn't mean it will only persist data from the current log file.

If you'd like to test it first, try it with a couple of small sample logs, such as sample.log.1 and sample.log.2, and verify that the data accumulates as expected.

Also, take a look at --keep-last=15 if you'd like to limit the data retention window to the last 15 days.

[hraftery]

Oh nice, thanks. My only concern with a periodic goaccess access.log --persist --restore is the small data loss during rotation. Assume log rotation happens at midnight, then you could have cron jobs run like:

• 11:59pm: goaccess access.log --persist --restore # all data up to 11:59pm persisted
• 12:00am: logrotate # access.log becomes access.log.1 and access.log is emptied
• 12:01am: goaccess access.log --persist --restore # all data since rotation is persisted

The data between 11:59pm and 12:00am is lost. I could try to coordinate a goaccess run with each logrotate event to minimise data loss, but I'd prefer to decouple those processes if I can.

I think you're saying I can add --persist to my original step 2. My concern was that when I do step 3 the db will already have most of those entries. So instead of trying to merge I'm better off just persisting a whole day at a time. Maybe you're saying the merge is safe, and doing more frequent persists makes each run of goaccess simpler?

[allinurl]

This depends on how logrotate is configured.

Default logrotate behavior: access.log is renamed to access.log.1, so access.log.1 keeps the original inode. In that case, this should work well:

goaccess access.log access.log.1 --persist --restore

GoAccess tracks files by inode, last parsed line, and timestamp, so it should skip already-processed lines in access.log.1 and pick up any gap automatically.

copytruncate mode: access.log is copied to access.log.1, then the original access.log is truncated in place. That means access.log keeps the same inode, which can be tricky for incremental processing. In that case, I'd avoid relying only on inode tracking and use a prerotate hook to flush the current log before rotation:

prerotate
    goaccess /var/log/nginx/access.log --persist --restore > /dev/null 2>&1
endscript

Then your cron can continue running against access.log as usual:

goaccess access.log --persist --restore

So the main thing is to check your logrotate config for copytruncate. If it's not using copytruncate, processing both access.log and access.log.1 is usually the safer setup. If it is using copytruncate, I'd prefer the prerotate approach.

I'd still recommend testing it with a small sample log and a forced logrotate run first. Create a 10-line log, process it with goaccess, rotate it, add another 10 lines, and run goaccess again to confirm it behaves as expected. I believe the above is correct, but I haven't personally tested that exact logrotate workflow recently.

Let me know if that helps

[hraftery]

Oh goaccess access.log access.log.1 --persist --restore is really neat!

I didn't want to have to rely on a particular logrotate configuration, but if I only need to ensure the common delaycompress and create behaviour is in effect, then that all-in-one goaccess command is looking pretty attractive.

The one big drawback for me of taking advantage of inode tracking, is that I was hoping to be able to pre-process the access.log. Something like grep "^marker" access.log | goaccess --persist --restore --db-path /var/goaccess/marker, but maybe that's more trouble than it's worth.

I'll look into this, thanks.

[hraftery]

I've run the test as you describe and the results look good. I've attempted to make the test broadly informative / reusable. See here: hraftery@a28a76c

So to summarise this discussion:

1. goaccess builds up its database just fine, without loss of data, even with periodic runs and rotating log files, provided: a) you specify both the current log file and the first rotated file; and b) the periodic runs are at least as often as the rotations. Details follow.
2. this built-in functionality is a neat solution, provided you don't need to pre-process the log files. If you do, further steps are required to ensure newly rotated entries are piped to goaccess in addition to periodically piping the current log file.

Example command to execute option 1 follows. It can be added to a cron job for periodic execution.

goaccess /var/log/nginx/access.log /var/log/nginx/access.log.1 \
         --log-format=COMBINED \
         --restore --persist --db-path="/path/to/your/db" \
         -o "report.html"

[allinurl]

Heath, this is great information, thanks for sharing it here, especially for highlighting those findings around pre-processing log files.