New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

real-time with multiple log #600

Closed
tbarbette opened this Issue Dec 23, 2016 · 3 comments

Comments

Projects
None yet
2 participants
@tbarbette
Copy link

tbarbette commented Dec 23, 2016

Hi all,

Is there a way to have the real-time mode with a log in multiple files (log rotation) like access.log.0, access.log.1, ...? I use nginx, but it is not specific to it and should be a common problem.

It is easy without real-time as we can just aggregate all log and run the program on it.

The problem is that if I just use -f access.log.0 I do not have my old data (and not sure that when the log rotation is done it will keep the current data?). Or if I restart goaccess I would loose the old data.

One way I found was to load the aggregation of the old files with --keep-db, and then start goaccess with --read-from-disk and --keep-db but if I restart it multiple times the same day, the data from acess.log.0 is -re-read and kept multiple times. So still not a solution.

Another way would be to have an external program always running which takes all old files and tail the current one. But it seems a little over-kill.

Thanks,
Tom

@allinurl

This comment has been minimized.

Copy link
Owner

allinurl commented Dec 24, 2016

Currently there's no way to have the real-time functionality when piping multiple log files. This will be addressed in #428 and #459. As you said, a workaround would be to tail multiple logs and append them to a new log file and feed that to GoAccess.

However, just to clarify log rotation, GoAccess will display the the contents of the new log file as in the case of tail -F.

@tbarbette

This comment has been minimized.

Copy link
Author

tbarbette commented Dec 26, 2016

Ok ! Waiting for it.

For people with the same problem, I found a workaround in the mean time. My service script contains a first line for parsing the old files and save it to the database, and the second launches goaccess in real time :

zcat /var/log/nginx/access.log*gz | goaccess -p /etc/goaccess.conf -a -o /dev/null --keep-db-files
goaccess -p /etc/goaccess.conf -f /var/log/nginx/access.log --real-time-html -a -o /usr/share/nginx/html/goaccess.html --load-from-disk

With this method I can restart the service (or reboot) without double parsing and the real-time is working with the old data too.

@allinurl

This comment has been minimized.

Copy link
Owner

allinurl commented Feb 27, 2017

I've pushed a commit that enables the ability to output live stats from a stream or an unclosed STDIN, i.e., tail -f as well as parsing multiple logs in live mode:

goaccess --log-format=COMBINED access.log access.log.1

or

tail -f access.log | goaccess --log-format=COMBINED -

or even

tail -f access.log | goaccess --log-format=COMBINED -o report.html --real-time-html -

It also opens the possibility for live data filtering from the pipe such as:

tail -f access.log | grep -i --line-buffered 'firefox' | goaccess --log-format=COMBINED -

One thing to note is that tail -f will keep the pipe opened even when goaccess has already exited. For instance, tail -f syslog | grep -q 'cron' tail will exit on SIGPIPE, however it will only get SIGPIPE until an extra byte is written to the tail'd file. SIGTERM, SIGINT should close tail fine though.

Feel free to build from master to test this out, otherwise it will be pushed out in the upcoming release. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment