Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Time Served columns are missing when piping messages from Java #1648

Open
MosheElisha opened this issue Jan 12, 2020 · 1 comment
Open

Time Served columns are missing when piping messages from Java #1648

MosheElisha opened this issue Jan 12, 2020 · 1 comment

Comments

@MosheElisha
Copy link

@MosheElisha MosheElisha commented Jan 12, 2020

Hi,

I have a Java process that reads messages from Elasticsearch and pipes them to goaccess.

If I use the --real-time-html option when I start goaccess process from within Java, the Time Served columns (Avg, Max, Cum) are missing (attached real-time-report-without-ts).

This is the command I use from within Java:

/usr/local/bin/goaccess --output /opt/aua/goaccess/index.html --time-format '%T' --date-format '%d/%b/%Y' --log-format '%h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" "%^" %T %^ %^ %^' --real-time-html --fifo-in /tmp/goaccess.in --fifo-out /tmp/goaccess.out --invalid-requests /tmp/goaccess.invalid

If I don't use the --real-time-html option when starting goaccess, the T.S. columns are generated as expected (attached static-report-with-ts).

/usr/local/bin/goaccess --output /opt/aua/goaccess/index.html --time-format '%T' --date-format '%d/%b/%Y' --log-format '%h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" "%^" %T %^ %^ %^'

If I change my Java program to write the messages into a FileOutputStream instead of the Process.getOutputStream() and than use the file generated with --real-time-html everything works as expected (attached real-time-report-with-ts):

tail -F /tmp/messages.log | /usr/local/bin/goaccess --output /var/www/html/report.html --time-format '%T' --date-format '%d/%b/%Y' --log-format '%h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" "%^" %T %^ %^ %^' --real-time-html --fifo-in /tmp/goaccess.in --fifo-out /tmp/goaccess.out --invalid-requests /tmp/goaccess.invalid

I feel this is some buffering issue (something like grep --line-buffered) but I'm not sure what exactly. In my code I do the following for every log message:

byte[] messageBytes = (msgStr + System.lineSeparator()).getBytes();
processOutputStream.write(messageBytes);
processOutputStream.flush();

I configured goaccess with --with-getline thinking maybe it will make a difference but it didn't.
I also tried to configure with --enable-debug and run goaccess with --debug-file - the debug log file was created but it was empty.

BTW the https://goaccess.io/man mentions the option "--log-debug" but seems that the actual option is "--debug-file".

goaccess-reports.zip

Can you please suggest what I should check?

My setup:

$ goaccess --version
GoAccess - 1.3.
For more details visit: http://goaccess.io
Copyright (C) 2009-2016 by Gerardo Orellana

Build configure arguments:
  --enable-debug
  --enable-utf8
  --with-openssl
$ cat /etc/centos-release
CentOS Linux release 7.7.1908 (Core)

Thanks.

@allinurl

This comment has been minimized.

Copy link
Owner

@allinurl allinurl commented Jan 17, 2020

Thanks for pointing out the --log-debug, I'll have this fixed.

Are you able to run your code through gdb? or perhaps strace just to see if TS is actually being logged and how goaccess is reading it. let me know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.