Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

piping log into goaccess does not work correctly anymore #1708

Open
kochtopf opened this issue Mar 21, 2020 · 2 comments
Open

piping log into goaccess does not work correctly anymore #1708

kochtopf opened this issue Mar 21, 2020 · 2 comments

Comments

@kochtopf
Copy link

@kochtopf kochtopf commented Mar 21, 2020

just upgraded from Mar 12 -> Mar 21 and found the following...
cat test.log|goaccess -p go.cnf equals in Total requests count is higher than Valid requests
doing goaccess -f test.log -p go.cnf is still working OK (Total request = Valid requests). Looks like pipping stuff into goaccess does "eat" up / ignores some Valid requests.

BTW. The default config file also got removed somehow.
goaccess --dcf
No default config file found.
You may specify one with -p /path/goaccess.conf

@allinurl

This comment has been minimized.

Copy link
Owner

@allinurl allinurl commented Mar 25, 2020

Thanks for reporting this. Could you rebase and test it again? It was by default loading all data from disk.

Also, can you post the output of /usr/local/etc/goaccess?

@kochtopf

This comment has been minimized.

Copy link
Author

@kochtopf kochtopf commented Mar 25, 2020

diff goaccess.conf_OK goaccess.conf # this is what I'm using in ADDITION/OVERRIDE to the default goaccess.conf option file from make install
701,716d700
<
< time-format %T
< date-format %d/%b/%Y
< log-format '%^ %~%^ %^ %^ %^ %h %v %e [%d:%t %^] "%r" %s %b "%R" "%u" "%^" "%^" "%T" "%C"'
< sort-panel VISIT_TIMES,BY_DATA,DESC
< geoip-database /usr/local/etc/goaccess/GeoLite2-City.mmdb
< ignore-panel REMOTE_USER

to process like
Mar 25 18:55:06 srv3 nginx: xxx.xxx.xxx.xxx www.sample.me - [25/Mar/2020:18:55:06 +0100] "GET /url HTTP/1.1" 200 18934 "https://www.google.com/" "xxx" "XX" "0.000" "HIT"

cat test.log|goaccess -p /usr/local/etc/goaccess/goaccess.conf
Total Requests 22642 Unique Visitors 5779 Requested Files 9226 Referrers 0
Valid Requests 17114 Init. Proc. Time 2s Static Files 17 Log Size 0.0 B
Failed Requests 0 Excl. IP Hits 389 Not Found 28 Tx. Amount 244.16 MiB
Log Source STDIN

goaccess -f test.log -p /usr/local/etc/goaccess/goaccess.conf # this matches to build 186070c
Total Requests 22642 Unique Visitors 6201 Requested Files 11429 Referrers 0
Valid Requests 22077 Init. Proc. Time 3s Static Files 20 Log Size 11.23 MiB
Failed Requests 0 Excl. IP Hits 565 Not Found 37 Tx. Amount 334.51 MiB
Log Source test.log


done with af874a2

goaccess --dcf
No default config file found.
You may specify one with -p /path/goaccess.conf

done with 186070c

goaccess_OLDOK --dcf
/usr/local/etc/goaccess/goaccess.conf

PS: I do FLAGS="-O3" ./configure --enable-utf8 --enable-geoip=mmdb --with-getline
Looks like piping the SAME file gives different Valid Requests by a very large margin.
Thank you very much for reading.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.