Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance issues #1643

Open
ripperdoc opened this issue Dec 31, 2019 · 3 comments
Open

Performance issues #1643

ripperdoc opened this issue Dec 31, 2019 · 3 comments

Comments

@ripperdoc
Copy link

@ripperdoc ripperdoc commented Dec 31, 2019

Not sure if this is a bug but I am seeking some clarification. I'm trying to run goaccess on a 5GB large log file (1 year of traffic on a smallish site, approximately 23M hits, most of it crawlers of course). I struggle with performance.
On my local (old) MacBook Pro it's using one core fully and starts at 12000 lines/s but after a minute starts falling down to 3000. This is in-memory (when I ran using disk and left it over night and checked in the morning, it was down at 400 lines/s and still not finished). Command is:
goaccess -d -o report.html
On my server I tried running it in Docker, it did not fall down but instead Docker socket disconnects after a while:
read unix @->/var/run/docker.sock: read: connection reset by peer

Any clues or tricks I should think of? Can goaccess use more than one core? Is the slowdown I see normal? Have the docker issues been seen before?

@allinurl

This comment has been minimized.

Copy link
Owner

@allinurl allinurl commented Dec 31, 2019

I'm guessing the culprit is -d. Please try without resolving the IPs and let me know how it goes. For 23M hits, I'd still use the in-memory store.

BTW, what processor do you have and ram? Thanks

@ripperdoc

This comment has been minimized.

Copy link
Author

@ripperdoc ripperdoc commented Jan 2, 2020

I tested running it without -d. It has the same performance, starting at 12k slowing down to 2.5k/s. I have 2,6 GHz Intel Core i7 Macbook Pro.

@allinurl

This comment has been minimized.

Copy link
Owner

@allinurl allinurl commented Jan 3, 2020

To address your first question, currently there's no way to use more than one core. #377 will address this.

Are you able to test this out on a different Mac? I just want to know if this might be an isolated issue. Also, you could try building goaccess with -O3 and see if that helps, or even using ./configure --enable-debug to see if optimizations are not helping. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.