Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize log page render time #1

Closed
mindphluxnet opened this issue Mar 6, 2017 · 1 comment
Closed

Optimize log page render time #1

mindphluxnet opened this issue Mar 6, 2017 · 1 comment
Assignees

Comments

@mindphluxnet
Copy link
Owner

Currently, every IP address in a log is checked against the database every time a log page is rendered. If its not in the database, it'll be fetched from ipapi.co. This needs to be optimized since right now it's taking ages (about a minute on my Atom powered machine).

Ideas to test:

  • put all IPs in a dictionary and use this as lookup table. If an IP is already in there it will no longer be checked. I've tested an implementation of this idea but it didn't improve loading times, probably because it's still hundreds, if not thousands of IPs that need to be checked per log file.

  • constantly parse logs in the background and add IPs to the database so loading isn't stalled by the ipapi.co lookup. Should improve things, but even a log with already known IPs loads rather slowly.

@mindphluxnet
Copy link
Owner Author

mindphluxnet commented Mar 9, 2017

Greatly improved by switching to a local MaxMind GeoIP database while also committing to sqlite just once per parse run.

506b7f9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant