New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increased speed of loading syslog pages. #6433

Merged
merged 3 commits into from May 3, 2017

Conversation

Projects
None yet
6 participants
@TimStallard
Contributor

TimStallard commented Apr 15, 2017

When using the syslog functionality with many records (I tested with ~1.5 million), the UI becomes slow in several places:

  • Loading the main syslog page or the page for each device is very slow. This is due to the priorities dropdown, which is ordered by level. Adding an index on the levels and priorities combined resolves this.
  • Moving around the table for a single device is fairly slow, due to the count at the bottom being updated. A COUNT(timestamp) is currently performed, but there should not be any items with a null timestamp, so changing this to simply count rows using COUNT(*) significantly increases performance.
    These issues are still present when using additional filters, but I can't think of a nice way in which this would be resolved without adding many more combined indexes (or maybe one large index for device, program and priority?).

I agree to the conditions of the Contributor Agreement contained in doc/General/Contributing.md.

Please note

  • Have you signed the Contributors agreement - please do NOT submit a pull request unless you have (signing the agreement in the same pull request is fine). Your commit message for signing the agreement must appear as per the docs.
  • Have you followed our code guidelines?

Testers

If you would like to test this pull request then please run: ./scripts/github-apply <pr_id>, i.e ./scripts/github-apply 5926

Increased speed of loading syslog pages.
I agree to the conditions of the Contributor Agreement contained in doc/General/Contributing.md.
@mention-bot

This comment has been minimized.

Show comment
Hide comment
@mention-bot

mention-bot Apr 15, 2017

Thank you for submitting a PR @TimStallard! We have found the following @f0o, @geordish and @laf based on the history of these files to review this PR.

mention-bot commented Apr 15, 2017

Thank you for submitting a PR @TimStallard! We have found the following @f0o, @geordish and @laf based on the history of these files to review this PR.

@LibreNMS-CI

This comment has been minimized.

Show comment
Hide comment

LibreNMS-CI commented Apr 15, 2017

Auto-Deploy finished, Test PR at http://6433.ci.librenms.org or https://6433.ci.librenms.org

@laf laf added the Schema label Apr 17, 2017

@laf

This comment has been minimized.

Show comment
Hide comment
@laf

laf Apr 17, 2017

Member

@TimStallard How long did this index take to create for you?

Member

laf commented Apr 17, 2017

@TimStallard How long did this index take to create for you?

@laf laf added the Blocker 🚫 label Apr 17, 2017

@TimStallard

This comment has been minimized.

Show comment
Hide comment
@TimStallard

TimStallard Apr 17, 2017

Contributor

@laf I just ran a test on an instance with 1.76 million records, it took ~22.3 seconds to create. That's running inside an LXC container with two allocated cores and 2GB RAM.

I also ran the same test on a $40/mo digitalocean droplet (just running mariaDB with the database imported) and saw a similar time, ~23.8s.

Contributor

TimStallard commented Apr 17, 2017

@laf I just ran a test on an instance with 1.76 million records, it took ~22.3 seconds to create. That's running inside an LXC container with two allocated cores and 2GB RAM.

I also ran the same test on a $40/mo digitalocean droplet (just running mariaDB with the database imported) and saw a similar time, ~23.8s.

@laf

This comment has been minimized.

Show comment
Hide comment
@laf

laf Apr 27, 2017

Member

Waiting for #6469 then we can rebase and fix this.

@TimStallard We've gone live with a new CLA, can you please sign that one so we are able to merge.

Member

laf commented Apr 27, 2017

Waiting for #6469 then we can rebase and fix this.

@TimStallard We've gone live with a new CLA, can you please sign that one so we are able to merge.

@murrant murrant added this to the May 2017 Release milestone Apr 28, 2017

@laf

This comment has been minimized.

Show comment
Hide comment
@laf

laf May 3, 2017

Member

@TimStallard We're ready to merge this, can you please sign the new CLA please?

Member

laf commented May 3, 2017

@TimStallard We're ready to merge this, can you please sign the new CLA please?

@TimStallard

This comment has been minimized.

Show comment
Hide comment
@TimStallard

TimStallard May 3, 2017

Contributor

Sorry, I think something weird must have happened last time I tried. Should be signed now.

Contributor

TimStallard commented May 3, 2017

Sorry, I think something weird must have happened last time I tried. Should be signed now.

laf added some commits May 3, 2017

@LibreNMS-CI

This comment has been minimized.

Show comment
Hide comment

LibreNMS-CI commented May 3, 2017

Auto-Deploy finished, Test PR at http://6433.ci.librenms.org or https://6433.ci.librenms.org

@scrutinizer-notifier

This comment has been minimized.

Show comment
Hide comment
@scrutinizer-notifier

scrutinizer-notifier May 3, 2017

The inspection completed: No new issues

scrutinizer-notifier commented May 3, 2017

The inspection completed: No new issues

@laf laf merged commit 6b2ae00 into librenms:master May 3, 2017

3 checks passed

Auto-Deploy Build finished.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
license/cla Contributor License Agreement is signed.
Details
@laf

This comment has been minimized.

Show comment
Hide comment
@laf

laf May 3, 2017

Member

Thanks very much for contributing :)

Member

laf commented May 3, 2017

Thanks very much for contributing :)

@lock

This comment has been minimized.

Show comment
Hide comment
@lock

lock bot May 18, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed.

lock bot commented May 18, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed.

@lock lock bot locked as resolved and limited conversation to collaborators May 18, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.