Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Very slow when querying large amounts of data #138

Open
forever765 opened this issue Oct 26, 2019 · 15 comments
Open

Very slow when querying large amounts of data #138

forever765 opened this issue Oct 26, 2019 · 15 comments
Labels

Comments

@forever765
Copy link

@forever765 forever765 commented Oct 26, 2019

Grafana Version: 6.4.2
Grafana-PRTG Plugin Version: 4.0

Hi there,

The grafana-prtg plugin is very slow when querying large amounts of xml data. For example, I have a dashboard that shows the NIC traffic graph. This dashboard contains 2 grafana graph panels. Each panel contains 2 channels (inbound and outbound traffic).

The time range of the dashboard is "last 6 hours". When I refresh the dashboard, the first panel takes about 10 seconds to display, and the two panels fully display about 20s. This is really unbearable.

I use the Query Inspector of the grafana panel to check. In most cases, I can see the results of the query, but after a few seconds, chrome will prompt for a crash. The recurrence rate of chrome crash is 100%.

Using the same request as the grafana-prtg plugin, I found that the single channel xml file returned by prtg is 750KB (about 11,000 lines), because each panel has two channels, so a panel needs to process about 1.5MB of xml data. PRTG is set to collect data once in 30 seconds, so the amount of data in 6 hours is very large.

I want to know what the root cause of the problem is, and what can be done to increase the loading speed? In addition, is the grafana-prtg plugin sending and processing requests concurrent?

At last, I am very glad to see that the author is still maintaining this project, and I sincerely thank you for your contribution.

@neuralfraud

This comment has been minimized.

Copy link
Owner

@neuralfraud neuralfraud commented Oct 26, 2019

I was too lazy to try and parse XML nodes so I used XSLT to convert it into JSON. That is somewhat slow, but I've never seen a dashboard load that slowly. What is your query? click the query inspector button on one of those queries and paste in a sample of the data.. something seems awfully strange with the amount of data you say you're getting back considering that's only 720 datapoints.

@forever765

This comment has been minimized.

Copy link
Author

@forever765 forever765 commented Oct 27, 2019

These are the eth0 NIC traffic data of a Linux device. I guess the reason for so much data is because I set the device scan time of this linux device to 30 seconds.

The xml file is large, I uploaded it to Google Drive. Please check, thank you!
https://drive.google.com/open?id=17GzQydU9osGaSGYfAVQcwNBJV_EAHpia

@neuralfraud

This comment has been minimized.

Copy link
Owner

@neuralfraud neuralfraud commented Oct 27, 2019

Just pushed a new build, ditched the XML junk, can you try that new version? it's in the master branch.

@forever765

This comment has been minimized.

Copy link
Author

@forever765 forever765 commented Oct 27, 2019

I tried a new version of the master branch. When I clicked on the Query Inspector, chrome no longer crashes, even getting 10,000 objects is no problem.
But what makes me strange is that although the query inspector of metrics has data, there is no data on the panel, and all the chart panels of all the dashboards are like this.
picture link : https://drive.google.com/open?id=1yVurPK_4Fv-yfLPHeNx4eWxNw9FAsV-z

sorry...I mistakenly closed the issuse.

@olsonnn

This comment has been minimized.

Copy link

@olsonnn olsonnn commented Oct 27, 2019

CPU usage on my grafana test instance dropped a lot after this build. Good job!

@forever765 forever765 closed this Oct 27, 2019
@forever765

This comment has been minimized.

Copy link
Author

@forever765 forever765 commented Oct 27, 2019

After the plugin is updated to the latest version, the datasource cannot be added. It does not work according to the original configuration. The prompt "Unexpected error". I tried it many times, and changed the account number as well, until the PRTG prompts "overload protection".

@neuralfraud neuralfraud reopened this Oct 27, 2019
@neuralfraud

This comment has been minimized.

Copy link
Owner

@neuralfraud neuralfraud commented Oct 27, 2019

@forever765 New build.. Try it and let me know

I had no idea PRTG API performed language translation. That was kind of silly.

You may have to update your dashboard queries for any panel that is looking at a traffic/bandwidth channel (e.g. Traffic In)

you will now notice that the query builder shows the channels with the volume and speed. You can still use Regex style matches, e.g. /Traffic/ (or the Chinese equivalent) or simply select them individually.

@neuralfraud

This comment has been minimized.

Copy link
Owner

@neuralfraud neuralfraud commented Oct 27, 2019

After the plugin is updated to the latest version, the datasource cannot be added. It does not work according to the original configuration. The prompt "Unexpected error". I tried it many times, and changed the account number as well, until the PRTG prompts "overload protection".

This may have been caused by me trying to use the secure storage for the password - i have reverted that change since it was causing problems and it is poorly documented.

@forever765

This comment has been minimized.

Copy link
Author

@forever765 forever765 commented Oct 28, 2019

@neuralfraud Thanks for your help, I tested it with the latest version of the plugin and it worked!

Latest test situation: The first time you open the dashboard, the traffic panel loading takes about 10s (unlike before, the new version of multiple panels is almost loaded at the same time), grafana performs the dashboard auto refresh, the panel only needs 1~2s to load it! This is a huge leap!

But when I tested the addition of the prtg data source again, grafana only prompted "datasource updated" and did not display "PRTG API version: 19.4.52.3515+". This new datasource is indeed unavailable. But fortunately, the datasource built by the old version of the plugin can still work normally.

@neuralfraud

This comment has been minimized.

Copy link
Owner

@neuralfraud neuralfraud commented Nov 17, 2019

Any update? Not aware of any issues, let me know if this can be closed.

@forever765

This comment has been minimized.

Copy link
Author

@forever765 forever765 commented Nov 18, 2019

@neuralfraud Hi there, the current situation is that the first time you load a panel, it is very slow and requires 15~20s (network traffic graph). This situation is especially noticeable when using the grafana variable and refreshing the page.
Can you solve this problem?

@neuralfraud

This comment has been minimized.

Copy link
Owner

@neuralfraud neuralfraud commented Nov 18, 2019

@forever765

This comment has been minimized.

Copy link
Author

@forever765 forever765 commented Nov 18, 2019

@neuralfraud Between the prtg and grafana machines is a 1000Mbps intranet, on the same switch. In addition, this slow query will only happen on the graph panel.

@olsonnn

This comment has been minimized.

Copy link

@olsonnn olsonnn commented Nov 18, 2019

i've a question regarding this: are requests going to prtg serial or in parallel? Is there a away to change/ tweak settings here?

@alotufo

This comment has been minimized.

Copy link

@alotufo alotufo commented Jan 7, 2020

I just came here to say that we've been using this plugin for several months and it has been great, but painfully slow with data requests. I thought that since the plugin hadn't seen much development that I wasn't going to see any improvement.

I tried 4.0.4 a few days ago and it has made the load times 10x faster! I tried using Grafana with this plugin on a RaspberryPi previously and it was impossible on 4.0.3. It just froze up the browser from requests. It now runs fine as well.

I only needed to change some settings for channel names and everything else worked with my existing dashboards.

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.