2.1.0-pre1 - Light theme - Dark graphite PNG graphs

[gerrickw]

Summary:
Looks like grafana is hardcoded to fetch graph images as a dark background from graphite when creating the url. When a user decides to use graphite png in a light background the graphite images are shown in the dark layout.

Steps to Reproduce:

1. Browse to http://play.grafana.org/?panelId=3&fullscreen&theme=light&edit
2. Note: When setting the theme to light on the user, instead of by url, the same problem happens.
3. Click on "Display Styles" tab
4. Click on the radio button "Graphite PNG".
5. Click Back to dashboard
6. Notice: Black background image graph shown to user in a light theme.

Linking to latest commit in master where possibly a conditional operator could be used to remove bgcolor/fgcolor if a light theme:

grafana/public/app/panels/graph/graph.js

Lines 460 to 461 in c628ac9

	url += '&bgcolor=1f1f1f'; // @grayDarker & @grafanaPanelBackground
	url += '&fgcolor=BBBFC2'; // @textColor & @grayLighter

[torkelo]

yea, Graphite PNGs are only supported with a dark theme. Graphite PNGs are going to be phased out now that Grafana 2.0 has support to render grafana graphs and panels to PNG, and this rendering does support dark & light

[gerrickw]

Wow, that is disappointing. The benefit of the graphite images is the huge amount of browser lag that occurs when refreshing 50 graphs on a page at span 1 or 2 with the flot graphs. There is a difference between slowness of waiting for an image to download compared to the browser hanging to render so many graphs.

Is this something you are going to improve before phasing out a image support?

[torkelo]

@gerrickw if you get browser lag when rendering client side graphs that is due to you having old graphite install. If you had graphite 0.9.12 + maxDataPoints patch or 0.9.13 you would see that Grafana can render graphs faster clientside than the Graphite server side PNGs (I have done benchmarks).

The slowness is caused by a problem in the graphite json api pre 0.9.13 (or 0.9.12 with patch applied).
http://docs.grafana.org/installation/performance/

As it turns out browsers are more optimized at rendering graphics than a python lib :)

[torkelo]

The fix to graphite-web was implemented in March 2013, graphite-project/graphite-web#170

And was merged into 0.9.x branch in : graphite-project/graphite-web#334
But was merged to late to be included in graphite 0.9.12

[gerrickw]

I am on master for graphite so this is not the issue. You can recreate the issue if creating a large number of graphs with ~20-40 metrics each with 50 graphs displaying on a page. When refreshing with control+r your browser will hit 100% cpu and hang the page when attempting to scroll down.

[torkelo]

This dashboard with 25 graphs loads complete with all graphs in about a second. http://play.grafana.org/dashboard/db/perf-test

But I can see that if you have that many graphs and that many series it can overload the browser and its single processing thread. But a dashboard with 50 graphs is not what Grafana is designed for, and is poor dashboard design (lots of requests to graphite that are pointless if you have to scroll down to see the graphs, unless you always scroll down, and if you zoom in on a graph do you always scroll up and down to check all other graphs?).

I usually create templated dashboards instead of huge dashboards (or more target dashboards), so everything can be viewed on a single screen without scrolling. But if you want to scroll and have a large dashboard create rows and collapse the rows that you do not want to show. Collapsed rows to not cause queries & renders.

I can see that for your graphite the multi threaded nature of the server, it can be a little faster with server side PNGs that with client side graphs. But its is a edge case that can be worked around (maybe not to your 100% liking).

[gerrickw]

I suppose scrolling isn't the only issue as some of my views are not scrolling down. Mainly when the problem happens you can do no other action on the page until the graphs load. I tried reproducing the issue on play.grafana.org, but I actually am unable to see the same problem. I exported your perf-test, replaced metrics with ....., removed the legend, and changed them to a span of 1 and noticed the graphs only loaded one or two at a time. Possibly you have a limit on your server side or only a few cores to supply metrics to the frontend. When looking at chrome's network inspection tab it takes 12 seconds to render 44 graphs that contain ~40 metrics per graph, which is very slow compared to my setup at work.

I'll have to check at work tomorrow if true, but I am wondering since the graphite responses return quickly when requested from the browser, it is being overloaded and slowing down. This may also make it appear your perf-test seem okay as your server response is slow so the browser can easily keep up.

So you know my usecase, I am using templated scripted dashboards that gives the user a way to research production issues by expanding and collapsing metrics by row. There is a default set of 3 rows expanded for the most useful and then they can expand additional if needing to research more. Each row contains 12 graphs/pool of servers with a span of 1 with no legend. If a pool/graph looks out of place they can zoom or change the span and attempt to match other pool's data to other expanded rows to figuring out a problem. Scrolling only happens if the user notices a problem and needs to expand rows to tie spikes in data to other metrics. I provide multiple links from the drop down to other scripted/templated dashboards that can provide zoomed legends views, and various other views for research.

I'll let you know of my findings tomorrow. Thanks for getting back to me. :)

[torkelo]

how exactly do you get meaningful graphs with 40 series in small 1span graphs with no legend???

[torkelo]

Did a perf test just for fun with my local graphite install. A graph with 50 series, graphite returns json in 350ms, client side rendering (graph about 1200px wide 500 pixels high) completes in about 1.5 seconds.

Switching to graphite PNG, the image returns from graphite after 5.7 seconds. But given enough graphs the parallellism of the server could win I guess. But interesting finding, usually only have about 10 series max in my graphs, more than that and they become to messy to be useful. So I have actually not test grafana a lot with graphs with 30+ series, so was thinking that PNG would be faster for graphs with many series but graphite PNGs are a lot slower it seems even when having many series.

[gerrickw]

When I have mentioned slower I also mean in terms of a usable browser. Currently at work with 50 graphs with ~15-30 metrics it returns in 1.5 seconds with additional time hanging. As I add graphs the time increases with more noticeable browser lag. With images it may take slightly longer in general, but with 16 cores and tmpfs it isn't that long, but I can continue using the browser to investigate production issues.

This becomes more of a problem if enabling auto-refresh and I am using the mouse to look at the tooltip, attempting to zoom, or clicking a graph menu to open a link and the browser stops responding. Using Firefox with a few dashboards auto refreshing will permanently halt my browser. Some people are okay with the slowness while researching as they understand quite a bit of data against the browser, while others feel it is a bad user experience and would rather live with the slight loss of usability and load time so their machine doesn't feel like it is freezing. Is there a way to queue up events instead of attempting to have the browser process everything at once if the server is fast?

Meaningful small graphs when the lines should be consistent is easy to spot outliers. Then progress to different views to diagnose issues. They also are not that small on larger resolutions. If they have a smaller screen size I detect and adjust the span or they can manually change.