Memory Leak with Playlist

[cong-trinh]

Seems like there is another bug somewhere or there a regression for #1497

I am running the latest version of Grafana (2.6) in playlist mode and the playlist seems to be continuously consuming memory. The timeframe until the system runs out of memory seems to differ from 2-5 hours.

Tested on:
Ubuntu Desktop:
Chrome
Firefox

Raspberry pi2:
Iceweasel
Epiphany

The playlist are backed by both InfluxDB 0.9 and cloudwatch dashboards.

I currently circled around the problem by using a tab rotator which has keep the "playlist" going for the last 5 days.

[bergquist]

The playlist feature have been rewritten in master. But I guess some memory profiling would be nice. Might not be playlist feature that causes problem.

[torkelo]

it is probably not the playlist feature itself but some memory leak when switching dashboards,

[torkelo]

thanks for reporting this! Fixed in master (and in 3.0 pre release builds)

[discoduck2x]

just tried build 1/20/2016, 12:05:40 PM , still get memory problem?

[leleobhz]

I'm requesting this bug to be reopened. I'm trying the grafana-latest.deb:

docker[20483]: 2016/03/30 14:05:57 [I] Starting Grafana
docker[20483]: 2016/03/30 14:05:57 [I] Version: 3.0.0-pre1, Commit: v2.6.0-1610-g04d142c, Build date: 2016-03-29 15:09:49 +0000 UTC

And Im running a Playlist with 2 screens and other with 3 screens, one playlist per TV (A LG SmartTV).

Past some minutes (From 30 min to 1 hour), TV crashes the browser telling me there is no enough memory to execution. I saw the last fixes commited in this bug report, this is why I'm using the latest build, but no progress.

It a Lonelly screen is sustained in TV, we do not have any crashes.

What can I do to help and what can be done to fix this?

[torkelo]

@leleobhz that is incredibly strange, after a full rotation of the dashboards grafana will do a full reload, and restart the playlist. No memory leak should be possible after the reload. The only cause for your issue is if you have thousands of dashboards in a single playlist.

[utkarshcmu]

This memory occurs with our tvs as well here but after a week or couple of weeks. Not after 30 mins.

[leleobhz]

@torkelo I agree with you, this is why I asked to open it for investigation. In my browser, i ran the heap recorder:

leonardo@host:~$ ls -lah grafana_*
-rw-rw-r-- 1 leonardo leonardo 18M Mar 30 13:31 grafana_2.6_playlist.heaptimeline
-rw-rw-r-- 1 leonardo leonardo 40M Mar 30 13:27 grafana_3.0_latest_playlist.heaptimeline

If I have a way to send it privatelly (Since have my employee information inside, like URLs), I can provide you these files to a deep investigation. Its possible?

Just for reference, im running https://github.com/monitoringartist/grafana-xxl modified to get grafana_latest_amd64.deb and to get the main grafana-zabbix repository instead the old forked (Dockerfile in http://paste.pound-python.org/show/Hrcm4Qi3X0O5NiwfZI1I/ )

[dannygoulder]

I am also getting a memory leak issue with IE11 on Win7 (32-bit). Running a playlist with 4 dashboards (three with Graphite datasources and 1 with elasticsearch/graphite mixed, in case that is relevant). Not sure how to figure out where the memory is leaking, but the memory usage of that IE instance gets up to 1.4 GB or so after 3-5 hours and then it stops working. I was previously running the same playlist on 2.6 without issue (it ran for days and days without problems). Haven't tried Chrome yet.

[dannygoulder]

Sorry. Forgot to mention this still happens on beta2

[torkelo]

I am sorry, I have no idea how you can be seeing memory leaks in Grafana 3.0 .

Testing this with 5 dashboards with many graphs, switch dashboard every 2 seconds. Zero memory leak shown after a long time, either in the tab javascript heap, or in the OS process monitor. As I said before the whole tab reloads after a every playlist loop. So the memory leak must be caused by browser extensions you have that cant handle many page reloads.

[dannygoulder]

Hi, @torkelo, thanks for looking at this. I did some more investigation with IE and it would appear that the leak is caused by Internet Explorer. IE lets you take a heap snapshot and they are showing as 9-20MB. Obviously the JS heap is not growing, it just seems as though IE is leaking memory across reloads (perhaps it is caching the entirety of the navigation history for usage of the back button?).

You are right, the leak doesn't appear to be in the JS code - but I am still curious as to why this didn't happen with 2.6.

[torkelo]

@dannygoulder in 2.6 the playlist feature did not do any full page reloads, that was introduced in 3.0 to remove any issues with a js memory leak

[dannygoulder]

Ok that's great. I tried running IE without addons but that didn't help.

I ran the same playlist for days on end without any issue.

Any chance to make the old behaviour an option for users of crappy browsers (I don't have a choice, sadly!)

On 13 Apr 2016, at 18:07, Torkel Ödegaard notifications@github.com wrote:

@dannygoulder in 2.6 the playlist feature did not do any full page reloads, that was introduced in 3.0 to remove any issues with a js memory leak

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[DMarby]

I'm seeing this behaviour when only running one dashboard that has auto-refresh as well.
Running Grafana 3.0, on a Raspberry Pi 2 in Chromium, using InfluxDB as a backend.

[leleobhz]

Testing this with 5 dashboards with many graphs, switch dashboard every 2 seconds. Zero memory leak shown after a long time, either in the tab javascript heap, or in the OS process monitor. As I said before the whole tab reloads after a every playlist loop. So the memory leak must be caused by browser extensions you have that cant handle many page reloads.

Here I suffer this issue in LG Smart TV 47LB5800-SB.

Thanks.

[tubemeister]

For what it's worth...

I ran into this memory leak as well, with Grafana 3.0 (Ubuntu) on a Pi 2B with Chromium. I ran into the memory leak on the playlist feature (5m cycle) but manually reloading (F5) dashboards turned out to be leaky as well. The Nagios screen next to it also leaks, but the pages are so tiny compared to Grafana that the effect was only noticeable over weeks.

Upgrading the Pi from Wheezy to Jessie and from Chromium 22 to 48 seems to have mostly fixed the memory leak issue. It still leaks a bit, but significantly less.