grafana persistent volume increasing to more than 20Gi

[chaoyun]

Please include this information:

What Grafana version are you using?

Grafana v5.3.0

What datasource are you using?

Postgres/ Elastisearch/ Promethsus

What OS are you running grafana on?

Centos

What did you do?

I'm using the k8s persistent volume for the grafana. I only set 1Gi volume size for it.
But after several days later, when I check the grafana.db file under my host, it turns to be 20Gi.
What's the content in this file? And why the size increased frequently?
grafana@grafana-0:/var/lib/grafana/data$ ls -al
total 20619528
drwxrwxrwx 2 root root 24 Dec 28 00:57 .
drwxrwxrwx 1 grafana grafana 29 Dec 28 00:55 ..
-rwxrwxrwx 1 grafana grafana 21098926080 Dec 28 00:57 grafana.db

What was the expected result?

What happened instead?

If related to metric query / data viz:

Include raw network request & response: get by opening Chrome Dev Tools (F12, Ctrl+Shift+I on windows, Cmd+Opt+I on Mac), go the network tab.

[torkelo]

maybe you have encountered this bug? #12864

Can you inspect the sqlite3 db (using sqlite3) and check the row count in the dashboard_version table ?

Are you using provisioning? Maybe have the same dashboard in two different mapped folders

[chaoyun]

yes, I'm using provisioning. I created a new folder under dashboard/,
under dashboard/ folder there're 37 dashboards. Under dashboard/dev/ folder, I have 29 dashboards.
But the dasbhoard name is different btw these two folders. Some pannels may be the same.
And another thing is most of the dashboard refresh interval is 5s.

And I tried to set the versions_to_keep = 1, seems still growing very fast.

yesterday, I delete this grafana.db, about 20h, it becomes 1G now.
You can see the rows is 80661 for the dashboard_version table. Every refresh interval, the dashboard_version table will create one row?

[grafana-data-0]# ls -al
total 1106116
drwxrwxrwx 2 root root 24 Dec 29 09:46 .
drwxr-xr-x 5 root root 55 Dec 11 14:59 ..
-rwxrwxrwx 1 472 472 1119068160 Dec 29 09:46 grafana.db

[ grafana-data-0]# sqlite3 grafana.db
SQLite version 3.7.17 2013-05-20 00:56:22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .database
seq name file

---

0 main /var/data/monitoring/grafana-data-0/grafana.db
sqlite> select count(*) from dashboard_version;
80661

[chaoyun]

And I found some "uid" is same btw these two dashboard folders, is that will cause the issue?

[chaoyun]

I've changed the uid to let each dashboard unique. And I also removed some duplicate panels. Now after 3days, the grafana.db increased to 3G. Could you please take a look? And find the root cause? I also upgrade the grafana to v5.4.2.
[root@chaosu-master-01 cmts-hm-decoder]# ls -al /var/data/monitoring/grafana-data-0/
total 3402756
drwxrwxrwx 2 root root 24 Jan 2 09:41 .
drwxr-xr-x 5 root root 55 Dec 11 14:59 ..
-rw-r--r-- 1 472 472 3419291648 Jan 2 09:41 grafana.db

sqlite> select count(*) from dashboard_version;
224835

[torkelo]

closing this as it's the same as #14674 , we will try to investigate this asap.

[chaoyun]

hi, torkelo
I have some clue for this issue. When my dashboard yaml file as the below will have the issue, that is having both the "default" and my newly created folder. Using this yaml, after grafana re-start, I can see some dashboards should under the dir dev/, but it turns out under the General folder. And I can see "Folder not found" error.

apiVersion: 1

providers:

• name: 'default'
  orgId: 1
  folder: ''
  type: file
  disableDeletion: true
  updateIntervalSeconds: 60
  options:
  path: /var/lib/grafana/dashboards
• name: 'test1'
  orgId: 1
  folder: 'test1'
  type: file
  disableDeletion: true
  updateIntervalSeconds: 60
  options:
  path: /var/lib/grafana/dashboards/test1
• name: 'test2'
  orgId: 1
  folder: 'test2'
  type: file
  disableDeletion: true
  updateIntervalSeconds: 60
  options:
  path: /var/lib/grafana/dashboards/test2

=================
After I changed to the below one, remove the "default", the issue is gone.

apiVersion: 1

providers:

• name: 'k8s'
  orgId: 1
  folder: 'k8s'
  type: file
  disableDeletion: true
  updateIntervalSeconds: 60
  options:
  path: /var/lib/grafana/dashboards/k8s
• name: 'test1'
  orgId: 1
  folder: 'test1'
  type: file
  disableDeletion: true
  updateIntervalSeconds: 60
  options:
  path: /var/lib/grafana/dashboards/test1
• name: 'test2'
  orgId: 1
  folder: 'test2'
  type: file
  disableDeletion: true
  updateIntervalSeconds: 60
  options:
  path: /var/lib/grafana/dashboards/test2

[torkelo]

Oh, that explains it , the provisioning system checks for files sub folders , so with that default you map the same files 3 times

[mario-amato]

FYI, I was able to reproduce this on version 6.4.3 by having non-unique dashboard provider names.
There are 9 yaml files in the conf/provisioning/dashboards path. Each one has the following structure:

apiVersion: 1
providers:
 - name: 'Test Dashboard Provider'
   orgId: 1
   folder: 'Test'
   type: file
   editable: true
   disableDeletion: true
   options:
     path: ./dashboards/test

Previously the name values were set to default for every dashboard provision. This caused the database_version table to grow unbounded.
The fix was to name every name unique in every dashboard provision file. I realize that having non-unique names is probably an invalid configuration; however, grafana stil starts up and runs. Perhaps the service should bail on loading the configuration with some useful messaging?

[nvtkaszpir]

@torkelo

Notice that

closing this as it's the same as #14674 , we will try to investigate this asap.

that issue id refers to THIS issue (aka to itself)? :)

[nvtkaszpir]

notice for anyone using k8s - ensure to not have duplicated dashboards via configmaps (I mean the contents)

[wwyhy]

Hi,

we did not using the dashboard provisioning, but the grafana.db is also take up 17GB during pass 7 weeks.

The grafana version is v4.4.3. any how can we solve this issue?

sqlite> select count(*) from dashboard_version;
1165732