Do you know what volumes to create so all grafana and influxdb settings are persisted?

[bikerdan]

I removed and recreated a container and have to start from scratch setting up the datasource and graphs and things. Thanks for your help!

[samuelebistoletti]

I'm not sure I understand your point. I made a lot of changes between version 1 and 2, maybe I lost something back. If you destroy a container, yes, you lose all your data. In case you destroy a container now, a database backup should be mandatory.
If you intend to ask me why I removed VOLUME instructions from Dockerfile in the new version, the answer is simply because I run some tests and they all failed upon upgrade of the old versions of Grafana and Influxdb, too much has changed. I preferred remove VOLUME instead of waste time trying to figure out where the problem was that led to the update to fail.

[bikerdan]

Most docker images that are potentially used in production have some way to persist their state. In this case, if for any reason the container needs to be recreated, you have to re-configure everything from scratch. Datasources, Grafana dashboards, EVERYTHING.

I understand there could be issues with data persisted between the 1 and to 2 versions and why you removed the volumes from the Dockerfile. (So you wouldn't have to support data incompatibilities between version 1 and 2. Right? Doesn't seem like it would be an issue if there was a warning in the README talking about the upgrade from 1 to 2)

The only question I had was what volumes I could create to persist the necessary data to keep all of the container's state. Is it as simple as creating a volume for the /var/lib/grafana and /var/lib/influxdb directories, or is there more to it than that?

[samuelebistoletti]

You're right on the first point, I'll think about adding VOLUME entries again on Dockerfile to enforce persistence, in the meantime I'm adding a warning on the README as you suggested.

Speaking about your question I have to run some tests in order to answer, should be simple as you describe, but it is just to ensure that nothing has changed about directories in InfluxDB and Grafana. I'll keep you updated.

[bikerdan]

Thanks for looking into this. And thanks for your work on this project. It has been really useful for us!

[arsaboo]

Indeed, persistence will be great feature to use this image in production.

[TheBishopOfSoho]

Hi there, want to chase up on this, are there any plans to implement volume support? This image works great as is, but need to figure out how to persist the data

[TheBishopOfSoho]

Did some fiddling about and think there is an issue with the setup on this. I am likely wrong, but cannot see why this would be. Anyway, I figured out that there are 3 locations that would need to be made into volumes, /var/lib/grafana, /var/lib/influxdb, but the problem one here is MySQL, included as it is it will NOT work with volumes due to well documented permissions issues with host volumes.
So I looked into removing MySQL, so far so good, volumes mounted fine, and I changed the Grafana.ini file to use ENV variables for the mysql details, but the issue Im now facing is that ENV vars are not being passed into the Container (from either the Docker file or the run command) meaning they have to be hard wired into the files which is not very useful. I feel like I am nearly there but this ENV issue is all that is preventing volumes from working properly

EDIT: Ignore the env issues, I couldnt see ENV variables when logging in via ssh, if you docker exec instead they are present. So I just need to connect Grafana to external DB and should have volumes working

EDIT2: Sorry for the spamming, but once I had changed the Grafana DB values to an external DB, removed the MYSQL stuff from the DOckerFile, and made volumes for influxdb and grafana, everything working as it should.

[samuelebistoletti]

Hi,
happy to know you appreciate this project. I will fix this as soon as possible, otherwise if someone want to fix that, it will be a pleasure to review an eventual pr.
Thanks

[TimSoethout]

I worked around this by using sqlite3 as database for Grafana, and then putting grafana.db on a volume.

Start the container with environment variable: GF_DATABASE_TYPE=sqlite3
and volume: ./grafana:/var/lib/grafana where the first part is the directory on your host.

[matthewmueller]

@TimSoethout could you share the changes to your Dockerfile to make that happen? Or is it just the environment variable? Also did you extract out the influxdb volume?

[TimSoethout]

@matthewmueller I don't think I changed the Dockerfile. The environment variable GF_DATABASE_TYPE sets the config value for grafana to sqlite. It follows the order of the configuration file. Then just add a volume on the correct default directory where the grafana.db file is saved.

I used docker compose for this.

docker-compose.yaml

version: '2'
services:
 metrics:
    image: samuelebistoletti/docker-statsd-influxdb-grafana
    ports:
      - '80:3003' # grafana
      - '81:8083' # influxdb-admin
      - '8086:8086' # influxdb
      - '8088:8088' # backup
      - '8089:8089/udp'
      - '8125:8125/udp' # statsd
    volumes:
      - ./grafana:/var/lib/grafana
    environment:
      - "GF_DATABASE_TYPE=sqlite3"

[ssbarnea]

I found this container while looking for a fully packed docker based monitoring solution but reading this ticket am really worried because monitoring database persistence across upgrades is clearly one of top feature needed.

I would like to hear some comments from @samuelebistoletti regarding this as I never expected to lose database when I update the container to a newer version.

If someone is aware of another monitoring container designed to assure persistency across update please let me know, I would prefer to save a good number of hours in deploying my home monitoring solution.

[jschilperoord]

What I did was this. I added the following volumes to the docker run command I use. For example:

docker run --ulimit nofile=66000:66000 -v /mnt/influxdb:/var/lib/influxdb -v /mnt/mysql:/var/lib/mysql -v /mnt/grafana:/var/lib/grafana -d --name docker-statsd-influxdb-grafana -p 3003:3003 -p 3004:8888 -p 8086:8086 -p 22022:22 -p 8125:8125/udp samuelebistoletti/docker-statsd-influxdb-grafana:latest

Then I made sure the users inside the container were able to write in those folders by executing the following commands from my host:

chown -R 105:106 /mnt/mysql
chown -R 107:109 /mnt/grafana
chown -R root:root /mnt/influxdb

Because there are no valid mysql data files. The mysql database process fails to start. I fixed this by getting a shell inside the container:

docker exec -ti docker-statsd-influxdb-grafana /bin/bash

Inside the container I killed the mysql process. By using kill -9 if I had to. Because the next commands fails if there are files inside /var/lib/mysql. Then I executed (stil inside the container):

mysqld --initialize-insecure --user=mysql

To generate the default data files for mysql. And started mysql by executing:

supervisorctl start mysql

Then I executed this snippet from the init mysql script to create the grafana db and user:

echo "Creating user" echo "CREATE DATABASE grafana" | mysql --default-character-set=utf8 echo "CREATE USER '${MYSQL_GRAFANA_USER}' IDENTIFIED BY '${MYSQL_GRAFANA_PW}'" | mysql --default-character-set=utf8 echo "GRANT ALL PRIVILEGES ON *.* TO '${MYSQL_GRAFANA_USER}'@'%' WITH GRANT OPTION; FLUSH PRIVILEGES" | mysql --default-character-set=utf8

Now I saw grafana starting up and data coming in. So I can now at least remove the entire container with a docker rm --force docker-statsd-influxdb-grafana without losing all my data.

Will try to figure this out some more and provide a PR to do this in a decent way.

[srijken]

@TimSoethout I used this compose file, and that works great. What I do miss though is that the influxdb contents aren't in a volume. I tried creating a volume on /var/lib/influxdb, but then influxdb crashes :(

Anyone any ideas how to also get the influxdb data in a docker volume?

[humm3r1]

@TimSoethout I used this compose file, and that works great. What I do miss though is that the influxdb contents aren't in a volume. I tried creating a volume on /var/lib/influxdb, but then influxdb crashes :(

Anyone any ideas how to also get the influxdb data in a docker volume?

Hey Srijken,

I figured this out today since we use this container to monitor some environmental sensors and want to have persistent data.

I figured out it is mostly a permissions issue for the influxdb part. Give the /mnt/influxdb 999:999 permissions with chown and it should start working right away. I discovered this by setting the /var/lib/influxdb with chown to influxdb:influxdb and it began to work. Then back on the host I found the polkitd:input user/group set which maps back to 999 within the container.

So you want to run this on the host (not container) to set the permissions correctly for influxdb:
chown -R 999:999 /mnt/influxdb

Here is my pasteable script for this container to pull a copy from github and spin it up with persistent volumes:

docker pull samuelebistoletti/docker-statsd-influxdb-grafana:latest
mkdir /mnt/influxdb /mnt/mysql /mnt/grafana
docker run --restart=unless-stopped --ulimit nofile=66000:66000 -v /mnt/influxdb:/var/lib/influxdb:Z -v /mnt/mysql:/var/lib/mysql -v /mnt/grafana:/var/lib/grafana -d --name docker-statsd-influxdb-grafana -p 3003:3003 -p 3004:8888 -p 8086:8086 -p 22022:22 -p 8125:8125/udp samuelebistoletti/docker-statsd-influxdb-grafana:latest
chown -R 105:106 /mnt/mysql
chown -R 107:109 /mnt/grafana
chown -R 999:999 /mnt/influxdb
docker exec -ti docker-statsd-influxdb-grafana /bin/bash
supervisorctl stop mysql
sleep 5
rm -rf /var/lib/mysql
mysqld --initialize-insecure --user=mysql
supervisorctl start mysql
sleep 5
echo "Creating user"
echo "CREATE DATABASE grafana" | mysql --default-character-set=utf8
echo "CREATE USER '${MYSQL_GRAFANA_USER}' IDENTIFIED BY '${MYSQL_GRAFANA_PW}'" | mysql --default-character-set=utf8
echo "GRANT ALL PRIVILEGES ON . TO '${MYSQL_GRAFANA_USER}'@'%' WITH GRANT OPTION; FLUSH PRIVILEGES" | mysql --default-character-set=utf8

[dimitriy-k]

I am using this image now, but I think I will switch to some other image where my data can be stored outside of the container. It is definitely not ok that if you remove container all the data is gone. How do you want to update container to the the latest image? So, never update you container or use some backup exports :(

[samuelebistoletti]

I'm working on a new release in order to persist data. Plus a stack configuration file. Sorry to be late but I'm very busy at work.

[ssbarnea]

@samuelebistoletti Thanks for the update! I am looking forward to see it. Once the persistence is official I could probably trust using the bundle and even upgrading the container without being worried that all my data would be lost.

Maybe in longer term even a sample k8s example deployment config.

[feutl]

I'm working on a new release in order to persist data. Plus a stack configuration file. Sorry to be late but I'm very busy at work.

Any news on this?
Want wanted to start monitoring my network and this docker does everything I need ;) including telegraf :) which would be great for my purpose.

Anything on persistence ?

Thanks

[emilw]

Hi,
This docker image looks great but i also need the volume support, any updates on when it will be released?

It feels like your image are the most trust worthy on docker, so i rather use yours than any forks :-)!

[G-eos]

@TimSoethout I used this compose file, and that works great. What I do miss though is that the influxdb contents aren't in a volume. I tried creating a volume on /var/lib/influxdb, but then influxdb crashes :(
Anyone any ideas how to also get the influxdb data in a docker volume?

Hey Srijken,

I figured this out today since we use this container to monitor some environmental sensors and want to have persistent data.

I figured out it is mostly a permissions issue for the influxdb part. Give the /mnt/influxdb 999:999 permissions with chown and it should start working right away. I discovered this by setting the /var/lib/influxdb with chown to influxdb:influxdb and it began to work. Then back on the host I found the polkitd:input user/group set which maps back to 999 within the container.

So you want to run this on the host (not container) to set the permissions correctly for influxdb:
chown -R 999:999 /mnt/influxdb

Here is my pasteable script for this container to pull a copy from github and spin it up with persistent volumes:

docker pull samuelebistoletti/docker-statsd-influxdb-grafana:latest
mkdir /mnt/influxdb /mnt/mysql /mnt/grafana
docker run --restart=unless-stopped --ulimit nofile=66000:66000 -v /mnt/influxdb:/var/lib/influxdb:Z -v /mnt/mysql:/var/lib/mysql -v /mnt/grafana:/var/lib/grafana -d --name docker-statsd-influxdb-grafana -p 3003:3003 -p 3004:8888 -p 8086:8086 -p 22022:22 -p 8125:8125/udp samuelebistoletti/docker-statsd-influxdb-grafana:latest
chown -R 105:106 /mnt/mysql
chown -R 107:109 /mnt/grafana
chown -R 999:999 /mnt/influxdb
docker exec -ti docker-statsd-influxdb-grafana /bin/bash
supervisorctl stop mysql
sleep 5
rm -rf /var/lib/mysql
mysqld --initialize-insecure --user=mysql
supervisorctl start mysql
sleep 5
echo "Creating user"
echo "CREATE DATABASE grafana" | mysql --default-character-set=utf8
echo "CREATE USER '${MYSQL_GRAFANA_USER}' IDENTIFIED BY '${MYSQL_GRAFANA_PW}'" | mysql --default-character-set=utf8
echo "GRANT ALL PRIVILEGES ON . TO '${MYSQL_GRAFANA_USER}'@'%' WITH GRANT OPTION; FLUSH PRIVILEGES" | mysql --default-character-set=utf8

Thanks a lot, I add the follwing line to store user database for web login to influxdb
-v /mnt/chronograf/chronograf-v1.db:/root/chronograf-v1.db
Note, users rights was not working fine for me. I log into the contrainer and chance the rights before restarting and eveything goes fine.
Note, using commands like docker cp -a /var/lib/mysql /mnt/ help to retrieve data of a running container and move to local the files.