Backing up correctly before performing updates

[jaxjexjox]

Hi @jarischaefer

Apologies, big apologies for this, I've been using docker for years but this is my first 'multi-docker' system with 2 dockers talking to one and other.

How do I absolutely, safely make a full backup of my configuration, so that I can comfortably update it, knowing it can be rescued?
Again, I'm very sorry, I know this is probably elementary stuff but all my docker containers I've "backed up" the -v /data content and that's it, I've never had to backup a database before.

Here's how she was spun up.

sudo docker run -p 8669:80 -e DB_HOST=libredb -e DB_NAME=librenms -e DB_USER=librenms -e DB_PASS='PASSHERE' -e APP_KEY=base64:APIKEYISHERE= -e BASE_URL=http://URLISHERE:8669 -e TZ=Australia/Melbourne --link mariadb:libredb -v /dockerconfigs/librenms/logs:/opt/librenms/logs -v /dockerconfigs/librenms/rrd:/opt/librenms/rrd -v /dockerconfigs/librenms/nagios_plugins:/mount/nagios_plugins -v /dockerconfigs/librenms/conf.d:/opt/librenms/conf.d/ --name librenms -d --restart always jarischaefer/docker-librenms:latest

sudo docker run -e MYSQL_ROOT_PASSWORD=XXXXXXXXXXX -e TZ=Australia/Melbourne -v /dockerconfigs/mariadbOther:/var/lib/mysql --name mariadb -d --restart always mariadb:latest --innodb_file_per_table=1 --lower_case_table_names=0

I'm happy to follow the upgrade guide but now that I've finally got a copy working I like, it's become useful and I'd hate to lose the work put into it.
Thank you, a lot for your time, really

[jarischaefer]

I will have to write new documentation regarding backups.

Offline backups

The easiest method to create a backup is to stop all containers, in which case, if the image's implementation and the configuration used to start the container are correct, the directories used as volumes can simply be copied. If the directory structure is librenms/web/[logs|rrd|...] for the web container and librenms/db for MariaDB, simply stop both containers and copy the librenms parent directory. You can verify the backup by creating a separate LibreNMS instance using the backed up directories and comparing the contents of the existing and separate installation.

Online backups

Creating a backup at runtime (no docker stop) usually entails knowledge of the inner workings of an application or container. This is because, at the time you issue cp or a similar command, it may not be guaranteed that the data being copied remains in a logically consistent state if it is concurrently updated by the application during the copying process. For instance, MariaDB or MySQL have to be backed up using dedicated utilities such as mysqldump --single-transaction. Furthermore, if multiple containers share state, logical consistency must be guaranteed collectively, not merely for each individual container.

MariaDB backups

A binary backup of MariaDB is probably less safe than an SQL backup, but faster to create and restore. If you are using mariadb:latest instead of fixed version numbers, a sudden major jump in versions could lead to problems, since the MariaDB version must be compatible with the binary backup. This is not a problem in many cases, since MariaDB can upgrade old storage formats, but it may not be true for all storage engines, configurations and versions.

Creating an SQL backup and compressing it to gzip on the fly
MariaDB's port must be published for this using docker run -p 3306:3306, or you have to figure out (using docker inspect) the internal IP of the container: mysqldump --host=127.0.0.1 --port=3306 --user=mysqluser -p --single-transaction librenms | gzip -1 > librenms.gz

The password prompt can be avoided by either passing the password via command line, by using the implicit ~/.my.cnf, or by passing an additional configuration file using defaults-extra-file=/some/path/.my.cnf. This file would be located on the host, or wherever the backup is run from.

Passing the password via command line

This is less secure than the other options: mysqldump --host=127.0.0.1 --port=3306 --user=mysqluser -pmysqlpass --single-transaction librenms | gzip -1 > librenms.gz

Passing the password via .my.cnf

Create a .my.cnf file in the home directory of the user running the backup, or specify mysqldump --defaults-extra-file=/some/path/.my.cnf. Make sure the file is protected using chmod 600 or similar commands.

[mysqldump]
user=mysqluser
password=mysqlpass

[jaxjexjox]

I will have to write new documentation regarding backups.

Offline backups

The easiest method to create a backup is to stop all containers, in which case, if the image's implementation and the configuration used to start the container are correct, the directories used as volumes can simply be copied. If the directory structure is librenms/web/[logs|rrd|...] for the web container and librenms/db for MariaDB, simply stop both containers and copy the librenms parent directory. You can verify the backup by creating a separate LibreNMS instance using the backed up directories and comparing the contents of the existing and separate installation.

Online backups

Creating a backup at runtime (no docker stop) usually entails knowledge of the inner workings of an application or container. This is because, at the time you issue cp or a similar command, it may not be guaranteed that the data being copied remains in a logically consistent state if it is concurrently updated by the application during the copying process. For instance, MariaDB or MySQL have to be backed up using dedicated utilities such as mysqldump --single-transaction. Furthermore, if multiple containers share state, logical consistency must be guaranteed collectively, not merely for each individual container.

MariaDB backups

A binary backup of MariaDB is probably less safe than an SQL backup, but faster to create and restore. If you are using mariadb:latest instead of fixed version numbers, a sudden major jump in versions could lead to problems, since the MariaDB version must be compatible with the binary backup. This is not a problem in many cases, since MariaDB can upgrade old storage formats, but it may not be true for all storage engines, configurations and versions.

Creating an SQL backup and compressing it to gzip on the fly MariaDB's port must be published for this using docker run -p 3306:3306, or you have to figure out (using docker inspect) the internal IP of the container: mysqldump --host=127.0.0.1 --port=3306 --user=mysqluser -p --single-transaction librenms | gzip -1 > librenms.gz

The password prompt can be avoided by either passing the password via command line, by using the implicit ~/.my.cnf, or by passing an additional configuration file using defaults-extra-file=/some/path/.my.cnf. This file would be located on the host, or wherever the backup is run from.

Passing the password via command line

This is less secure than the other options: mysqldump --host=127.0.0.1 --port=3306 --user=mysqluser -pmysqlpass --single-transaction librenms | gzip -1 > librenms.gz

Passing the password via .my.cnf

Create a .my.cnf file in the home directory of the user running the backup, or specify mysqldump --defaults-extra-file=/some/path/.my.cnf. Make sure the file is protected using chmod 600 or similar commands.

[mysqldump]
user=mysqluser
password=mysqlpass

I'll be honest with you, I'm actually terrified to STOP the containers.
I could be wrong, I've been playing with your docker build of librenms for a few years on and off, perhaps I know better now, but when I did last time, I had difficulty spinning it back up, to the same state.

As I said in my op, my 'dumber' dockers, I just stop, pull new, re-start and off we go.

With libre, I feel like the API key or something may have impacted it starting again properly.
Regardless it's time to play and learn! So thank you.

For what it's worth, I'm more than comfortable, with running multiple varieties of backups, storage is cheap and since I don't entirely know what I'm doing, I am happy to take multiple types.

Here's what I put together, before I realised you replied:

sudo mkdir /dockerconfigs/mariadbOther/backup
sudo docker exec -i mariadbOther mariabackup --backup --target-dir=/var/lib/mysql/backup --user=root --password=XXXXXXXXX
sudo docker exec -i mariadbOther mariabackup --prepare --target-dir=/var/lib/mysql/backup


sudo mkdir /dockerconfigs/mariadbOther/backupdump
sudo docker container exec mariadbOther bash -c "mysqldump -u root -pzxasqw12 --all-databases > "/var/lib/mysql/backupdump/all.sql""


sudo mkdir /backups/mariadbOther
sudo mkdir /backups/mariadbOther/backupdump
sudo mkdir /backups/mariadbOther/backup 

sudo mv /dockerconfigs/mariadbOther/backupdump /backups/mariadbOther/
sudo mv /dockerconfigs/mariadbOther/backup /backups/mariadbOther/

I would actually prefer your method, take a cold, offline copy of the entire DB directory, so I will try that too shortly, thank you.

[jaxjexjox]

@jarischaefer

Thanks for your help, it's always appreciated!
I'm really happy to admit, I was incorrect. I did manage to stop librenms, then mariadb (assuming that's correct order?)

I then backed up the fully offline data folders.
Pulled a new container for libre (thank you)
Re-ran them and they worked, did need to run something different to your instructions due to this error.
"Your database is out of date" "lnms migrate"
This did not fix the problem:
https://github.com/jarischaefer/docker-librenms#12-database-update
"docker exec librenms setup_database."

None the less, it's back up and running, with a new version of librenms and I feel a little safer stopping and starting.

I'll be honest here, I've been playing with your container on and off for so long, that it might have been back before I know to map content out, with the -v command, not sure.

So to be clear, my method was as follows

docker stop libre
docker stop maria
docker rm libre
docker rm maria
docker pull libre:specifictag
docker run maria (same command / variables I used months ago)
docker run libre (same command / variables I used months ago, except :specifictag for version)
docker exec -i libre /bin/bash
./lnms migrate

That was about it.

I have only 2 questions remaining (sorry)

1, does that method seem correct to you?
2, at what point should I be considering updating the mariadb image for my container, if ever?
(I assume at some point in time, it's the smart thing to do for security, features or performance?)

Thank you!

[jarischaefer]

There should be no need to run migrations. Was this message already present on the previous instance, or did it only appear after copying? Perhaps there is an issue with MariaDB's volumes. If you are unsure about losing data, there is no need to remove the containers, merely stopping them is sufficient. I assume all volumes are explicit via -v and not implicit (see docker volume ls).

Unless you have a good reason, just stick to a version that still receives updates: Docker Hub. New versions might introduce security features (algorithms, authentication schemes) that require new versions of the CLI and PHP libraries. The opposite, having a recent client, but a very old server, may also cause problems.

[jarischaefer]

@jaxjexjox Some docs are available now: https://github.com/jarischaefer/docker-librenms#backups

[jaxjexjox]

Well, hello @jarischaefer I successfully upgraded my basic setup at home several months ago.

Sadly, I've attempted to upgrade both librenms and mariadb at my test work install, with no luck unfortunately.
The long and the short of it is I am indeed being asked again, to migrate the database and this time it fails.

Here's what I've got now:

Version 23.1.0 - Wed Jan 25 2023 03:43:30 GMT+1100
Database Schema 2022_09_03_091314_update_ports_adsl_table_with_defaults (248)
Web Server nginx/1.18.0
PHP 8.1.14
Python 3.10.6
Database MariaDB 10.10.3-MariaDB-1:10.10.3+maria~ubu2204
Laravel 8.83.27
RRDtool 1.7.2

I can upgrade my mariadb within the same chain as you outlined above, this works ok.
MariaDB 10.10.7-MariaDB-1:10.10.7+maria~ubu2204
(I have reverted this to try and fix the problem)

Unfortunately when I upgrade my librenms that's where the problems begin (regardless of mariadb version, including unchanged)

Error from the web:

Database inconsistencies found during a database error, please fix to continue.
Your database is out of date!
./lnms migrate

Then this:

USER@SERVERNAME:/dockerconfigs$ sudo docker exec -i librenmsOther /bin/bash
su librenms
cd /opt
cd librenms
./lnms migrate

                      APPLICATION IN PRODUCTION.

WARN Command cancelled.

I've googled a few solutions and not getting much traction.
Any thoughts? Setup database doesn't work (I have backups so I'm happy to try random things at this point)

[jaxjexjox]

and after all that and at least 90 minutes messing around, I managed to get it working.

sudo docker exec librenmsOther setup_database

Seems to work better than

sudo docker exec -i librenmsOther /bin/bash

then manually running the command?
I'm not sure why this is, I thought I was starting to grasp some of this stuff, none the less re-setting up the database made migrating to a new version work fine!

Confused as to why this is but none the less I successfully upgraded again.
I'm working and thanks for your hard work.

[jaxjexjox]

@jarischaefer I'm finally closing this off, you've been more than helpful and I should have closed it earlier.

One final thing, if you get a chance, can you merge / pull any changes from the official for a new build when you get a chance (I've no idea what criteria you use for new versions?) but I successfully got them to add a new feature very easily, I was quite impressed.

librenms/librenms#15985

Thanks for your hard work!