Skip to content

Commit

Permalink
Adding option for flushing/locking MySQL/MariaDB before taking a snap…
Browse files Browse the repository at this point in the history
…shot. Improved Readme.
  • Loading branch information
hyper_ch committed Oct 28, 2018
1 parent c5666cd commit aa27687
Show file tree
Hide file tree
Showing 3 changed files with 33 additions and 16 deletions.
3 changes: 3 additions & 0 deletions .my.cnf.example
@@ -0,0 +1,3 @@
[mysql]
user=root
password=my-database-password
31 changes: 17 additions & 14 deletions README.md
Expand Up @@ -2,8 +2,8 @@

A little zfs snapshot tool including backup script. The tool consists of two scripts:

* easysnap: It will zfs snapshots and remove old ones if necessary. No configuration file needed, info is in custom dataset properties.
* easysnapRecv: It will allow to pull snapshots from other servers. A configuration file is needed.
* easysnap: It will zfs snapshots and remove old ones if necessary. No configuration file needed as all required information is in the dataset's user properties.
* easysnapRecv: It will allow to pull snapshots made by easysnap from other servers. A configuration file is needed.

## easysnap

Expand All @@ -12,18 +12,19 @@ easysnap is a simple bash script which will take snapshots of designated dataset
### How to use

1. Clone this repo or just fetch the easysnap script and make it executable.
1. You need to a custom user property to the desired dataset. You can do this by `zfs set easysnap:hourly="240:200" pool/path/to/dataset`, where
1. You need to set a user property on the desired dataset. You can do this with `zfs set easysnap:hourly="240:200:m" pool/path/to/dataset`, where
* `easysnap:hourly` is the property name.
* `easysnap:` is the mandatory part. Easysnap will search for this property and must not be changed.
* `hourly` is also mandatory but it can be anything. Instead of `hourly` you can provide any string; common ones would be `frequent`, `hourly`, `daily`, `weekly`, `monthly` because they can easily be scripted to run by cron or systemd timers.
* `240:200`:
* `240` is mandatory and it indicates that it should keep 240 snapshots. Instead for a positive number of snapshots you could also use:
* `-1` (unlimited snapshots); or
* `0` (remove all easysnap:xxx snapshots);
* `false ` to not make any snapshots/changes.
* `:200` is optionaal and it indicates that easysnap will print a notice when there's less than 200GB free space on the pool.
* __Notice:__ easysnap removes/adjusts only the number of snapshots provided for this run. E.g. if you run the `./easysnap hourly` routine then it will only remove snapshots containing the string `easysnap-hourly` in the snapshot name. No other snapshots are touched. This is intentional as you might want to make custom snapshots at some points and those should not be automatically removed.
1. When you have setup at least one dataset, you will need to add a cronjob or systemd timer to actually call the easysnap tool with the jobname that it should run. E.g. if you setup `easysnap:hourly` you could add a cron like `0 * * * * /path/to/easysnap hourly`. The parameter provided to the easysnap script (in this case `hourly`) must match the desired user property in the dataset `easysnap:hourly`. You can also run the script manually whenever you want, just call it like `./easysnap hourly`. In the [cron.example](cron.example) file you have examples for the common snapshot intervals.
* `-1`: keep unlimited snapshots; or
* `0`: remove all easysnap:xxx snapshots; or
* `false `: do not not make any snapshots/changes.
* __Notice:__ easysnap only removes/adjusts the number of snapshots provided for this interval. E.g. if you run the `./easysnap hourly` routine then it will only remove snapshots containing the string `easysnap-hourly` in the snapshot name. No other snapshots are touched. This is intentional as you might want to make custom snapshots at some points and those should not be automatically removed.
* `:200` is optional and it indicates that easysnap will print a notice when there's less than 200GB free space on the pool.
* `:m` is optional and it indicates that MySQL/MariaDB tables should be locked before the snapshot and unlocked after the snapshot. If MySQL/MariaDB is started with privileges, you'll also need to create a `.my.cnf` file in the user's home under which the script is run, e.g. `/root/.my.cnf`, so that it reads out username and password from that file, see [.my.cnf.example](.my.cnf.example). If you don't want to be warned about remaining space, you can also just set the value like `240::m`.
1. When you have setup at least one dataset, you will need to add a cronjob or systemd timer to actually run the easysnap script with the interval that it should run. E.g. if you setup `easysnap:hourly` you could add a cron like `0 * * * * /path/to/easysnap hourly`. The parameter provided to the easysnap script (in this case `hourly`) must match the desired user property in the dataset `easysnap:hourly`. You can also run the script manually whenever you want, just call it like `./easysnap hourly`. In the [cron.example](cron.example) file you have examples for the common snapshot intervals.

### Format of the snapshots

Expand All @@ -35,21 +36,23 @@ pool/path/to/DS@1540490401_easysnap-frequent_2018-10-25_20h00-CEST
```

For sorting order the first part `1540490401` is the unix timestamp. This will ensure, that we don't end up with two snapshots with the same name. The second part is the `easysnap-interval`, so that we can make sure we only add/delete snapshots of the supplied job interval given. The last part is date / time provided incl. timezone `2018-10-25_20h00-CEST` for easier understanding of when a snapshot was actually taken.
* For sorting order the first part `1540490401` is the unix timestamp. This will ensure, that we don't end up with two snapshots with the same name.
* The second part is the `easysnap-interval`, so that we can make sure we only add/delete snapshots of the supplied interval.
* The last part is date / time provided incl. timezone `2018-10-25_20h00-CEST` for easier understanding of when a snapshot was actually taken.


## esaysnapRecv

easysanpRecv is a simple bash script which will zfs send/receive datasets or incremental snapshots from the same machine or a remote one to the machine where the script is run. It also can handle encrypted datasets that are currently in zfs master. By default it will also send/recv the intermediary snapshots between to runs but that can be disabled if wanted. Also, if it fails to receive the intermediary snapshots it will fallback to just an incremental send.
easysnapRecv is a simple bash script which will zfs send/receive datasets or incremental snapshots from the same machine or a remote one to the machine where the script is run (pull). It also can handle encrypted datasets (that feature is still in zfs master). By default it will just send/recv the incremental snapshot

### How to use

1. You need to make a configuration file in `/etc/easysnap` named linke `easysnap.frequent`, `easysnap.hourly` etc. It must correspondent with the easysnap interval name chosen.
1. You need to make a configuration file in `/etc/easysnap/` named like `easysnap.frequent`, `easysnap.hourly` etc. It must correspondent with the easysnap interval name chosen.
1. The config has the following format: `localDS,keyDS,raw,intermediary,exportPool,remoteHost,remoteDS,snaps,reqFreeG`
* `localDS`: The path to the local dataset that shall be used, e.g. `tank/path/to/Backups/xxx`. __Notice:__ If the local dataset does not exist, it will be created and it will just have the latest snapshot. If you want to have also previous intermediary snapshots, then you should first make a zfs send/recv starting from the snapshot you want.
* `keyDS`: In case you make use of encryption, you will need to provide the path to the dataset that holds the encryption (it's a different one when inheritance is used). I just by for all my encrypted zfs the following: `tankXXX/encZFS`. If your local dataset has no encryption, just leave it empty.
* `keyDS`: In case you make use of encryption, you will need to provide the path to the dataset that holds the encryption key. Because of inheritance, child datasets of an encrypted dataset will also be encrypted, but you'll need to provide the key just to the parent dataset with the original encryption. In my case I just do for all my encrypted zfs the following: `tankXXX/encZFS` and then I create child datasets like `tankXXX/encZFS/Nixos` -> so the value to provide would be `tankXXX/encZFS`. If your local dataset has no encryption, just leave this empty.
* `raw`: In case the remote dataset that you want to receive is encrypted you can set the raw flag to `y`, then it will be sent in raw mode. This is espeically useful when the receiving machine is untrusted, e.g. features no encryption by default. That way the receiving dataset will still be encrypted.
* `intermediary`: By default easysnap will use the `-i` flag to just send an incremental snapshot. If you also want intermediar snapshots then set this option to `y` and easysnap will try to send the intermediary snapshots also with the `-I` flag. However, if intermediary sending fails, it will output a notice and retry with just incremental snapshot. __Notice:__ If you use the intermediary sending, it may also send snapshots from other tools and other frequncies. E.g. if you just do a daily send, it will for example also include hourly and frequent snapshots and they won't get removed by the deleting, as deleting of snapshots strictly adheres to just the chosen interval.
* `intermediary`: By default easysnap will use the `-i` flag to just send an incremental snapshot. If you also want intermediary snapshots then set this option to `y` and easysnap will try to send the intermediary snapshots by using the `-I` flag. However, if intermediary sending fails, it will output a notice and retry with just incremental snapshot. __Notice:__ If you use the intermediary sending, it may also send snapshots from other tools and other intervals. E.g. if you just do a daily send, it will for example also include hourly and frequent snapshots and they won't get removed by the deleting, as deleting of snapshots strictly adheres to the chosen interval.
* `exportPool`: Set to `y` if you want to export the pool after receiving is done. This can be useful if you want to make backups onto removable media such as usb thumb drives or external usb drives. Easysnap will automatically try to import the pool for receiving when it's not imported already. The export happens at the very end of routine.
* `remoteHost`: Provide the remote user and host that shall send the dataset, e.g. `root@myserver.tld` or you can also use like `root@localhost`
* `remoteDS`: The path to the remote dataset that shall be sent.
Expand Down
15 changes: 13 additions & 2 deletions easysnap
@@ -1,4 +1,4 @@
#!/usr/bin/env bash
#!/usr/bin/env bash

#------------------------------------------------------------------------------#
# #
Expand Down Expand Up @@ -55,7 +55,18 @@ makeBookmark() {
}

makeSnapshot() {
[[ "${dbs}" == *"m"* ]] && mariaDBLock
zfs snapshot "${localDS}@${now}" || _Info "Couldn't create snapshot on ${localDS}";
[[ "${dbs}" == *"m"* ]] && mariaDBUnlock

}

mariaDBLock() {
mysql -e "FLUSH LOGS; FLUSH TABLES WITH READ LOCK;" || _Info "Couldn't lock MySQL/MariaDB on ${localDS}"
}

mariaDBUnlock() {
mysql -e "UNLOCK TABLES;" || _Info "Couldn't unlock MySQL/MariaDB on ${localDS}"
}

removeLocalSnapshots() {
Expand Down Expand Up @@ -116,7 +127,7 @@ now=$(date "+%s_easysnap-${interval}_%Y-%m-%d_%Hh%M-%Z")
# We want to get the snapshot as close to the designated time as possible, so we first just make the snapshots
while read -r localDS property value source; do
# Split the value into number of snapshots and minimal free space
IFS=: read -r snaps reqFreeG <<<${value}
IFS=: read -r snaps reqFreeG dbs <<<${value}
# Put the dataset name into and min. free space into an array for checking free size at the end of the runtime.
(( reqFreeG > 0 )) && dsArr["${localDS}"]="${reqFreeG}"
# Check if snaps is a number
Expand Down

0 comments on commit aa27687

Please sign in to comment.