-
Notifications
You must be signed in to change notification settings - Fork 12
Using shrink‐backup
To create a backup, letting shrink-backup decide appropriate size:
sudo /path/to/shrink-backup -a /path/to/backup.img
To update the backup img:
sudo /path/to/shrink-backup -U /path/to/backup.img
For further information, keep reading...
First, lets create a backup of a Raspberry pi os 64bit lite running on a Raspberry pi 4b 8GB.
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
I have a network share mounted at /mnt/backup with a directory called rpi4b8gb so there is where I will put the img file.
I suspect I will make some installs on the system and I prefer to keep my cache files after updates so the size of my used files will increase. Therefore I add 2GiB of space, ie 2048MB.
If I wanted to let the script decide a good size for the img file I would use the -a option.
I also want to exclude a directory I have in my home, so I opt to use exclude.txt to add that directory by using the -t option. Since I make use of the exclude.txt file the script will disable default exclude paths, so I have to make sure I exclude everything unnecessary.
Then I have yet another directory, but this one I want to be created, but the contents should be excluded.
I edit the exclude.txt file inside the same directory as I have the script, in my case /home/USERNAME/bin, direct path is preferable over ~/bin.
But before anything, we take a look at the system we are backing up with lsblk (I notice the sd-card is 14.6GB) and then make sure the script is executable.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
mmcblk0 179:0 0 14.6G 0 disk
├─mmcblk0p1 179:1 0 256M 0 part /boot
└─mmcblk0p2 179:2 0 14.3G 0 part /
$ pwd
/home/USERNAME
$ ls -l
drwxr-xr-x 2 USERNAME USERNAME 4096 Jul 28 22:23 bin
drwxr-xr-x 2 USERNAME USERNAME 4096 Jul 31 20:59 excluded_directory
drwxr-xr-x 2 USERNAME USERNAME 4096 Jul 31 21:14 excluded_directory_files
$ ls -l bin/
-rw-r--r-- 1 USERNAME USERNAME 37053 Jul 31 21:11 shrink-backup
$ chmod u+x bin/shrink-backup
$ ls -l bin/
-rwxr--r-- 1 USERNAME USERNAME 37053 Jul 31 21:11 shrink-backup
$ nano bin/exclude.txt
/lost+found
/proc/*
/sys/*
/dev/*
/tmp/*
/run/*
/mnt/*
/media/*
/var/swap
/home/USERNAME/excluded_directory
/home/USERNAME/excluded_directory_files/*
$ ls -l bin/
-rw-r--r-- 1 USERNAME USERNAME 166 Jul 31 21:16 exclude.txt
-rwxr--r-- 1 USERNAME USERNAME 37053 Jul 31 21:11 shrink-backup
I want to read what is happening in more detail afterwards, because, why not? So I include the -l option that will create a log file inside the bin directory (the same directory as I have the script in).
But I do NOT want to be bothered by silly questions like "are you sure you want to overwrite the file" and such if a file already exists so I also include the -y option. BE CAREFUL WHEN USING THIS OPTION!
$ pwd
/home/USERNAME
$ sudo ./bin/shrink-backup -tyl /mnt/backup/rpi4b8gb/rpi.img 2048
I can see that the backup is done:
## Backup done.
## Autoexpand filesystem at boot: true
## /mnt/backup/rpi4b8gb/rpi.img is 3.9G with 2048MiB extra space included.
## Exiting and cleaning up...
## Please stand by...
## Done.
## Elapsed time: 02.50
We can check on the img file to make sure everything looks correct.
$ ls -lh /mnt/backup/rpi4b8gb/
-rw-r--r-- 1 root root 3.9G Jul 31 21:28 rpi.img
Looks good. WAY smaller than the actual size of the sd-card witch is 14.6GB
But I want to see more in detail what happened, there was a lot of text passing on my screen!!
shrink-backup.log
2023-07-31 21:25:56 [INFO] - Debugging requested, writing to log file shrink-backup.log...
2023-07-31 21:25:56 [INFO] - Update existing img file selected, UPDATE=false
2023-07-31 21:25:56 [INFO] - Requesting size from resize2fs selected, RESIZE2FS_RUN=false
2023-07-31 21:25:56 [INFO] - Prompt for user confirmation, PROMPTS=false
2023-07-31 21:25:56 [INFO] - Auto expansion enabled, AUTOEXPAND=true
2023-07-31 21:25:56 [INFO] - Custom device path selected, CUSTOM_DEV_PATH=false
2023-07-31 21:25:57 [INFO] - -f selected by user, using exclude.txt
2023-07-31 21:25:57 [DEBUG] - Checking if EXCLUDE_FILE exists
2023-07-31 21:25:57 [INFO] - ./bin/exclude.txt does exist
2023-07-31 21:25:57 [DEBUG] - Executing make_img function
2023-07-31 21:25:57 [DEBUG] - 2 partitions detected, setting ROOT_DEV_PATH=/dev/mmcblk0p2 and BOOT_DEV_PATH=/dev/mmcblk0p1
2023-07-31 21:25:57 [DEBUG] - Calculating size for dd to cover bootsector in blocks (512B block size) and adding 256 to overlap into root
2023-07-31 21:25:57 [INFO] - BOOTSECTOR=532480 blocks - this is actually the startsector for the root partition
2023-07-31 21:25:57 [INFO] - DDBOOTSECTOR=532736 blocks
2023-07-31 21:25:57 [DEBUG] - Calculating TOTAL (space needed for files on root) by adding USED_SPACE and ADDED_SPACE
2023-07-31 21:25:57 [INFO] - USED_SPACE=1729224704 Bytes
2023-07-31 21:25:57 [INFO] - ADDED_SPACE=2147483648 Bytes
2023-07-31 21:25:57 [INFO] - TOTAL=3876708352 Bytes
2023-07-31 21:25:57 [DEBUG] - Calculating .img file size by adding BOOTSECTOR to TOTAL
2023-07-31 21:25:57 [INFO] - TRUNCATE_TOTAL=4149338112 Bytes
2023-07-31 21:25:57 [WARNING] - -y selected by user. prompts are disabled
2023-07-31 21:26:03 [DEBUG] - Removing /mnt/backup/rpi4b8gb/rpi.img
2023-07-31 21:26:05 [INFO] - Using dd to create bootsector, bs=512 count=532736
------------------------------------------------------------------------------
2023-07-31 21:26:19 [DEBUG] - Running: dd bs=512 count=532736 if=/dev/mmcblk0 of=/mnt/backup/rpi4b8gb/rpi.img conv=noerror,sync status=progress
532736+0 records in
532736+0 records out
272760832 bytes (273 MB, 260 MiB) copied, 14.2411 s, 19.2 MB/s
------------------------------------------------------------------------------
2023-07-31 21:26:21 [INFO] - Using truncate to resize img file to 3957MB
2023-07-31 21:26:23 [DEBUG] - Running: truncate --size=4149338112 /mnt/backup/rpi4b8gb/rpi.img
2023-07-31 21:26:24 [INFO] - Using losetup -P to loop /mnt/backup/rpi4b8gb/rpi.img
2023-07-31 21:26:24 [DEBUG] - Running: losetup -P /dev/loop0 /mnt/backup/rpi4b8gb/rpi.img
2023-07-31 21:26:26 [INFO] - Using sfdisk to remove root partition
2023-07-31 21:26:26 [DEBUG] - 2 partitions detected, setting LOOP0=/dev/loop0p2 and LOOP1=/dev/loop0p1
2023-07-31 21:26:26 [DEBUG] - Running: sfdisk --delete -f -q /dev/loop0 2
2023-07-31 21:26:28 [INFO] - Using parted to recreate root partition
2023-07-31 21:26:28 [DEBUG] - Running: parted -s -a none /dev/loop0 unit B mkpart primary ext4 272629760 100%
2023-07-31 21:26:30 [INFO] - Using mkfs.ext4 to format root filesystem
------------------------------------------------------------------------------
2023-07-31 21:26:32 [DEBUG] - Running: mkfs.ext4 /dev/loop0p2 -U 3ad7386b-e1ae-4032-ae33-0c40f5ecc4ac -L rootfs -F -v
mke2fs 1.46.2 (28-Feb-2021)
fs_types for mke2fs.conf resolution: 'ext4'
Discarding device blocks: done
Filesystem label=rootfs
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
236640 inodes, 946462 blocks
47323 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=970981376
29 block groups
32768 blocks per group, 32768 fragments per group
8160 inodes per group
Filesystem UUID: 3ad7386b-e1ae-4032-ae33-0c40f5ecc4ac
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
------------------------------------------------------------------------------
2023-07-31 21:26:34 [INFO] - Using e2fsck to check filesystem
------------------------------------------------------------------------------
2023-07-31 21:26:34 [DEBUG] - Running: e2fsck -y -f /dev/loop0p2
e2fsck 1.46.2 (28-Feb-2021)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
rootfs: 11/236640 files (0.0% non-contiguous), 34950/946462 blocks
------------------------------------------------------------------------------
2023-07-31 21:26:35 [INFO] - Creating temp directory and mounting img file
2023-07-31 21:26:35 [DEBUG] - Running: mktemp -d -t shrink_img-XXXXXXXXXX, TMP_DIR=/tmp/shrink_img-9EQhdKpjf1
2023-07-31 21:26:35 [DEBUG] - Running: mount /dev/loop0p2 /tmp/shrink_img-9EQhdKpjf1
2023-07-31 21:26:36 [INFO] - Separate boot partition detected, mounting boot inside img file
2023-07-31 21:26:37 [DEBUG] - Fetching boot mount path from fstab, BOOT_PATH=/boot
2023-07-31 21:26:37 [DEBUG] - Creating boot directory in img file, mkdir /tmp/shrink_img-9EQhdKpjf1/boot
2023-07-31 21:26:37 [DEBUG] - Running: mount /dev/loop0p1 /tmp/shrink_img-9EQhdKpjf1/boot
2023-07-31 21:26:38 [DEBUG] - Executing do_rsync function
2023-07-31 21:26:39 [INFO] - Backing up to /mnt/backup/rpi4b8gb/mnt/backup/rpi4b8gb/rpi.img
2023-07-31 21:26:39 [INFO] - Creating temporary file to store rsync output
2023-07-31 21:26:39 [DEBUG] - Running: mktemp -d -t rsync-XXXXXXXXXX, tmp_file=/tmp/rsync-6TAElCMPwH
2023-07-31 21:26:39 [DEBUG] - Running: rsync -ahvD --exclude-from=./bin/exclude.txt --exclude=/mnt/backup/rpi4b8gb/* --info=progress2 --stats --delete --force --partial --delete-excluded / /tmp/shrink_img-9EQhdKpjf1
------------------------------------------------------------------------------
2023-07-31 21:28:32 [INFO] - Rsync report:
Number of files: 43,927 (reg: 34,050, dir: 3,941, link: 5,936)
Number of created files: 43,607 (reg: 33,733, dir: 3,938, link: 5,936)
Number of deleted files: 1 (dir: 1)
Number of regular files transferred: 33,733
Total file size: 1.59G bytes
Total transferred file size: 1.56G bytes
Literal data: 1.56G bytes
Matched data: 0 bytes
File list size: 1.18M
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1.56G
Total bytes received: 683.85K
sent 1.56G bytes received 683.85K bytes 14.24M bytes/sec
total size is 1.59G speedup is 1.02
------------------------------------------------------------------------------
2023-07-31 21:28:32 [INFO] - Rsync done
2023-07-31 21:28:32 [DEBUG] - Executing expand_fs function
2023-07-31 21:28:33 [INFO] - Raspberry pi detected
2023-07-31 21:28:33 [DEBUG] - Creating expansion script /tmp/shrink_img-9EQhdKpjf1/expand-fs.sh
2023-07-31 21:28:35 [DEBUG] - Adding 'init=/expand-fs.sh' to /tmp/shrink_img-9EQhdKpjf1/boot/cmdline.txt
2023-07-31 21:28:35 [DEBUG] - Creating /tmp/shrink_img-9EQhdKpjf1/expand-fs.sh executable: chmod +x /tmp/shrink_img-9EQhdKpjf1/expand-fs.sh
2023-07-31 21:28:35 [INFO] - Raspberry pi filesystem autoresizing at boot
2023-07-31 21:28:36 [INFO] - Finalizing filesystem
2023-07-31 21:28:37 [DEBUG] - Unmounting boot partition: umount /tmp/shrink_img-9EQhdKpjf1/boot
2023-07-31 21:28:38 [DEBUG] - Unmounting root partition: umount /tmp/shrink_img-9EQhdKpjf1
------------------------------------------------------------------------------
2023-07-31 21:28:46 [DEBUG] - Running: e2fsck -y -f -v /dev/loop0p2
e2fsck 1.46.2 (28-Feb-2021)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
/lost+found not found. Create? yes
Pass 4: Checking reference counts
Pass 5: Checking group summary information
rootfs: ***** FILE SYSTEM WAS MODIFIED *****
43620 inodes used (18.43%, out of 236640)
10 non-contiguous files (0.0%)
29 non-contiguous directories (0.1%)
# of inodes with ind/dind/tind blocks: 0/0/0
Extent depth histogram: 37766/2
438699 blocks used (46.35%, out of 946462)
0 bad blocks
1 large file
33734 regular files
3940 directories
0 character device files
0 block device files
0 fifos
0 links
5936 symbolic links (5843 fast symbolic links)
0 sockets
------------
43610 files
------------------------------------------------------------------------------
2023-07-31 21:28:46 [INFO] - /mnt/backup/rpi4b8gb/rpi.img is 3.9G with 2048MB extra space included
2023-07-31 21:28:46 [INFO] - Img file created and backup done
2023-07-31 21:28:46 [DEBUG] - Cleanup function called with exit 0
2023-07-31 21:28:46 [DEBUG] - Removing loop0 in cleanup function: losetup -d /dev/loop0
2023-07-31 21:28:46 [DEBUG] - Removing temp directory in cleanup function: rm -rf /tmp/shrink_img-9EQhdKpjf1
2023-07-31 21:28:46 [DEBUG] - Removing temp file in cleanup function: rm /tmp/rsync-6TAElCMPwH
2023-07-31 21:28:46 [INFO] - Elapsed time: 02.50
2023-07-31 21:28:46 [DEBUG] - Exiting script
##############################################################################
There is a LOT of info in there, but to point out a few things:
2023-07-31 21:25:57 [DEBUG] - 2 partitions detected, setting ROOT_DEV_PATH=/dev/mmcblk0p2 and BOOT_DEV_PATH=/dev/mmcblk0p1
The script correctly identifies the boot partition, that is good.
2023-07-31 21:25:57 [WARNING] - -y selected by user. prompts are disabled
2023-07-31 21:26:03 [DEBUG] - Removing /mnt/backup/rpi4b8gb/rpi.img
Ooops, there was apparently a file there already. But since I selected not to be disturbed by silly questions, it over wrote the file without asking me. Oh well, hope there was nothing important in that img file.
ALWAYS BE CAREFUL WHEN USING -y OPTION!!!
272760832 bytes (273 MB, 260 MiB) copied, 14.2411 s, 19.2 MB/s
The script copied the whole boot partition plus a tiny bit more, very good.
2023-07-31 21:28:33 [INFO] - Raspberry pi detected
2023-07-31 21:28:33 [DEBUG] - Creating expansion script /tmp/shrink_img-9EQhdKpjf1/expand-fs.sh
2023-07-31 21:28:35 [DEBUG] - Adding 'init=/expand-fs.sh' to /tmp/shrink_img-9EQhdKpjf1/boot/cmdline.txt
2023-07-31 21:28:35 [DEBUG] - Creating /tmp/shrink_img-9EQhdKpjf1/expand-fs.sh executable: chmod +x /tmp/shrink_img-9EQhdKpjf1/expand-fs.sh
2023-07-31 21:28:35 [INFO] - Raspberry pi filesystem autoresizing at boot
Auto expansion seems to work witch is great news, so I don't have to use raspi-config to expand the filesystem after restoring this image.
I use my favorite tool to "burn" the img to another sd-card, insert that card into the raspberry pi and boot it up to make sure that everything actually works correctly. Writing it to the sd-card took about 5 mins thanks to the small size. The new sd-card is also smaller, only 7.4G, but this should be no issue since the img file is only 3.9G.
After the automatic reboot, I am presented with a login prompt. I connect to it via ssh again to make sure everything is as it should.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
mmcblk0 179:0 0 7.4G 0 disk
├─mmcblk0p1 179:1 0 256M 0 part /boot
└─mmcblk0p2 179:2 0 7.1G 0 part /
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 7.0G 1.7G 5.0G 26% /
devtmpfs 3.6G 0 3.6G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 1.6G 1.1M 1.6G 1% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
/dev/mmcblk0p1 255M 31M 225M 13% /boot
tmpfs 782M 0 782M 0% /run/user/1000
We can also see that the directories/files we excluded in exclude.txt are not there, or at least an empty directory like we requested.
$ ls -l
drwxr-xr-x 2 USERNAME USERNAME 4096 Jul 31 21:25 bin
drwxr-xr-x 2 USERNAME USERNAME 4096 Jul 31 21:14 excluded_directory_files
$ ls -l excluded_directory_files/
total 0
Everything seems to be ok, so I do some changes on the os and now want to do an update of my img. I achieve this by using the -U option.
I still want to use my excluded.txt file so I include -t in the options.
Same with the logfile, -l.
But this time I have learned my lesson about using the -y option so I remove that.
$ sudo ./bin/shrink-backup -Utl /mnt/backup/rpi4b8gb/rpi.img
## Debugging selected, writing to /home/USERNAME/bin/shrink-backup.log
## Updating /mnt/backup/rpi4b8gb/rpi.img
Do you want to continue? [y/n]
Sure, lets update the file so I press y.
sent 35.44M bytes received 5.32K bytes 3.38M bytes/sec
total size is 1.60G speedup is 45.26
## Enabling fs-autoexpand...
## Raspberry pi detected...
## Raspberry pi filesystem autoresizing at boot...
## Cleaning up...
## Please stand by...
## Backup done.
## Exiting and cleaning up...
## Please stand by...
## Done.
## Elapsed time: 00.51
Everything seems ok. Not many files needed to be updated in this case, mostly logs and stuff.
If I want more information I can ofc read the added entries in shrink-backup.log.
Now, lets go Bob Ross mode and do something crazy with this. Lets write the image to a usb stick, Raspberry pi os should be able to boot from a usb out of the box if you have a rpi4 at least (make sure you remove the sd-card from the rpi), otherwise you might need to do some changes.
But can I use an img file I created with the script to do it?
I have a 128G [114.5GiB] SanDisk 3.2Gen1 stick, Lets try!
First I "burn" the image to the usb-stick.
I remove the sd-card, plug in the usb-stick and boot up the rpi.
After powering on the rpi I have to wait for it to switch from sd-card to usb and do the resizing thing, but after a minute or so I can ssh into it.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 114.6G 0 disk
├─sda1 8:1 1 256M 0 part /boot
└─sda2 8:2 1 114.3G 0 part /
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 113G 1.7G 107G 2% /
devtmpfs 3.6G 0 3.6G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 1.6G 1.1M 1.6G 1% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
/dev/sda1 255M 31M 225M 13% /boot
tmpfs 782M 0 782M 0% /run/user/1000
We can see that the full size of the usb-stick is being used.
Making a backup from another device than mmcblk does not require anything extra from you. shrink-backup will look where root resides and back up that device. If a separate boot partition exists (like on rpi), it MUST be on the same device as root!
The location boot is mounted does not matter, shrink-backup will scan fstab to make sure the correct path is used.
This time I will use the -a tag to let resize2fs calculate an appropriate size instead of manually adding the 2GB.
Still using the log and exclude.txt.
$ sudo ./bin/shrink-backup -atl /mnt/backup/rpi4b8gb/rpi.img
## Debugging selected, writing to /home/USERNAME/bin/shrink-backup.log
dumpe2fs 1.46.2 (28-Feb-2021)
resize2fs 1.46.2 (28-Feb-2021)
##############################################################################
# A backup will be created at /mnt/backup/rpi4b8gb/rpi.img
# Total img size: 3297MB
# Resize2fs decide minimum (root partition): 3037MB
# Autoexpand filesystem on boot: true
# Using exclude.txt: true
# Write to logfile: true
##############################################################################
Do you want to continue? [y/n] y
WARNING!!! WARNING!!! WARNING!!!
/mnt/backup/rpi4b8gb/rpi.img
FILE ALREADY EXISTS!!!
Do you want to overwrite? [y/n]
This time I get prompted about the already existing img file, but I'm ok with overwriting it this time, so I press y to confirm.
When done I write the img file to an sd-card and plug it into the rpi.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
mmcblk0 179:0 0 7.4G 0 disk
├─mmcblk0p1 179:1 0 256M 0 part /boot
└─mmcblk0p2 179:2 0 7.1G 0 part /
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 7.0G 1.7G 5.0G 26% /
devtmpfs 3.6G 0 3.6G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 1.6G 1.1M 1.6G 1% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
/dev/mmcblk0p1 255M 31M 225M 13% /boot
tmpfs 782M 0 782M 0% /run/user/1000
And just for fun I once again wrote the img file to the usb-stick and booted from it.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 114.6G 0 disk
├─sda1 8:1 1 256M 0 part /boot
└─sda2 8:2 1 114.3G 0 part /
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 113G 1.7G 107G 2% /
devtmpfs 3.6G 0 3.6G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 1.6G 1.1M 1.6G 1% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
/dev/sda1 255M 31M 225M 13% /boot
tmpfs 782M 0 782M 0% /run/user/1000
Rember: A backup is not really a backup until you have restored from it