Linux Utility to clone disk reading parallel to write
using dd to clone disk takes double of the time due to the fact it write data after each block read; with clone-disk instead you can clone the disk in half time respect dd allowing each disk to run to maximum speed ( at the mean time at write speed of destination disk ). It works by 8 slots of 64MB each used by reader and writer; at first reader fill 8 slots without blocking itself and writer start when first slot available to write down and when finished it signal reader to reuse slot consumed while writer wait again for any of reader slot become available. Slots are filled and consumed sequentially and with each reader slot is associated a slot length to manage end of the disk chunks reading ( disk is not multiple of 64MB so last writing could smaller but sw also consider this case ).
- Requirements: Download NET Core SDK
- Install the tool:
dotnet tool install -g clone-disk
- To update if already installed:
dotnet tool update -g clone-disk
- if
~/.dotnet/tools
dotnet global tool isn't in path it can be added to your~/.bashrc
echo 'export PATH=$PATH:~/.dotnet/tools' >> ~/.bashrc
Note: disk must umounted
clone-disk <source> <dest>
Warning : double check source and destination device the program take a change to interrupt at start prompt pressing
ctrl+c
copy contents of source disk /dev/sdb to target disk /dev/sdc
clone-disk /dev/sdb /dev/sdc
You can discover which disk are source and destination in either ways:
-
watch at serial number
lsblk -o NAME,SERIAL,SIZE --nodeps
to identify which device is the source and which the destination -
alternatively, watch at
tail -f /var/log/syslog
to see which disk is source and destination by connecting disks after OS started. Connect source disk and write down device name then connect destination and do the same to fill program arguments correctly.
example of syslog ( note [sdc] name when disk attached )
[ 30.831533] sd 2:0:0:0: [sdc] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 30.831578] sd 2:0:0:0: [sdc] Write Protect is off
[ 30.831582] sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[ 30.831637] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 30.845045] sd 2:0:0:0: [sdc] Attached SCSI disk
Note : to plug disk at runtime may you need to configure AHCI mode, ESATA or HotPlug feature.
- disk size are detected by querying
/sys/class/block/devicename/size
Use vscode
Enter project folder and start vscode, then click on Restore popup to restore nuget packages
cd clone-disk
code .
Tune arguments in .vscode/launch.json
( "args" under "configurations" ) then Hit F5
You can test using loopback devices for a test purpose initializing these as follow ( replace XX, YY with free numbers watching at /dev/loop* files already allocated )
dd if=/dev/zero of=test1 bs=1M count=100
dd if=/dev/zero of=test2 bs=1M count=100
losetup -fP test1
losetup -fP test2
srcloop=$(losetup --list | grep `pwd`/test1 | awk '{print $1}')
dstloop=$(losetup --list | grep `pwd`/test2 | awk '{print $1}')
if [[ "$srcloop" == "" || "$dstloop" == "" ]]; then
echo "couldn't find some loop dev"
else
mkdir -p SRC
mkfs.ext3 $srcloop
mount $srcloop SRC
ls /etc > SRC/file.txt
umount SRC
clone-disk --non-interactive $srcloop $dstloop
mount $srcloop SRC
mkdir -p DST
mount $dstloop DST
diff SRC/file.txt DST/file.txt
if [ "$?" == "0" ]; then echo "tested successfully"; else echo "test ERROR"; fi
umount SRC
umount DST
losetup -d $srcloop
losetup -d $dstloop
rm -f test1 test2
fi
now you can safely test using /dev/loopXX /dev/loopYY arguments
- 0 : ok
- 3 : can't fit source with destination device size
root@bigone:/opt/clone-disk# /root/tmp-test-clone
retrieving device size [/sys/class/block/sdb/size] = 3907029168 ( x 512 bytes blocks ) = 2000398934016 bytes = 1.8 Tb
retrieving device size [/sys/class/block/sdc/size] = 3907029168 ( x 512 bytes blocks ) = 2000398934016 bytes = 1.8 Tb
source disk = /dev/sdb size = 1.8 Tb
dest disk = /dev/sdc
<=== read 17915904 bytes to bucket N. 0 read offset [1.8 Tb] speed = 87.8 Mb/s
===> write 17915904 bytes to bucket N. 0 write offset [1.8 Tb] speed = 87.8 Mb/s
*** FINISHED
For your own verification you can double check result by issueing and md5sum
dd if=/dev/sdb bs=512 count=$(cat /sys/class/block/<devicename>/size) | md5sum
example ( note : start in parallel from two terminals to avoid output mixing or redirect to a file )
$ dd if=/dev/sdb bs=512 count=$(cat /sys/class/block/sdb/size) | md5sum
3907029168+0 records in
3907029168+0 records out
2000398934016 bytes (2.0 TB, 1.8 TiB) copied, 20865.2 s, 95.9 MB/s
7d498cf3de1867f4c1a92fd00bb792a3 -
$ dd if=/dev/sdc bs=512 count=$(cat /sys/class/block/sdc/size) | md5sum
3907029168+0 records in
3907029168+0 records out
2000398934016 bytes (2.0 TB, 1.8 TiB) copied, 22016.3 s, 90.9 MB/s
7d498cf3de1867f4c1a92fd00bb792a3 -
After disk cloned it has same content and same GUID partition number ; for a backup purpose you can leave backup disk the same but for some other reason you could need to change GUID partition ; to do that use following
sgdisk -G <device>