Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add automatic resizing of partitions #18

Open
shasheene opened this issue Nov 9, 2019 · 8 comments
Open

Add automatic resizing of partitions #18

shasheene opened this issue Nov 9, 2019 · 8 comments
Assignees
Labels
enhancement New feature or request

Comments

@shasheene
Copy link
Member

shasheene commented Nov 9, 2019

A recurring feature request is automatic resizing of partitions depending on the size of the destination drive.

Clonezilla supports this (#4). However, this feature is being treated as a separate task.

@shasheene
Copy link
Member Author

shasheene commented Dec 19, 2020

Clonezilla's "-r" argument can only be used to grow filesystems to fit into their partitions, it cannot shrink partitions or filesystems:

If the target disk size is larger than the source disk, you can try to use option "-k1" which will create the partition table proportionally in the target disk and turn on option "-r" to resize the filesystem in the partition automatically. This is useful to make use all of the target disk size.

Also from the Clonezilla FAQ:

Q: Why Clonezilla can NOT restore an image which is saved from a large drive to a smaller drive? Any workaround?
A: It's not easy to implement such a feature, since Clonezilla now is a partition "imaging" tool, by "imaging", it means clonezilla actually does not know the files themselves, clonezilla just knows where the used blocks are. Because of this reason, the target partition size must be equal or larger than the original one so that clonezilla can restore the used blocks on that partition. If the target partition size is smaller, then it will go wrong.
Unless Clonezilla has file-based function in the future. Maybe...
If you really still want to make this, check this FAQthe doc herethis forum or the program fsarchiver.

and another entry:

Q: Is that possible I can restore to smaller HDD when the image was taken from bigger one? Or to clone a larger disk to a smaller one?
A: This is always a challenge to Clonezilla because it's block-based tool, not file-based one.
Two possible solutions:

Method 1: If you are sure the used data blocks on the source disk are within the disk boundary of destination one, yes, it's possible to make that with the options "-k1" and "-icds" enabled in the expert mode.

Method 2: Use Clonezilla live >= 2.6.7-28 or 20200703-focal. For older version it does not always work.

  • First use GParted live to resize the source disk, and make all partitions to be equal to or smaller than the size of destination disk.
  • Then boot Clonezilla live.
    • For disk to disk, follow this.
    • For image to disk, follow this.
    • The only difference is, just remember to choose expert mode, and enable "-icds" (while do not choose any option about "-k*").

Given Rescuezilla's recent addition of Image Explorer (beta) support, there might be a (complex) pathway to implement this. However this doesn't sound like a great solution.

@rescuezilla rescuezilla deleted a comment from candrapersada Jan 3, 2021
@shasheene
Copy link
Member Author

Deleted comment that was a duplicate of this comment

@shasheene
Copy link
Member Author

shasheene commented Apr 20, 2021

Growing partitions is easy to implement. All Rescuezilla would need to do is restore the partition table and partclone backup, then use the correct programs to grow the partition and filesystem. For completeness, Clonezilla's implementation in growing the filesystem is found in sbin/ocs-resize-part. Here are the filesystem supported:

  • reiserfs (via /sbin/resize_reiserfs of reiserfsprogs package)
  • vfat|fat16|fat32 (/usr/sbin/fatresize from fatresize package)
  • ext2|ext3|ext4 (via /sbin/resize2fs of e2fsprogs package)
  • ntfs (via /sbin/ntfsresize of ntfs-3g package)
  • xfs (via /usr/sbin/xfs_growfs of xfsprogs package)
  • jfs (via mount -o remount,resize
  • btrfs (via btrfs filesystem resize max)
  • nilfs2 (via /usr/sbin/nilfs-resize of nilfs-tools package)

As best I can tell, Clonezilla doesn't first modify the partition table other than relying on a partition table proportional restore. Rescuezilla v2.2 implements this sbin/ocs-resize-part behavior faithfully, but achieving the ability to automatically grow and shrink the filesystems across all supported image formats would be much more useful.

One of the key features of FOG Project is the ability to restore to disks smaller than original. FOG Project mostly relies on partclone (like the industry-standard Clonezilla format that Rescuezilla uses) but uses different metadata so FOG Project images have only recently become supported by Rescuezilla. I have noticed that FOG Project images often have a modified sfdisk partition table backup, which presumably provides flexibility in how the partition table is restored. According to the FOG Project wiki describing its Single Disk - Resizable image format, it appears FOG Project's default settings is to make a dd backup of all partitions, then shrink the dd backup, then make a partclone backup of that data before finally deleting the unnecessary dd backup:

This is the default choice used by FOG as it works in most cases and allows for deployment to smaller size disks as well. It takes a copy of every partition on the disk, and resizes partitions that has excessive free space to a smaller size where possible. Each resizable partition will go through a "Resizing filesystem" process for each partition that is to be resized. This process can take some time depending on how severe disk fragmentation is. The partitions that are shrank will be shrunk down to only 2GB of free space on their partition. This allows an image taken from a 6TB drive with only 20GB of used space to be deployable to a drive with a total capacity of 25GB roughly. When the partitions are laid onto the destination drive, all resized partitions are intelligently expanded to utilize the entire drive.

Given Rescuezilla's primary goal is to be fully compatible with the industry-standard Clonezilla format, it does not make sense for Rescuezilla generate backups uses the exact approach taken by FOG Project, because custom logic needed to expand the images won't be available in Clonezilla, and would therefore break Rescuezilla's full Clonezilla interoperability. Also because Rescuezilla supports all the partclone-based imaging formats (Clonezilla, Redo Backup and Recovery, FOG Project, Redo Rescue and Foxclone), almost all input images will not be pre-shruken. So a different approach to FOG Project is required.

So for deploying non FOG Project images, it may be possible to stream the output of partclone into the relevant filesystem resizing application, but this relies on piping the output between applications. Given streams do not provide random access this may not be easy. Side note: I need to further investigate how FOG Project is able to pre-shrink images.

Right now the cleanest approach might be associating the partclone image to a block device using partclone-utils and either operating on the block device using the shrink filesystem commands, or mounting the block device and copying files to the destination. The advantage of this approach is it would exercise some code paths that gets tested widely with Rescuezilla Image Explorer, including the planned indexed compression archive functionality. This approach will of course first require a lot of work to make partclone-utils more robust.

shasheene added a commit to shasheene/rescuezilla-dev that referenced this issue May 7, 2021
Always try to grow the filesystem to fit the partition size, which is the
behavior of Clonezilla's "-r" option used by default in its restoredisk and
restoreparts function. Installs missing fatresize package as the other packages
listed in [1] are already installed.

Some filesystems (XFS, JFS, BTRFS and NILFS) requires mounting before resizing,
and some (but not all) do a filesystem check step first. This commit ports
Clonezilla's ocs-resize-part script to Python, so the exact approach used in
Clonezilla is replicated here.

[1] rescuezilla#18 (comment)
shasheene added a commit to shasheene/rescuezilla-dev that referenced this issue May 7, 2021
Uses the original FOG Project sfdisk partition table backup rather than FOG
Project's shrunken sfdisk file.

Clonezilla's ability to grow filesystems maps nicely to the original partition
table backup, but implementing resizing the shrunken partition table before
restoring is more complex task that is not immediately required.

rescuezilla#18 (comment)
shasheene added a commit that referenced this issue May 7, 2021
Always try to grow the filesystem to fit the partition size, which is the
behavior of Clonezilla's "-r" option used by default in its restoredisk and
restoreparts function. Installs missing fatresize package as the other packages
listed in [1] are already installed.

Some filesystems (XFS, JFS, BTRFS and NILFS) requires mounting before resizing,
and some (but not all) do a filesystem check step first. This commit ports
Clonezilla's ocs-resize-part script to Python, so the exact approach used in
Clonezilla is replicated here.

[1] #18 (comment)
shasheene added a commit that referenced this issue May 7, 2021
Uses the original FOG Project sfdisk partition table backup rather than FOG
Project's shrunken sfdisk file.

Clonezilla's ability to grow filesystems maps nicely to the original partition
table backup, but implementing resizing the shrunken partition table before
restoring is more complex task that is not immediately required.

#18 (comment)
@shasheene
Copy link
Member Author

shasheene commented May 7, 2021

The FOG Project implementation of the shrinking feature described in the previous comment is a very inefficient and roundabout way that makes both the backup and restore operations really slow. It's much more efficient to implement the feature as follows:

  1. Fix remaining problems in the partclone-utils project (which will also improve the beta Image Explorer’s low-compatibility)
    • This will unlock the ability of any partclone image to be expressed as a regular block device
  2. Create the filesystem on the destination disk (with the same partition flags as the source, as usual), mount the partclone-utils block device and the destination filesystem, and simply copy files in using a tool like rsync.
    • The biggest "gotcha" I foresee is the unique UUID identifiers will be regenerated and will need to be carefully handled to avoid causing compatibility issue compared to existing block-level approach.

Despite all the work it will require, the payoff seems worthwhile: automatic resizing when restoring or cloning to disks of smaller capacity to the original will be a powerful feature.

@mzky
Copy link
Contributor

mzky commented May 28, 2021

I have tried to restore the system to a small hard disk using the following methods

Backup system:

tar Pzcfp backup.tgz --exclude=/proc --exclude=/mnt --exclude=/lost+found --exclude=/sys --exclude=/media --exclude=/run --exclude=/boot --exclude=/dev --exclude=/var/spool/postfix /

Restore the system:
By starting the U disk system, manually partition the disk, and decompress the system backup files (About 3-5 minutes)

tar zxf backup.tgz

Repair GRUB :
Populate GRUB with the obtained UUID:

blkid 

vim /boot/grub/grub.conf
root="obtained UUID"

vim /etc/fstab
uuid="obtained UUID"

Under the Centos7 system, it can be used normally.

@shasheene
Copy link
Member Author

shasheene commented May 28, 2021

Exactly. The backup.tgz that you describe is an archive of all files on your system, and the approach you describe involves partitioning the disk, copying the files in and then modifying GRUB bootloader.

This is the approach I will implement for Rescuezilla. It uses partclone-utils to mount the backup image's existing partclone files before doing any copying, so that backups created by any supported image format automatically become able to be restored to disk smaller than original.

(This feature is something that no other open-source disk imaging and cloning tool has yet implemented. Not even Clonezilla. FOG Project's roundabout implementation of this feature using "Single Disk - Resizable image format" is way too slow and doesn't use this approach)

@alphanis
Copy link

alphanis commented Feb 1, 2024

It will be great if Rescuezilla support this feature. I think Foxclone does, according to its manual page 23. https://foxclone.org/downloads/foxcloneV51.pdf

@shasheene
Copy link
Member Author

@alphanis Foxclone appears to only support this with the same manual workaround as the current version of Rescuezilla. From that project's manual:

Screenshot_2024-02-01_20-20-17

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Development

No branches or pull requests

3 participants