Implement new YUM backup method (based on ZYPPER) #1464
Conversation
…PER functionality for YUM-managed systems)
…le required for BACKUP=YUM feature
|
@N3WWN Because I made the current (rather initial and experimental) I added the other ReaR upstream members as reviewers Regarding your other PR which adds a file backup option |
|
@N3WWN Debug "WARNING: ..." I wonder if a warning as a debug message makes much sense. |
|
@N3WWN In general instead of things like if [[ "$CLONE_ALL_USERS_GROUPS" =~ ^[yY1] ]]; then use if is_true "$CLONE_ALL_USERS_GROUPS" ; then cf. "Relax-and-Recover functions" at For me it looks af if you misunderstood or misuse how |
|
Nice! git log -p --follow usr/share/rear/rescue/default/900_clone_users_and_groups.sh tells the truth. |
|
@N3WWN YUM backup method which will recreate a RPM-based system that uses the yum package manager in the same manner as the ZYPPER backup method The "recreate" therein might be a misunderstanding about the See the new BACKUP=ZYPPER method is intended to ... "install a system from scratch" . . . also use ReaR generically for all kind of installations . . . ReaR system deployment and |
|
Looks like I can't respond to each msg, so I'll have to do so all at once :)
Though I wasn't aware of KIWI before now, I could definitely see this as an option using BACKUP_PROG_INCLUDE/BACKUP_PROG_EXCLUDE! My goal was to be able to have as small of a backup ISO as possible but with a complete system restore. The versions of the packages may change since we're installing from YUM repos, but I haven't seen it as a problem with the CentOS 6.x and CentOS 7.x testing that I've done, purposely using older 6.x and 7.x VMs that were restored with the latest RPMs from the public repos. I make VMs a LOT for supporting our customers and would like to be able to take a full system backup without storing all of the files that are the same as what is in the RPMs so that I can destroy the VM and have a minimal-size backup. I used to use ReaR to make full backups:
That method took 2.7GB of dynamic disk and turned it into about a 1.4GB ISO. The same 2.7GB of dynamic disk, but with BACKUP=YUM is now a 210MB ISO. :)
Those were in the original files that I modified, but can certainly strip them out or alter them to be included in the log only. We can keep the history of those Debug messages between ourselves. ;)
The I saw that the users/groups were being cloned into the rescue system and just carried that forward to cloning from the rescue system into the restore system. My reasoning being that, since the system was installed from scratch via RPM, which causes the user and group IDs sometimes change (they did in my testing), the files will be restored with the wrong UID and/or GID during the file restore (which will be included a later PR). Instead of completely reinventing the wheel, I just painted it a different color. :)
Would it be better to say As it sits now, with the exception of the cloning of users and groups, I think that the YUM method is behaving the same as the ZYPPER method (but without the repository specification), but I may be missing something as I've not used ZYPPER itself very much. Thanks for all of the feedback so far! I'm excited to contribute and am thrilled that you seem to be equally excited about my contribution! :) Just let me know how you'd like me to proceed... |
|
@N3WWN Your use case (get the backup smaller) is very interesting! Regarding clone_users_and_groups I did right now Regarding using the CLONE_USERS CLONE_GROUPS |
|
@N3WWN One main part with all what is strictly additional The rest which might possibly change any existing behaviour Because I assume your changes in It seems your changes in |
|
@N3WWN |
|
I was wondering if I should have broken up the PRs into much smaller bites, and I guess I should have. ;) I believe that we should be able to add file backups to ZYPPER by symlinking to the appropriate files in the YUM tree. IIRC, I kept the file backups and RPM backups segregated completely since I planned on separate PRs for those two segments of the feature. The changes to usr/share/rear/rescue/default/900_clone_users_and_groups.sh look good and appear to address the issues I saw with duplicate users and groups.
For the new configuration item, RECREATE_USERS_GROUPS=yes/no, do you think that would be best set up per backup method? i.e. YUM_RECREATE_USERS_GROUPS and ZYPPER_RECREATE_USERS_GROUPS? Would it be appropriate to duplicate the code from usr/share/rear/rescue/default/900_clone_users_and_groups.sh in usr/share/rear/restore/YUM/default/405_clone_users_and_groups.sh (basically as it is now) but using the new RECREATE_USERS_GROUPS config item? Or would it be preferred that this code be extracted as functions in a library? Duplicating the code will allow us to more easily merge the main YUM feature code as it won't touch any existing code, but, by definition, duplicates code in a project.
Yes, not a problem at all. I'm totally on-board with ongoing support of the changes. Do I need to "unpull" #1464 or anything to break it into 3 distinct PRs (YUM, usr/share/rear/rescue/default/500_ssh.sh and usr/share/rear/rescue/default/900_clone_users_and_groups.sh)? |
The issues derived from how/where user IDs and group IDs are stored and how they differ on the three systems involved: the source system, the rescue/recovery system and the target system. Let's say that we have 2 system users sshd and postfix (users randomly picked, this issue may not show up with these particular users, especially since these users may have predefined UIDs on many systems). On the source system, postfix was installed with the OS and ssh was added later, so let's say that postfix has a UID of 80 and sshd has a UID of 90. From the tar info pertaining to the
This means the archive contains stuff like /etc/ssh/sshd_config with a user of sshd and a UID of 90 and /var/lib/postfix/master.lock with a user of postfix and a UID of 80. On the target system, we reinstall the RPMs and the UIDs no longer match... let's say sshd is now UID 80 and postfix is now UID 90. We perform the file restore on the rescue/recovery system, which thinks these users have their original IDs, funny things can happen, even if we chroot into the target system to do the file restore. I fought with this for days; only a few files would have incorrect ownership and/or SELinux contexts after an otherwise successful restore. Usually, the system would be usable, but I just wasn't content with having the differences. rear/usr/share/rear/restore/YUM/default/405_clone_users_and_groups.sh actually excludes /etc/passwd, /etc/group and /etc/shadow from being restored. We still include them in the archive so we can see the contents later if we need to reference them, though. The reason for excluding them from the restore goes along with these other issues... if any UID or GID change or are otherwise different between the source and target systems, the UID and GID on the target system disk won't match the UID and/or GID specified in the files in /etc. Let me know if my rambling is making sense or if I need to dig up the exact circumstances where the issues arose. |
|
To get your YUM backup method merged, just leave out the Regarding symlinking YUM and ZYPPER scripts: I would prefer to wait with symlinking YUM and ZYPPER scripts Regarding "double-clone-users-and-groups": Many thanks for your descriptive and explanatory If I understand it correctly wrong numerical user IDs Accordingly - as far as I see probably even by default - the In particular for the backup methods YUM and ZYPPER I would implement a generic RECREATE_USERS_GROUPS I would recommend to implement RECREATE_USERS_GROUPS Instead of restore/YUM/default/405_clone_users_and_groups.sh # 395_recreate_users_and_groups.sh
#
# Recreate the same users and groups that exist
# in the ReaR recovery system also in the target system.
# Skip if not needed.
# For the 'test' one must have all array members
# as a single word i.e. "${name[*]}" because
# it should succeed when there is any non-empty
# array member, not necessarily the first one:
test "${RECREATE_USERS_GROUPS[*]}" || return
I think the script number must be less than the script number Check the "rear -s recover" output in which ordering the Alternatively if NNN_recreate_users_and_groups.sh In any case also the implementation of a generic Until a generic RECREATE_USERS_GROUPS functionality |
|
👍 to splitting PRs into different topics. This discussion made me curious about one detail: Why do our users so far not report lots of problems with restoring files while the users and groups are not defined? My only guess is that all tools behave like tar: If the user name is not known then simply set the user id (which is the only thing that is stored in the file system in any case) and hope for the best. As all backup strategies will both restore With both the YUM and ZYPPER recovery methods this strong correlation between user/group database and file content is on purpose broken. The user/group database is recovered via RPM and then some files are recovered from a backup. In my opinion this discrepancy is the root cause for these problems that you are describing here. ReaR has always prided itself in correctly separating between the rescue system and the recovered target system. The current implementation of YUM and ZYPPER also breaks with this separation and the complexity you describe here is a result of that. I can imagine a slightly different approach that might be easier to follow:
The main difference is running the restore of the additional files in the context of the system that was installed via RPM. There of course the tar logic of first looking up users in the database and then falling back to IDs works as intended. If the UIDs will have changed in the recovered system then the restored files will also get the new UIDs. This for sure works well for all users and groups created in RPM package scripts. If you want to also handle manually created users then I suggest to add the users that are missing in the target system after installing all the RPMs instead of creating all users beforehand. The whole point of this exercise is maintaining a strict separation of concerns between files (and information) governed by different owners:
To summarize: I have the impression that if YUM and ZYPPER use chroot where applicable that then the users and groups in the rescue system don't matter. About the actual installation of the chroot via yum, I haven't done this myself before but I remember building chroot environments with steps like the ones mentioned in http://blog.parahard.com/2011/06/five-steps-to-create-fedora-chroot-jail.html, especially the part of initializing the RPM DB, installing some basic RPMs via rpm (and not yum) and putting some files into the chroot to make the scripts happy. Could you maybe also let us know why you don't use existing tools like https://github.com/dozzie/yumbootstrap or https://github.com/rpm-software-management/mock which already contain a lot of expert knowledge about creating a fresh RPM-based setup in a chroot? |
|
@N3WWN @schlomo |
|
Thanks for the info on how to update the PR to exclude the common files. I'm still testing 900_clone_users_and_groups.sh for both "YUM" and "YUM+file backup" but hope to get the PR updated before the weekend.
Right now I'm symlinking to some ZYPPER files, but will remove the symlinks and create copies of the files before I update the PR so they exist in isolation until they stabilize.
I think the problem stems from having a system that has dynamic UID/GID assignment during package installation. Source system has UIDs/GIDs assigned dynamically when it was built. Those UIDs/GIDs are copied over to the ReaR rescue/restore system. Target system has UIDs/GIDs assigned dynamically when it is built by the ReaR rescue/restore system and there exist files on the target filesystem which adhere to those UIDs/GIDs. We now have two databases with dissimilar assignment of those UIDs/GIDs within one running OS. The tar archive also has a mapping of username and group to UID and GID. If the ReaR rescue/restore system performs the restore natively (non-chroot), tar will use the ReaR rescue/restore system's UID/GID mappings when writing the files. If the ReaR rescue/restore system performs the restore via chroot within the target environment, tar will use the target system's UID/GID mappings when writing the files. This is almost perfect, except when the user or group doesn't exist and tar has to fall back on the UID or GID stored in the tar archive. What happens then is that the files are restored with the wrong UID and/or GID. The double-clone-users-and-groups should take care of this by ensuring that all users and groups that exist on the source system will exist (well, should exist) on the target system when the files are restored.
I think because it requires a perfect storm of using tar (or other backup method that stores both usernames/groups and UIDs/GIDs and falls back to UID/GID if the username or group are missing) and a system rebuilt from a method that does so from scratch using packages (ZYPPER or YUM). Since I ran into it in my testing, I worked on a solution to resolve it should anyone else encounter it, regardless of how often it may actually occur. 🎲
I suppose there may exist other tools that may be a better solution at this time... I haven't looked, honestly, since I have been advocating ReaR within my organization since I first fell in love with it many months ago. When I saw the brand new functionality that ZYPPER provides, I saw an opportunity to contribute a similar YUM feature (based heavily on the ZYPPER code) and thought I'd add a file backup capability, also. The double-clone-users-and-groups solution is actually only required for the file backup capability, so I could exclude it from the initial YUM PR, also, if that would be better. Then, YUM and ZYPPER will operate almost identically.
I understand where you're coming from and going with this and will attempt to get this done ASAP, also. 😁 Thanks! |
|
Regarding This is a very interesting new functionality for ReaR ... let AutoYaST install all RPM software packages of the complete system and afterwards restore a backup that only contains those files that are added or changed compared to the original files in the RPM software packages. In this case the restore script could run as a so called "chroot script" (with "chrooted=false" or "chrooted=true" as needed in the particular case). Backup restore via chroot within the target system For third-party backup software backup restore In contrast backup restore in the ReaR recovery system Therefore from my current point of view I would first and In general regarding backup restore for YUM and ZYPPER it is not an usual file-based backup/restore method ... During "rear mkbackup" it will basically only save a list of installed RPM packages ... during "rear recover" it will basically only install those RPM packages as pristine RPM packages ... Any other files that should be restored (e.g. prepared config files or /home directories) must be done by "Using Multiple Backups for Relax-and-Recover", see doc/user-guide/11-multiple-backups.adoc Now as I read this again I wonder meanwhile @N3WWN I have a question: |
…scue/default/900_clone_users_and_groups.sh to ReaR upstream versions
… earlier ReaR upstream version to fix PR
There was a problem hiding this comment.
Now it looks good:
https://github.com/rear/rear/pull/1464/files
shows only additions but no changes
of existing code.
|
@N3WWN |
…es of those files
…/default/400_prep_yum.sh, usr/share/rear/restore/YUM/default/400_restore_packages.sh and usr/share/rear/restore/YUM/default/405_clone_users_and_groups.sh to avoid unintended expansion. Update usr/share/rear/restore/YUM/default/405_clone_users_and_groups.sh to use is_true "$CLONE_ALL_USERS_GROUPS" due to recent changes to that variable. Improve storing and usage of releasever in usr/share/rear/prep/YUM/default/400_prep_yum.sh. Copy all repo files from source to target. Improve error handling and package name fall back during package installation.
In my dev branch that contains the yet-to-be-submitted YUM+backup PR, I used the NETFS backup workflow as my starting point, but forced the use of tar. Now that I'm becoming more and more familiar with the inner workings of ReaR, I see that I may be able to alter how the backup is performed to use whatever other file backup methods ReaR supports. Regarding the lack of any mention of 'tar', that's because that'll be in a future PR 😁 Sadly, the somewhat interconnectedness of usr/share/rear/rescue/default/900_clone_users_and_groups.sh between YUM and YUM+backup have blurred the lines between those two in the discussion. With the future work that I'll do for RECREATE_USERS_GROUPS and 395_recreate_users_and_groups.sh, which I think will be entirely within the YUM+backup PR, that interconnectedness will cease, at least for the time being. Merging on Monday sounds great. I think I've finished the changes that I'm going to commit for this PR, so it should be stable between now and Monday. Thanks and have a great weekend! |
|
@N3WWN test "$ZYPPER_ROOT_PASSWORD" && root_password="$ZYPPER_ROOT_PASSWORD" perhaps you meant to use YUM_ROOT_PASSWORD? |
|
Regardless of minor issues like |
|
Thank you sir! 😀 I'll get the <X>_ROOT_PASSWORD thing sorted out and will submit a new PR for it. |
|
@N3WWN FYI: Because one cannot have two BACKUP=... at the same time But the plain YUM and ZYPPER RPM package installation To only get some configuration files coiped into the Therefore my idea to only get some files coiped into I.e. my idea is to add something like a |
YUM+backup adds the ability to backup and restore files to the YUM method. This is a continuation of the YUM backup method, cf. #1464 The YUM method, like the ZYPPER method, recreates the target system from scratch using a list of RPMs captured from the source system. The only files backed up via YUM+backup are those which were either not provided by RPM packages or were provided by RPM packages, but have been modified. This minimizes the number of files included in the backup while still capturing every file on the system not explicitly excluded using options like BACKUP_PROG_EXCLUDE, EXCLUDE_MOUNTPOINTS, etc. For details see the new configuration options for the YUM method in conf/default.conf and the new documentation of the YUM and ZYPPER methods in doc/user-guide/14-ZYPPER-and-YUM.adoc
Hello @N3WWN, according to the diff, there are actually many files that could have been symlinked, because they contain no significant changes. Those are: Why have you converted those symlinks to copies? I am asking because I am doing a cleanup and having to change so many identical files is not very productive. |
|
@pcahyna Since a longer time there have been almost no further changes In particular I noticed zero user interest about my ZYPPER method In contrast the YUM method was further developed Accordingly when there are duplicated scritps |
|
Oops - I think I misunderstood It is not about duplicates of YUM and ZYPPER scripts. It is about YUM scripts that are basically copies of other scripts. |
|
@jsmeix indeed most of the files I am concerned with are not from ZYPPER but from NETFS (and were converted to copies in commit 3d9cc24). |
|
@pcahyna |
Hi!
This is my first major commit to this project, so I hope that I'm doing this correctly.
This PR is to add a YUM backup method which will recreate a RPM-based system that uses the yum package manager in the same manner as the ZYPPER backup method.
I symlinked files where possible and copied and updated files when changes were necessary for the YUM method.
I have also updated a couple of existing files to [hopefully] improve them. The primary changes which could likely be made common to all methods are:
usr/share/rear/rescue/default/500_ssh.sh
usr/share/rear/rescue/default/900_clone_users_and_groups.sh
usr/share/rear/restore/YUM/default/405_clone_users_and_groups.sh
usr/share/rear/restore/YUM/default/940_generate_fstab.sh
If you'd prefer that I submit separate PRs for each, please let me know.
Upon acceptance of the YUM method, I will submit another PR which adds a file backup option to the YUM method. This will recreate the system from RPM and restore any non-RPM-originated files or RPM-originated files which may have changed (or been removed).
Let me know if there's a better way for me to submit changes of this type or if there's anything I can do to help with the acceptance of this PR.
Thanks!
-Rich Alloway (Rogue Wave)