Support full system backup and bare metal restore

[dinoboy197]

It would be immensely helpful to include the ability to have Back In Time backup absolutely everything (grub, system files, programs, user files, etc) and enable bare metal restore.

Maybe its something I could help out with?

[Germar]

Help is always very welcome! 👍

BIT only works on file level. So to backup grub we'd need an image first (e.g. with dd). This can be done with a user-callback script. Take a look at bit-team/user-callback for examples.

All the rest can be done with BIT already. For sure, there is no One-Click restore feature yet. You'd need to create partitions and manually write back grub.

There is also bug #58 making troubles when restoring from LiveCD.

[dinoboy197]

BIT only works on file level. So to backup grub we'd need an image first (e.g. with dd). This can be done with a user-callback script. Take a look at bit-team/user-callback for examples.

Got it, makes sense. I'm going to try to use BIT for the first time in a couple of days (excited - had mostly used a bunch of rsync and duplicity scripts in the past) and then will report back. Maybe we can come up with some steps involved in putting together some bare metal restore ability.

[Germar]

That would be great!

Here's a list on what would need to be done for full featured bare metal restore (feel free to extend that):

1. boot-loader backup and restore
2. partition-table backup and recreation of partitions
3. format partitions (optional)
4. rebuild /etc/fstab if UUIDs where in use
5. ability to load different /etc/passwd and /etc/group to fix bug Restore the correct file owner and group fail if they are not present in system #58

[dinoboy197]

Awesome, thanks. I've been busy with other things to start on this - but look forward to taking a swing sometime. Your list of items sounds perfect - that's what I was thinking. Will circle back soon.

[dinoboy197]

OK, I've been doing some brainstorming and realize that there are many. many potential features that could be included in such a project.

So that this does not become a huge single herculean effort (with final value delivery only at the very end), I thought it might be good to break the work up into phases, delivering some user value with each feature. I'm putting together a quick phase document now; will share in the next day or two for you to review. Stay tuned!

[Germar]

Sounds great! 👍

[dinoboy197]

OK, take a look at this. Hopefully this will start to bring whatever ideas we have in both of our heads into alignment (they might be quite different at the moment!)

If you try to comment / edit, I'll give you permission based on the email that you use, but will restrict for now so the internet masses don't take over this doc and vandalize it.

[dinoboy197]

Thanks for your comments. I'm going to get started on some work and will send some code your way when I start to have some things. Will likely ask you again for pointers that are BIT specific.

[dinoboy197]

I've made some modifications to a local branch that I'm attempting to test. I'm running sudo -i ./backintime-qt4 to open the UI, create a full system backup profile (which works perfectly) and then click the "Take Snapshot" button to test out a full system backup.

However, from what I can see, "Take Snapshot" forks a local system process which runs code in the pre-installed version of backintime, rather than running my local code modifications in the current directory.

How do you test out local modifications without doing a full install of your source tree into /usr/bin and /usr/share/backintime (as I'd rather not hack around in the pre-installed packages if possible)? If there is some dev instruction document that I'm not seeing that you can point me at, I'll just follow that.

[Germar]

Please use pkexec backintime-qt to start GUI as root.

I used to test only in several VMs where I mount my dev path via sshfs, create new .deb packages (makedeb.sh) and install them with dpkg. This is quite slow but it was the only way until I fixed launching BackInTime from source recently. But I'm planing to move to the same procedure as you used.

I also stumbled over this problem (calling backintime backup from /usr/bin instead of source) last week during adding integration tests. I think it would be best to insert source path to $PATH environ if started from there. Hang on, I'll add this right now.

There is no documentation at all, sorry.

[Germar]

Should work now.

btw it would be great if you could add unittests in common/test/ for new methods you add. I just started to use unittests (there where some French students last year who contributed this). There are still a lot to add. But I think it's easier to write those tests directly after writing the main method because you still now what it was supposed to do and don't need to dig around first.

You can run this tests with

cd common
./configure
make test

[dinoboy197]

That helps a lot, thank you!

Yes, I will certainly add unit tests.

Please use pkexec backintime-qt to start GUI as root.

Turns out this actually doesn't work for me. I had tried it when I first started but couldn't get it to run, which is why I've been using sudo -i backintime-qt4 to launch the BIT UI. Below is sample output from running pkexec (I was prompted for my password each time and entered it, then hit ok before the errors occur):

traack@lune:~/Programming/github/dinoboy197/backintime (master)$ git fetch upstream
traack@lune:~/Programming/github/dinoboy197/backintime (master)$ git merge --ff-only upstream/master
Already up-to-date.
traack@lune:~/Programming/github/dinoboy197/backintime (master)$ ls
AUTHORS  common  LICENSE     makedeb-src.sh   qt4        TODO          updatecopyright.sh  VERSION
CHANGES  debian  makedeb.sh  make-tarball.sh  README.md  TRANSLATIONS  updateversion.sh
traack@lune:~/Programming/github/dinoboy197/backintime (master)$ cd qt4
traack@lune:~/Programming/github/dinoboy197/backintime/qt4 (master)$ ls
app.py                       docbook           net.launchpad.backintime.policy                 qt4systrayicon.py  snapshotsdialog.py
backintime-qt4               icon.py           net.launchpad.backintime.serviceHelper.conf     qt4tools.py
backintime-qt4.desktop       logviewdialog.py  net.launchpad.backintime.serviceHelper.service  restoredialog.py
backintime-qt4-root.desktop  man               plugins                                         serviceHelper.py
configure                    messagebox.py     __pycache__                                     settingsdialog.py
traack@lune:~/Programming/github/dinoboy197/backintime/qt4 (master)$ pkexec backintime-qt4
Error executing ./backintime-qt4: No such file or directory
traack@lune:~/Programming/github/dinoboy197/backintime/qt4 (master)$ pkexec ./backintime-qt4
Error executing ./backintime-qt4: No such file or directory
traack@lune:~/Programming/github/dinoboy197/backintime/qt4 (master)$ pkexec /home/traack/Programming/github/dinoboy197/backintime/qt4/backintime-qt4

Back In Time
Version: 1.1.9

Back In Time comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; type `backintime --license' for details.

No protocol specified
app.py: cannot connect to X server :0.0
traack@lune:~/Programming/github/dinoboy197/backintime/qt4 (master)$

You certainly don't need to troubleshoot this for me, since I can launch the UI another way, but thought you might have some clever idea about what's going on.

[Germar]

Oh, sure. My fault, sorry.

pkexec needs a policy in /usr/share/polkit-1/actions. Take a look at qt4/net.launchpad.backintime.policy. So unless you'd create a custom policy for this, pkexec /path/to/backintime-qt4 will never work. But sudo -i should be fine for testing, too.

[Germar]

@dinoboy197 I'm planing to release 1.1.10 on January 9th. Is this okay for you? Would you like to merge something more before next release?

[dinoboy197]

Go right ahead! I'm cooking up some additional parts when I find time, but don't hold up your relase on their account. I'll just send PRs when they are ready and tested.

[PMarxen]

Before you make multiple work have a look at http://sourceforge.net/projects/systemback/
A combination with BackInTime would be fine, but it looks like a lot of work...

Maybe parts for the SystemBackup can taken for Back in Time if there is a smart way for integrating in Software and Interface.

[emtiu]

In my opinion, this feature would be out-of-scope, and very difficult to implement, given all the different types of bootloading environments and partitioning configurations (BIOS vs. UEFI, MBR vs. GPT, ESPs with boot loaders vs. boot managers. A full-disk-imaging solution is probably much more suited for this than BackInTime.

[mYnDstrEAm]

Much support for this feature request. It would be very useful and it's really important. I would find it best if there was one backup tool that did everything, basically a best backup tool and one tool instead of things being splintered into one tool each for countless backup purposes and types.

Basically, it's also about making things as easy and fail-safe and quick as possible (including selecting which backup tool to install), so you have all your files backed up and the barrier to running full backups is low even for people who don't like to or can't tinker around...you simply have an option to run a full backup and an easy process for restoring it.

I also asked about this at the issue linked just above this comment, take a look.

I don't think this would be or should be out-of-scope in principle - maybe just as of right now because there are more important issues (but maybe somebody else could create pullrequests nevertheless or this gets worked on in the future).

For the difficulty of implementation, it wouldn't have to be perfect, just better than the current BackInTIme as root backups. For example a "full system backup" option that adds excludes of the /proc directory and */node_modules, checks the "Preserve extended attributes (xattr)" setting in the "Expert Options" and keeps only very few (e.g. two) backups by default. Different types could be aggregated here (or at some file that this repo would use) and over time. If some other tools like TimeShift already has such functionality, all the better because then you could use some of the libraries they use and/or parts of their code and configs etc.

[buhtz]

I finally vote to close and see no strong argument to keep this open.

It is out of scope not only because of limited resources int he projects team. The limit is rsync which is not intended to do such full backups or restores.

Please use the right tool for the right job.