qubes-backup-restore does not restore dom0 from 3.2 backup

[betweenscenery]

Qubes OS version:

4.0 rc3

Affected TemplateVMs:

n/a

---

Steps to reproduce the behavior:

backup appvms and dom0 from an up to date qubes 3.2 machine, and attempt to restore them on a 4.0 rc3 machine with $ qubes-backup-restore

Expected behavior:

appvms and dom0 are restored successfully

Actual behavior:

appvms are restored but dom0 is not

General notes:

• I have verified that the backup does contain dom0 and have verified its integrity.
• I have reproduced this bug on two 4.0 rc3 machines
• I have tried --ignore-username-mismatch, --ignore-missing and various other arguments for qubes-backup-restore with no success. (n.b. the username does match)
• The behavior of qubes-desktop-restore is consistent with dom0 not being present in the backup.

---

Related issues:

[marmarek]

Do you get any specific error? Or simply dom0 is ignored?
Also, what version of qubes-core-admin-client package do you have? Does it looks similar to #3362?

[marmarek]

Do you get any specific error? Or simply dom0 is ignored?
Also, what version of qubes-core-admin-client package do you have? Does it looks similar to #3362?

[betweenscenery]

dom0 is simply ignored, as if absent, however when I modified the source to force an attempt at restoring it, it did error like #3362 .
qubes-core-admin-client is 4.0.12-0.1.fc25

[betweenscenery]

dom0 is simply ignored, as if absent, however when I modified the source to force an attempt at restoring it, it did error like #3362 .
qubes-core-admin-client is 4.0.12-0.1.fc25

[taradiddles]

fwiw, I'm encountering more or less the same problem (4.0rc3, fully updated with -testing packages).

1- with 3.2, I've made a backup of dom0 only
2- with 4.0rc3, using qvm-backup-restore with the file created a 1-, a vm named "disp-no-netvm" is found instead of dom0 (see output below); when restoring, a vm named "disp-no-netvm" is indeed created but its private img doesn't contain anything from the former dom0.

app: Checking backup content...
The following VMs are included in the backup:

---------------+-------+----------+-------+-------+
          name |  type | template | netvm | label |
---------------+-------+----------+-------+-------+
 disp-no-netvm | AppVM |      n/a |     - |   red |

app: The above VM archive(s) will be verified.
app: Existing VMs will NOT be removed or altered.
qubesadmin.backup: -> Verifying disp-no-netvm...
qubesadmin.backup: Extracting data: 0 to restore
qubesadmin.backup: -> Done.

the backup made at 1- is functional, ie. I could extract dom0's home from the backup file with the manual emergency restore procedure described in

https://www.qubes-os.org/doc/backup-emergency-restore-v3/

(that really saved me - I thought I had lost all my dom0 customizations !)

[taradiddles]

fwiw, I'm encountering more or less the same problem (4.0rc3, fully updated with -testing packages).

1- with 3.2, I've made a backup of dom0 only
2- with 4.0rc3, using qvm-backup-restore with the file created a 1-, a vm named "disp-no-netvm" is found instead of dom0 (see output below); when restoring, a vm named "disp-no-netvm" is indeed created but its private img doesn't contain anything from the former dom0.

app: Checking backup content...
The following VMs are included in the backup:

---------------+-------+----------+-------+-------+
          name |  type | template | netvm | label |
---------------+-------+----------+-------+-------+
 disp-no-netvm | AppVM |      n/a |     - |   red |

app: The above VM archive(s) will be verified.
app: Existing VMs will NOT be removed or altered.
qubesadmin.backup: -> Verifying disp-no-netvm...
qubesadmin.backup: Extracting data: 0 to restore
qubesadmin.backup: -> Done.

the backup made at 1- is functional, ie. I could extract dom0's home from the backup file with the manual emergency restore procedure described in

https://www.qubes-os.org/doc/backup-emergency-restore-v3/

(that really saved me - I thought I had lost all my dom0 customizations !)

[tasket]

@marmarek There was a dom0-related change from PR#43. I think this went to qubes-core-admin-client 4.0.13.* but not sure.

I'm using 4.0.13 and testing with an R4.0-produced dom0 archive (after backing up home with tar) yields an error: "skipping; VM dom0 already exists"

I'll locate a Qubes 3.2 archive and test with that also.

---

_restore_vms_metadata(), in restore.py:

# First load templates, then other VMs

The loop at the above comment seems to need a conditional so it doesn't try to create a new 'dom0'.

[tasket]

@marmarek There was a dom0-related change from PR#43. I think this went to qubes-core-admin-client 4.0.13.* but not sure.

I'm using 4.0.13 and testing with an R4.0-produced dom0 archive (after backing up home with tar) yields an error: "skipping; VM dom0 already exists"

I'll locate a Qubes 3.2 archive and test with that also.

---

_restore_vms_metadata(), in restore.py:

# First load templates, then other VMs

The loop at the above comment seems to need a conditional so it doesn't try to create a new 'dom0'.

[tasket]

Loop in _restore_vms_metadata() is a side-issue.

The reported problem is due to the xml parsing of the older qubes.xml format, I believe. This is called from Core2Qubes.load() in core2.py.

I'll see if I can fix it.

[tasket]

Loop in _restore_vms_metadata() is a side-issue.

The reported problem is due to the xml parsing of the older qubes.xml format, I believe. This is called from Core2Qubes.load() in core2.py.

I'll see if I can fix it.

[tasket]

@marmarek - A number of problems here. Most significant is that _handle_dom0() method appears to be mismatched and non-functional to the rest of restore.py. Dispatcher code for the handlers passes a "stream" object but _handle_dom0() expects a path string, and nothing in the method is structured to handle a byte stream. This is especially true since the stream contains raw data of multiple unarchived files, so I don't see how that can be handled gracefully. In summary, the method probably should be refactored.

What I've done to get it working is to change the 'tar2_cmdline' in the case of dom0 to simply run cat, so the inner tar file will be passed directly as a stream to _handle_dom0(). Then the method simply runs tar -C /home/user/subdir_datetime -xz. This paranoid-like mode is optimal for me personally, since the idea of restoring old dom0 files to original working paths gives me nightmares, even with conflicting files being stashed.

What do you think?

[tasket]

@marmarek - A number of problems here. Most significant is that _handle_dom0() method appears to be mismatched and non-functional to the rest of restore.py. Dispatcher code for the handlers passes a "stream" object but _handle_dom0() expects a path string, and nothing in the method is structured to handle a byte stream. This is especially true since the stream contains raw data of multiple unarchived files, so I don't see how that can be handled gracefully. In summary, the method probably should be refactored.

What I've done to get it working is to change the 'tar2_cmdline' in the case of dom0 to simply run cat, so the inner tar file will be passed directly as a stream to _handle_dom0(). Then the method simply runs tar -C /home/user/subdir_datetime -xz. This paranoid-like mode is optimal for me personally, since the idea of restoring old dom0 files to original working paths gives me nightmares, even with conflicting files being stashed.

What do you think?

[marmarek]

What I've done to get it working is to change the 'tar2_cmdline' in the case of dom0 to simply run cat, so the inner tar file will be passed directly as a stream to _handle_dom0(). Then the method simply runs tar -C /home/user/subdir_datetime -xz. This paranoid-like mode is optimal for me personally, since the idea of restoring old dom0 files to original working paths gives me nightmares, even with conflicting files being stashed.

👍

The tar command in _handle_dom0 should handle appropriate options, including --use-compress-program (support for non-default compression), and t instead of x - for verification.

That would also ease testing it, without breaking the test machine. Currently dom0 backup test check only if backup do not crash, but not even try to restore it...

[marmarek]

What I've done to get it working is to change the 'tar2_cmdline' in the case of dom0 to simply run cat, so the inner tar file will be passed directly as a stream to _handle_dom0(). Then the method simply runs tar -C /home/user/subdir_datetime -xz. This paranoid-like mode is optimal for me personally, since the idea of restoring old dom0 files to original working paths gives me nightmares, even with conflicting files being stashed.

👍

The tar command in _handle_dom0 should handle appropriate options, including --use-compress-program (support for non-default compression), and t instead of x - for verification.

That would also ease testing it, without breaking the test machine. Currently dom0 backup test check only if backup do not crash, but not even try to restore it...

[tasket]

@marmarek - OK, but currently handler-assignment is skipped when verify-only is true, from the fixes I made in December to get verify-only working. That means the outer tar and the scrypt verification are checked, but not the inner tar.

So that fix would have to be changed for the inner tar to be checked, and each handle* method would need a conditional for verify-only.

[tasket]

@marmarek - OK, but currently handler-assignment is skipped when verify-only is true, from the fixes I made in December to get verify-only working. That means the outer tar and the scrypt verification are checked, but not the inner tar.

So that fix would have to be changed for the inner tar to be checked, and each handle* method would need a conditional for verify-only.

[tasket]

Side note:

XML parser sees dom0 in V3 archive as "StandaloneVM" and in V4 archive as "AdminVM". Had to switch a check from using vm.klass to vm.name to get both versions working.

[tasket]

Side note:

XML parser sees dom0 in V3 archive as "StandaloneVM" and in V4 archive as "AdminVM". Had to switch a check from using vm.klass to vm.name to get both versions working.

[marmarek]

@marmarek - OK, but currently handler-assignment is skipped when verify-only is true, from the fixes I made in December to get verify-only working. That means the outer tar and the scrypt verification are checked, but not the inner tar.

Oh, but if verify-only, then inner tar is called with t instead of x, so the handler get file list instead of file content anyway... This probably means need for verify-only specific handlers instead of skipping them.

BTW currently verify-only (and actual restore) do not notice if some VM data is completely missing in the archive. Only corrupted data is detected.

XML parser sees dom0 in V3 archive as "StandaloneVM" and in V4 archive as "AdminVM". Had to switch a check from using vm.klass to vm.name to get both versions working.

Better keep using vm.klass and fix core2.py to correctly detect dom0 class. Looks like beginning of import_core2_vm is wrong.

[marmarek]

@marmarek - OK, but currently handler-assignment is skipped when verify-only is true, from the fixes I made in December to get verify-only working. That means the outer tar and the scrypt verification are checked, but not the inner tar.

Oh, but if verify-only, then inner tar is called with t instead of x, so the handler get file list instead of file content anyway... This probably means need for verify-only specific handlers instead of skipping them.

BTW currently verify-only (and actual restore) do not notice if some VM data is completely missing in the archive. Only corrupted data is detected.

XML parser sees dom0 in V3 archive as "StandaloneVM" and in V4 archive as "AdminVM". Had to switch a check from using vm.klass to vm.name to get both versions working.

Better keep using vm.klass and fix core2.py to correctly detect dom0 class. Looks like beginning of import_core2_vm is wrong.

[andrewdavidwong]

BTW currently verify-only (and actual restore) do not notice if some VM data is completely missing in the archive. Only corrupted data is detected.

Branched to enhancement issue #3498.

[andrewdavidwong]

BTW currently verify-only (and actual restore) do not notice if some VM data is completely missing in the archive. Only corrupted data is detected.

Branched to enhancement issue #3498.

[tasket]

PR QubesOS/qubes-core-admin-client#52

[tasket]

PR QubesOS/qubes-core-admin-client#52

[qubesos-bot]

Automated announcement from builder-github

The package python2-qubesadmin-4.0.14-0.1.fc25 has been pushed to the r4.0 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package python2-qubesadmin-4.0.14-0.1.fc25 has been pushed to the r4.0 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component core-admin-client (including package python2-qubesadmin-4.0.14-0.1.fc26) has been pushed to the r4.0 testing repository for the Fedora template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component core-admin-client (including package python2-qubesadmin-4.0.14-0.1.fc26) has been pushed to the r4.0 testing repository for the Fedora template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-admin-client_4.0.14-1+deb10u1 has been pushed to the r4.0 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing buster-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-admin-client_4.0.14-1+deb10u1 has been pushed to the r4.0 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing buster-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-admin-client_4.0.14-1+deb9u1 has been pushed to the r4.0 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing stretch-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-admin-client_4.0.14-1+deb9u1 has been pushed to the r4.0 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing stretch-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component core-admin-client (including package python2-qubesadmin-4.0.14-0.1.fc26) has been pushed to the r4.0 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component core-admin-client (including package python2-qubesadmin-4.0.14-0.1.fc26) has been pushed to the r4.0 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package python2-qubesadmin-4.0.14-0.1.fc25 has been pushed to the r4.0 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package python2-qubesadmin-4.0.14-0.1.fc25 has been pushed to the r4.0 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-admin-client_4.0.14-1+deb9u1 has been pushed to the r4.0 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-admin-client_4.0.14-1+deb9u1 has been pushed to the r4.0 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update