qvm-backup-restore is slow #2986

woju · Aug 7, 2017

Qubes OS version (e.g., `R3.2`): R4.0

qvm-backup-restore pipes internally whole backup tarfile at least three times (header, qubes.xml, data). This is painful on big backups and frustrating if something goes wrong at a later stage for large (~1TB) backups. (cf. bliviet)

Ideally reading should be streamlined to at most two passes: 1) header + hmac check; 2) the rest. Tarfile layout should explicitly support this, if it does not already.

Cc: @marmarek

It does this only when you connect backup source directly to dom0, which you shouldn't do.

marmarek · Aug 7, 2017

It does this only when you connect backup source directly to dom0, which you shouldn't do.

And for the dom0 case, this is limitation of tar. If you find a way tell tar to stop reading the archive, when all the files mentioned on command line are found, that would greatly improve the situation. Otherwise, alternative, hacky, solution is to keep track of restored files in python and kill the tar process when it get all requested files - something I'd like to avoid.

As for your idea - it isn't that simple, because you know what files to expect (including qubes.xml + qubes.xml.hmac vs qubes.xml.enc) only after retrieving backup header.

marmarek · Aug 7, 2017

And for the dom0 case, this is limitation of tar. If you find a way tell tar to stop reading the archive, when all the files mentioned on command line are found, that would greatly improve the situation. Otherwise, alternative, hacky, solution is to keep track of restored files in python and kill the tar process when it get all requested files - something I'd like to avoid.

As for your idea - it isn't that simple, because you know what files to expect (including qubes.xml + qubes.xml.hmac vs qubes.xml.enc) only after retrieving backup header.

I did it from separate VM, granting it (unsuccessfully at first, that's how I know) necessary Admin API permissions. I felt it does three passes.

For some ideas how to do that, I'd rewrite it using tarfile and I'd forget about the part of spec that says the file that counts is the last one in the archive.

I probably know nothing about corner cases, but this doesn't take 2.5 hours:

import tarfile
t = tarfile.open('qubes-backup', ignore_zeros=True)

ti = t.next()
assert ti.name == 'backup-header'
backup_header = t.extractfile(ti).read()

ti = t.next()
assert ti.name == 'backup-header.hmac'
backup_header_hmac = t.extractfile(ti).read()

woju · Aug 7, 2017

I did it from separate VM, granting it (unsuccessfully at first, that's how I know) necessary Admin API permissions. I felt it does three passes.

For some ideas how to do that, I'd rewrite it using tarfile and I'd forget about the part of spec that says the file that counts is the last one in the archive.

I probably know nothing about corner cases, but this doesn't take 2.5 hours:

import tarfile
t = tarfile.open('qubes-backup', ignore_zeros=True)

ti = t.next()
assert ti.name == 'backup-header'
backup_header = t.extractfile(ti).read()

ti = t.next()
assert ti.name == 'backup-header.hmac'
backup_header_hmac = t.extractfile(ti).read()

While this may work for backup header, loading actual backup data into RAM is REALLY BAD IDEA (you've mentioned 1TB, no?). And having two different methods for extracting and verifying backup header and backup data is asking for bug in either of them.

marmarek · Aug 7, 2017

While this may work for backup header, loading actual backup data into RAM is REALLY BAD IDEA (you've mentioned 1TB, no?). And having two different methods for extracting and verifying backup header and backup data is asking for bug in either of them.

Hmmm.

Process Process-2:2:
Traceback (most recent call last):
  ...
  File "/usr/lib/python3/dist-packages/qubesadmin/backup/restore.py", line 1698, in _handle_appmenus_list
    stdin=stream)
  ...
FileNotFoundError: [Errno 2] No such file or directory: 'qvm-appmenus'
qubesadmin.backup.extract: Error while processing ...

and the same for all vms.

Looks like another 6 hours till I get back.

woju · Aug 7, 2017

Hmmm.

Process Process-2:2:
Traceback (most recent call last):
  ...
  File "/usr/lib/python3/dist-packages/qubesadmin/backup/restore.py", line 1698, in _handle_appmenus_list
    stdin=stream)
  ...
FileNotFoundError: [Errno 2] No such file or directory: 'qvm-appmenus'
qubesadmin.backup.extract: Error while processing ...

and the same for all vms.

Looks like another 6 hours till I get back.

It's not into RAM. t.extractfile gives a file-like object.

woju · Aug 7, 2017

It's not into RAM. t.extractfile gives a file-like object.

Running qvm-backup-restore in dedicated VM = paranoid restore mode. Not supporting appmenus is somehow intentional missing feature.

marmarek · Aug 7, 2017

Running qvm-backup-restore in dedicated VM = paranoid restore mode. Not supporting appmenus is somehow intentional missing feature.

Does it even work at all?

woju · Aug 7, 2017

Does it even work at all?

You tell me...

marmarek · Aug 7, 2017

You tell me...

I tell you it doesn't. Just don't tell Joanna...

woju · Aug 7, 2017

I tell you it doesn't. Just don't tell Joanna...

It's not into RAM. t.extractfile gives a file-like object.

Ah, so you're seeking solution for performance related issues, by passing the whole blob through python?

marmarek · Aug 7, 2017

It's not into RAM. t.extractfile gives a file-like object.

Ah, so you're seeking solution for performance related issues, by passing the whole blob through python?

No. I'm seeking solutions for performance-related issues by intentionally forgetting about relevant parts of tar file specification. If it takes python to misimplement it, so be it, if the alternative is to do it in C.

woju · Aug 7, 2017

No. I'm seeking solutions for performance-related issues by intentionally forgetting about relevant parts of tar file specification. If it takes python to misimplement it, so be it, if the alternative is to do it in C.

In theory, tar --seek should also greatly improve performance (it should simply seek through all ignored files). But in also in theory it should be automatically enabled...

marmarek · Aug 7, 2017

In theory, tar --seek should also greatly improve performance (it should simply seek through all ignored files). But in also in theory it should be automatically enabled...

No, it won't. According to python's tarfile module documentation, the file that counts under a given path is the last one in the archive (I think it this is that way to be able to do incremental backups on tape by simply appending new files). So you have to seek whole file anyway, especially with -i. GNU tar works with this assumption. I say we break that and hereby declare header and HMAC to be the first two files in the archive.

woju · Aug 7, 2017

No, it won't. According to python's tarfile module documentation, the file that counts under a given path is the last one in the archive (I think it this is that way to be able to do incremental backups on tape by simply appending new files). So you have to seek whole file anyway, especially with -i. GNU tar works with this assumption. I say we break that and hereby declare header and HMAC to be the first two files in the archive.

But seeking through the whole archive (just reading file headers) also should be much faster, even for 1TB archive.
I'm checking tar --occurrence option right now, which should do exactly what we want.

marmarek · Aug 7, 2017

But seeking through the whole archive (just reading file headers) also should be much faster, even for 1TB archive.
I'm checking tar --occurrence option right now, which should do exactly what we want.

For simple archive it helps. If you're brave enough, try adding --occurrence=1 to https://github.com/QubesOS/qubes-core-admin-client/blob/master/qubesadmin/backup/restore.py#L870

marmarek · Aug 7, 2017

For simple archive it helps. If you're brave enough, try adding --occurrence=1 to https://github.com/QubesOS/qubes-core-admin-client/blob/master/qubesadmin/backup/restore.py#L870

Me? Not brave enough? Just hold me my beer...

woju · Aug 7, 2017

Me? Not brave enough? Just hold me my beer...

You should get backup summary in seconds.

marmarek · Aug 7, 2017

You should get backup summary in seconds.

It got better!

woju · Aug 7, 2017

It got better!

Hmmm. Manpage says default for that option is 1.

Maybe the truth is "avaiable in the texinfo format".

woju · Aug 7, 2017

Hmmm. Manpage says default for that option is 1.

Maybe the truth is "avaiable in the texinfo format".

That was also my understanding, but now I think --occurrence[=N] means "The default N is 1, if you specify --occurrence without N".

marmarek · Aug 7, 2017

That was also my understanding, but now I think --occurrence[=N] means "The default N is 1, if you specify --occurrence without N".

Guys, I'm going to fix the issue about --occurence, but it leaves the bug around qvm-appmenus, so I'm going to file another ticket for that.

woju · Aug 8, 2017

Guys, I'm going to fix the issue about --occurence, but it leaves the bug around qvm-appmenus, so I'm going to file another ticket for that.

Automated announcement from builder-github

The package python2-qubesadmin-4.0.6-0.1.fc25 has been pushed to the r4.0 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

QubesOS/qubes-issues

Join GitHub today

qvm-backup-restore is slow #2986

Comments

woju commented Aug 7, 2017

Qubes OS version (e.g., R3.2): R4.0

This comment has been minimized.

marmarek commented Aug 7, 2017

This comment has been minimized.

marmarek commented Aug 7, 2017

This comment has been minimized.

woju commented Aug 7, 2017

This comment has been minimized.

marmarek commented Aug 7, 2017

This comment has been minimized.

woju commented Aug 7, 2017

This comment has been minimized.

woju commented Aug 7, 2017

This comment has been minimized.

marmarek commented Aug 7, 2017

This comment has been minimized.

woju commented Aug 7, 2017

This comment has been minimized.

marmarek commented Aug 7, 2017

This comment has been minimized.

woju commented Aug 7, 2017

This comment has been minimized.

marmarek commented Aug 7, 2017

This comment has been minimized.

woju commented Aug 7, 2017

This comment has been minimized.

marmarek commented Aug 7, 2017

This comment has been minimized.

woju commented Aug 7, 2017

This comment has been minimized.

marmarek commented Aug 7, 2017

This comment has been minimized.

marmarek commented Aug 7, 2017

This comment has been minimized.

woju commented Aug 7, 2017

This comment has been minimized.

marmarek commented Aug 7, 2017

This comment has been minimized.

woju commented Aug 7, 2017

This comment has been minimized.

woju commented Aug 7, 2017

This comment has been minimized.

marmarek commented Aug 7, 2017

andrewdavidwong added C: core enhancement labels Aug 8, 2017

andrewdavidwong added this to the Release 4.0 milestone Aug 8, 2017

woju self-assigned this Aug 8, 2017

This comment has been minimized.

woju commented Aug 8, 2017

woju referenced this issue Aug 8, 2017

marmarek added a commit to marmarek/qubes-core-admin-client that referenced this issue Aug 30, 2017

marmarek referenced this issue in QubesOS/qubes-core-admin-client Aug 30, 2017

marmarek closed this in QubesOS/qubes-core-admin-client#26 Aug 30, 2017

This comment has been minimized.

qubesos-bot commented Sep 14, 2017

qubesos-bot added the r4.0-dom0-cur-test label Sep 14, 2017

qubesos-bot referenced this issue in QubesOS/updates-status Sep 14, 2017

This comment has been minimized.

qubesos-bot commented Sep 14, 2017

qubesos-bot added the r4.0-fc24-cur-test label Sep 14, 2017

This comment has been minimized.

qubesos-bot commented Sep 14, 2017

qubesos-bot added the r4.0-jessie-cur-test label Sep 14, 2017

This comment has been minimized.

qubesos-bot commented Sep 14, 2017

qubesos-bot added the r4.0-fc25-cur-test label Sep 14, 2017

This comment has been minimized.

qubesos-bot commented Sep 14, 2017

qubesos-bot added the r4.0-stretch-cur-test label Sep 14, 2017

This comment has been minimized.

qubesos-bot commented Oct 30, 2017

qubesos-bot added r4.0-jessie-stable and removed r4.0-jessie-cur-test labels Oct 30, 2017

This comment has been minimized.

qubesos-bot commented Oct 30, 2017

qubesos-bot added r4.0-stretch-stable and removed r4.0-stretch-cur-test labels Oct 30, 2017

This comment has been minimized.

Qubes OS version (e.g., `R3.2`): R4.0