Transfer size indication is broken #26

Closed
a3nm opened this Issue Apr 26, 2016 · 2 comments

Projects

None yet

2 participants

@a3nm
Contributor
a3nm commented Apr 26, 2016

I have invoked buttersink to transfer snapshots from one filesystem to the other as follows:

$ sudo ./buttersink.py /mnt/mem/snapshots/ /mnt/fah/BACKUP/snapshots/mem/

The current partial output is:

  Optimal synchronization:
  2.4 TiB from 2 diffs in btrfs /mnt/fah/BACKUP/snapshots/mem
  1.082 TiB from 374 diffs in btrfs /mnt/mem/snapshots
  3.482 TiB from 376 diffs in TOTAL
  Keep: aeda...1b4d /mnt/fah/BACKUP/snapshots/mem/btrfs-mem-backup-1459375237 from None (1.2 TiB)
  Keep: fcf5...a56e /mnt/fah/BACKUP/snapshots/mem/btrfs-mem-backup-1459374875 from None (1.2 TiB)
  Xfer: 7b69...2383 /mnt/mem/snapshots/snapshot-1460156461 from aeda...1b4d /mnt/mem/snapshots/btrfs-mem-backup-1459375237 (~439.2 MiB)
 0:13:42.781178: Sent 5.524 GiB of 439.2 MiB (1287%) ETA: None (57.7 Mbps )                           
  Xfer: 419b...5bcf /mnt/mem/snapshots/snapshot-1459512061 from aeda...1b4d /mnt/mem/snapshots/btrfs-mem-backup-1459375237 (~107.4 MiB)
 0:04:09.215211: Sent 5.139 GiB of 107.4 MiB (4901%) ETA: None (177 Mbps )                            
  Xfer: e897...4e4a /mnt/mem/snapshots/snapshot-1460250061 from aeda...1b4d /mnt/mem/snapshots/btrfs-mem-backup-1459375237 (~4.398 GiB)

The percentages end up being larger than 100%, so something is wrong somewhere. (The percentages get incremented and go over 100% while progress information is being displayed.)

(I should also point out that buttersink's heuristic do not seem to be performing so well in this case. The snapshots here were created by a crontab, with diff sizes much smaller than the total volume size, so I think the optimal plan would be to transfer the diffs successively. However, buttersink is apparently trying to transfer the delta with a much older state of the data because it already exists on the destination filesystem.)

@AmesCornish
Owner

Yes, the percentages can be arbitrarily larger than 100%. Buttersink used heuristics to guess at the source diff size, and sometimes (often?) this is incorrect. If you are transferring remotely (e.g. over ssh), buttersink will perform a local diff ahead of time, to get a correct size before determining the optimal transfers. If you are transferring locally, it just uses the guesses, since it would take just as long to measure the actual source size as it does to just do the transfer.

One tip: you can force buttersink to do the pre-measuring if you use ssh:/localhost/path as your destination.

@AmesCornish AmesCornish closed this May 9, 2016
@a3nm
Contributor
a3nm commented May 10, 2016

Thanks for the explanations! I understand better now, but still I wonder
whether the display couldn't be improved so that this is less confusing
and does not look like a bug to the user. For instance, display "of
estimated" instead of "of" when using estimated sizes, and drop ETA (and
maybe the percentage itself) when percentage is above 100%?

Regarding the parenthesis at the end of my report, do you have any
ideas about why buttersink is not picking the right parents at all? Is
this related to the diff estimations being off? (I can open a different
bug with this same report if you like.)

Thanks again!

Antoine Amarilli

On Mon, May 09, 2016 at 11:29:38AM -0700, Ames wrote:

Yes, the percentages can be arbitrarily larger than 100%. Buttersink used heuristics to guess at the source diff size, and sometimes (often?) this is incorrect. If you are transferring remotely (e.g. over ssh), buttersink will perform a local diff ahead of time, to get a correct size before determining the optimal transfers. If you are transferring locally, it just uses the guesses, since it would take just as long to measure the actual source size as it does to just do the transfer.

One tip: you can force buttersink to do the pre-measuring if you use ssh:/localhost/path as your destination.


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#26 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment