Stats may be wrongly computed

[benrubson]

Hi,

In some situations, especially when transfer is aborted, stats are wrongly computed, at both sides.

Here's an example of a CTRL+C interrupted transfer :

# du -b /tmp/source/
3221229568    /tmp/source/

# rsync -a /tmp/source/ ...
sending incremental file list
/tmp/
/tmp/source/
/tmp/source/bf1
          1.07G 100%  147.89MB/s    0:00:06 (xfr#1, to-chk=2/5)
/tmp/source/bf2
          1.07G 100%  132.94MB/s    0:00:07 (xfr#2, to-chk=1/5)
/tmp/source/bf3
        521.34M  48%  124.39MB/s    0:00:04  ^C

# "cat receiver.log"
rsync: connection unexpectedly closed (146 bytes received so far) [generator]
sent 81 bytes  received 146 bytes  total size 3221225472
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [generator=3.2.0dev]

Results : on receiver side, sent / received numbers are wrong.

And with SIGUSR2 :

# du -b /tmp/source/
3221229568    /tmp/source/

# rsync -a /tmp/source/ ...
sending incremental file list
/tmp/
/tmp/source/
/tmp/source/bf1
          1.07G 100%  149.53MB/s    0:00:06 (xfr#1, to-chk=2/5)
/tmp/source/bf2
          1.07G 100%  133.87MB/s    0:00:07 (xfr#2, to-chk=1/5)
/tmp/source/bf3
        527.70M  49%  125.91MB/s    0:00:04  
Number of files: 5 (reg: 3, dir: 2)
Number of created files: 5 (reg: 3, dir: 2)
Number of deleted files: 0
Number of regular files transferred: 3
Total file size: 3.22G bytes
Total transferred file size: 3.22G bytes
Literal data: 2.15G bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 0
Total bytes received: 0
sent 0 bytes  received 0 bytes  -0.00 bytes/sec
total size is 3.22G  speedup is -9,223,372,036,854,775,808nf

# "cat receiver.log"
rsync: connection unexpectedly closed (150 bytes received so far) [generator]
sent 81 bytes  received 150 bytes  total size 3221225472
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [generator=3.2.0dev]

Results : on receiver side, sent / received numbers are wrong.
Results : on sender side, many of the --stats numbers are wrong.

Thank you 👍

[WayneD]

The receiving side is measuring the stats in 2 procs until the very end, so I disabled early stats except from the sender.

[benrubson]

Thank you Wayne.
I've updated my post above with some Results, and, as you can see, issue is not only on receiver side, but also on sender side, where the --stats numbers are wrong.
Instead of simply disabling the stats if / when they are wrong, do you think the accounting process could be modified in a manner that stats would be sort of "real-time" computed, so that when transfer is aborted, numbers are accurate, and available at both sides ?
Thank you again 👍

[WayneD]

If you're killing processes with internal-only signals, you have to live with the fact that you get crap in the results since the stats info hasn't been updated by the end-of-run data updating. The only way to make (at least some of) the stats constantly updated is a total re-architecture of the 3-process layout, which is something that I'm contemplating for the future but won't happen any time soon.