No such file or directory issue

[ceremcem]

Description

One usual run of sync-local.sh produces following output:

[✓] /mnt/erik/snapshots/rootfs/rootfs.20170612T1632 exists in destination, no need to send again.
[✓] /mnt/erik/snapshots/rootfs/rootfs.20170612T1642 exists in destination, no need to send again.
[✓] /mnt/erik/snapshots/rootfs/rootfs.20170612T1652 exists in destination, no need to send again.
[✓] /mnt/erik/snapshots/rootfs/rootfs.20170613T0357 exists in destination, no need to send again.
/mnt/erik/snapshots/rootfs/rootfs.20170613T1737 will be sent to destination.
Sending difference between /mnt/erik/snapshots/rootfs/rootfs.20170613T0357 and /mnt/erik/snapshots/rootfs/rootfs.20170613T1737
At subvol /mnt/erik/snapshots/rootfs/rootfs.20170613T1737
At subvol /mnt/erik/snapshots/rootfs/rootfs.20170613T1737                                                                                          ]
 115 B 0:00:02 [ 123 B/s] [  <=>                                                                                                                   ]At snapshot rootfs.20170613T1737
 160MiB 0:00:23 [6.92MiB/s] [                                                  <=>                                                                 ]
/mnt/erik/snapshots/rootfs/rootfs.20170614T0320 will be sent to destination.
Sending difference between /mnt/zencefil/snapshots/rootfs/rootfs.20170613T1737 and /mnt/erik/snapshots/rootfs/rootfs.20170614T0320
ERROR: open fil/snapshots/rootfs/rootfs.20170613T1737 failed. No such file or directory
ERROR: could not resolve rootid for /mnt/zencefil/snapshots/rootfs/rootfs.20170613T1737
WE DETECT AN ERROR in btrfs_send_diff! parent: /mnt/zencefil/snapshots/rootfs/rootfs.20170613T1737, curr: /mnt/erik/snapshots/rootfs/rootfs.20170614T0320
   0 B 0:00:00 [   0 B/s] [<=>                                                                                                                     ]
ERROR: empty stream is not considered valid
not setting ref as empty string, skipping!
/mnt/erik/snapshots/rootfs/rootfs.20170614T0331 will be sent to destination.
Sending difference between /mnt/zencefil/snapshots/rootfs/rootfs.20170613T1737 and /mnt/erik/snapshots/rootfs/rootfs.20170614T0331
ERROR: open fil/snapshots/rootfs/rootfs.20170613T1737 failed. No such file or directory
ERROR: could not resolve rootid for /mnt/zencefil/snapshots/rootfs/rootfs.20170613T1737
WE DETECT AN ERROR in btrfs_send_diff! parent: /mnt/zencefil/snapshots/rootfs/rootfs.20170613T1737, curr: /mnt/erik/snapshots/rootfs/rootfs.20170614T0331
   0 B 0:00:00 [   0 B/s] [<=>                                                                                                                     ]
ERROR: empty stream is not considered valid
not setting ref as empty string, skipping!
Synchronization finished. 00:05:41

Second run right after above output is as follows:

[✓] /mnt/erik/snapshots/rootfs/rootfs.20170612T1632 exists in destination, no need to send again.
[✓] /mnt/erik/snapshots/rootfs/rootfs.20170612T1642 exists in destination, no need to send again.
[✓] /mnt/erik/snapshots/rootfs/rootfs.20170612T1652 exists in destination, no need to send again.
[✓] /mnt/erik/snapshots/rootfs/rootfs.20170613T0357 exists in destination, no need to send again.
[✓] /mnt/erik/snapshots/rootfs/rootfs.20170613T1737 exists in destination, no need to send again.
/mnt/erik/snapshots/rootfs/rootfs.20170614T0320 will be sent to destination.
Sending difference between /mnt/erik/snapshots/rootfs/rootfs.20170613T1737 and /mnt/erik/snapshots/rootfs/rootfs.20170614T0320
At subvol /mnt/erik/snapshots/rootfs/rootfs.20170614T0320
At subvol /mnt/erik/snapshots/rootfs/rootfs.20170614T0320                                                                                          ]
At snapshot rootfs.20170614T0320
1.79GiB 0:01:44 [17.6MiB/s] [                                    <=>                                                                               ]
/mnt/erik/snapshots/rootfs/rootfs.20170614T0331 will be sent to destination.
Sending difference between /mnt/zencefil/snapshots/rootfs/rootfs.20170614T0320 and /mnt/erik/snapshots/rootfs/rootfs.20170614T0331
ERROR: open fil/snapshots/rootfs/rootfs.20170614T0320 failed. No such file or directory
ERROR: could not resolve rootid for /mnt/zencefil/snapshots/rootfs/rootfs.20170614T0320
WE DETECT AN ERROR in btrfs_send_diff! parent: /mnt/zencefil/snapshots/rootfs/rootfs.20170614T0320, curr: /mnt/erik/snapshots/rootfs/rootfs.20170614T0331
   0 B 0:00:00 [   0 B/s] [<=>                                                                                                                     ]
ERROR: empty stream is not considered valid
not setting ref as empty string, skipping!
Synchronization finished. 00:02:57

It looks like btrfs send keeps the snapshot - somewhat - open and we can not read a snapshot we just sent.

Re-running the sync-local.sh script over and over again solves the issue.

Version

smith-sync: 668e791 (relevant part here)
btrfs-progs: v4.11

Reproduction

Can not be reproduced by tests

[ceremcem]

It seems this error is normal as we are passing two snapshots to btrfs send to make it calculate the difference, but these two snapshots reside on different physical disks:

Sending difference between /mnt/zencefil/snapshots/rootfs/rootfs.20170613T1737 and /mnt/erik/snapshots/rootfs/rootfs.20170614T0320

A check point is added in order to avoid such problems in the future.

[ceremcem]