Syncoid cannot receive incremental stream: most recent snapshot does not match incremental source

[asmartin]

I recently configured sanoid and syncoid on Ubuntu 16.04 to take snapshots on host1 and sync them to host2. This appeared to work well at first, but eventually syncoid fails out with this error:

root@host1 # syncoid -r pool/dataset1 root@host2:pool/dataset1
Sending incremental pool/dataset1@syncoid_host1_2016-08-18:11:00:12 ... syncoid_host1_2016-08-18:11:07:58 (~ 4 KB):
Sending incremental pool/dataset1/data@autosnap_2016-08-18_11:00:01_hourly ... syncoid_host1_2016-08-18:11:07:59 (~ 4 KB):
cannot receive incremental stream: most recent snapshot of pool/dataset1/data does not
match incremental source
CRITICAL ERROR:  /sbin/zfs send -I pool/dataset1/data@autosnap_2016-08-18_11:00:01_hourly pool/dataset1/data@syncoid_host1_2016-08-18:11:07:59 | /usr/bin/pv -s 4096 | /usr/bin/lzop  | /usr/bin/mbuffer  -q -s 128k -m 16M 2>/dev/null | /usr/bin/ssh -c chacha20-poly1305@openssh.com,arcfour -p 22 -S /tmp/syncoid-root-root@host2-1471536477 root@host2 ' /usr/bin/mbuffer  -q -s 128k -m 16M 2>/dev/null | /usr/bin/lzop -dfc |  /sbin/zfs receive pool/dataset1/data' failed: 256 at /usr/local/bin/syncoid line 241.

According to the Oracle documentation, this error occurs if the data related to the dataset is modified on the destination (host2), however if I run zfs get mounted pool/dataset1/data on host2 it shows that it is not mounted, thus it seems unlikely that the data was modified on host2. I do see that atime is on, but again how does that matter since it isn't mounted. What do I need to do to resolve this error?

Thanks!

[jimsalterjrs]

You're almost certainly taking snapshots on the target side, and ending up with target snapshots taken locally which have the same names as source snapshots. This will especially happen if you run Sanoid with "production" templates on the target side as well as the source side.

If you run Sanoid on the target, you need to use the "backup" template (or just make sure autosnap is set to false) - new snapshots are supposed to replicate in from source, not be taken locally.

[asmartin]

Thanks for the quick response! Here is the setup I have

host1:

• pool/dataset1 - mounted dataset for data on host1
• pool/dataset2 - unmounted dataset for data from host2 synced here for HA

host2:

• pool/dataset1 - unmounted dataset for data from host1 synced here for HA
• pool/dataset2 - mounted dataset for data on host2

I have sanoid configured as follows on host1:

[pool/dataset1]
use_template = template_local
recursive = yes

[pool/dataset2]
use_template = template_remote
recursive = yes

#############################
# templates below this line #
#############################

[template_local]
hourly = 48
daily = 10
monthly = 2
yearly = 0
autosnap = yes
autoprune = yes

[template_remote]
hourly = 48
daily = 10
monthly = 2
yearly = 0
autosnap = no
autoprune = yes

and on host2:

[pool/dataset2]
use_template = template_local
recursive = yes

[pool/dataset1]
use_template = template_remote
recursive = yes

#############################
# templates below this line #
#############################

[template_local]
hourly = 48
daily = 10
monthly = 2
yearly = 0
autosnap = yes
autoprune = yes

[template_remote]
hourly = 48
daily = 10
monthly = 2
yearly = 0
autosnap = no
autoprune = yes

Do I have something wrong with the logic in the sanoid config files? Sanoid on host1 should only be making snapshots in pool/dataset1 and sanoid on host2 should only be making snapshots in pool/dataset2.

[jimsalterjrs]

Something has taken a snapshot with the same name but different content on host2 than on host1, be it Sanoid - with or without an earlier configuration that doesn't match the one shown here - or something else.

The error message you've given here - "most recent snapshot does not match incremental source" - can't be thrown any other way, that I'm aware of. Keep in mind also that Syncoid uses the -F argument for its zfs receive component in recent versions, which positively keeps changes in the current filesystem on the target from killing replication, whether it's mounted or not - ZFS itself forcibly rolls back any data that's changed since the most recent snapshot if necessary.

[jimsalterjrs]

Do you have any snapshots on host2 that are not created by Sanoid or Syncoid?

You've sanitized your hostnames here - do you have multiple hosts with the same actual hostname? (Keep in mind that Syncoid creates its own sync snapshots using the hostname, meaning that if you have multiple machines with the same hostname, you can create some collisions between syncoid snapshots.)

[asmartin]

I am only creating snapshots with sanoid and checked zfs list -t snapshot and all of the snapshots on both hosts are created with sanoid. I just checked and I am running version 1.4.6c (the latest release/tag), but it looks like in 1.4.7 you added -F on line 244, when the target already exists (my case). Could this be the problem? Is 1.4.7 stable/released?

[jimsalterjrs]

Honestly the -F shouldn't matter, because syncoid itself already forcibly rolls back the target prior to replication. That said, ANYTHING in Master on GitHub in my repo is something I consider "production ready" - if it's still in testing, I don't push to master, I either keep it local or fork a dev branch off. And yes, I'd give 1.4.7 a shot.

[asmartin]

Testing with 1.4.7 is working so far. I will keep testing for another day or two before considering this fixed. Can you tag 1.4.7 so I can download it directly rather than pulling from master?

[asmartin]

Running with 1.4.7 has been working for several days, I think this is the solution. Thanks for the help!