Snapshot date formatter causes replication task to fail

[krin-san]

Recently I performed an update to TrueNAS Scale Cobia 23.10.2 and run heavyscript backup to manually backup ix-applications before the OS update. This resulted in following snapshots being created for all ix-applications children datasets:

# zfs list -rt snapshot -o name ssd/ix-applications | less
...
ssd/ix-applications/default_volumes@ix-applications-backup-HeavyScript_2024_04_30_18_34_27
ssd/ix-applications/default_volumes@ix-applications-backup-system-update--2024-04-30_16:45:51

A replication task I have configured to export ix-applications backups onto a different poll on the same system started failing since then with the following error:

warning: cannot send 'ssd/ix-applications/default_volumes@ix-applications-backup-HeavyScript_2024_04_30_18_34_27': not an earlier snapshot from the same fs
Replication cannot continue because existing snapshot
ix-applications-backup-system-update--2024-04-30_16:45:51 is newer than
ssd/ix-applications/default_volumes@ix-applications-backup-HeavyScript_2024_04_30_18_34_27, but has an older date
in the snapshot name. To resolve the error, rename
ssd/ix-applications/default_volumes@ix-applications-backup-HeavyScript_2024_04_30_18_34_27 with a date that is older than
ix-applications-backup-system-update--2024-04-30_16:45:51 or delete snapshot
ssd/ix-applications/default_volumes@ix-applications-backup-HeavyScript_2024_04_30_18_34_27 from both the source and destination.
cannot receive: failed to read from stream.

Because I made a backup shortly before the OS update it helped me find out that HeavyScript and OS uses different date formatters for snapshot naming:

• ix-applications-backup-HeavyScript uses local (UTC+2) time zone: 18_34_27
• ix-applications-backup-system-update uses UTC time zone: 16:45:51

This causes consecutively created snapshots look out of order failing the replication tasks.

Possible fix

Change HeavyScript timestamp formatter to match the one used by OS.

Workaround

Delete "out-of-order" HeavyScript snapshots:

zfs list -rt snapshot -o name ssd/ix-applications | grep HeavyScript_2024_04_30_18_34_27 | xargs -n1 zfs destroy

[krin-san]

Another workaround for those who don't want to remove snapshots. Here I have renamed HeavyScript snapshots containing 12_19_10 to 10_19_10, basically converting UTC+2 to UTC:

zfs list -t snapshot -r ssd/ix-applications | grep 12_19_10 | awk '{print $1}' | xargs -I{} bash -c 'NAME={}; zfs rename $NAME ${NAME/12_19_10/10_19_10}'

[Heavybullets8]

https://github.com/Heavybullets8/heavy_script?tab=readme-ov-file#archived-repository-notice