File incorrectly zeroed when receiving incremental stream that toggles -L

[zrav]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	17.04
Linux Kernel	4.10.0-22-generic
Architecture	x86_64
ZFS Version	0.6.5.9
SPL Version	0.6.5.9

Describe the problem you're observing

I have found a data corruption issue in zfs send. In pools using 1M recordsize, incremental sends without the -L flag sometimes silently zero out some files. The results are repeatable. Scrub does not find any errors.

Tested on Ubuntu Xenial with 0.6.5.6 and Zesty with 0.6.5.9 on the same systems.

Describe how to reproduce the problem

Source pool:

root@oddity:~# modinfo zfs
filename:       /lib/modules/4.10.0-22-generic/kernel/zfs/zfs/zfs.ko
version:        0.6.5.9-2
license:        CDDL
author:         OpenZFS on Linux
description:    ZFS
srcversion:     42C4AB70887EA26A9970936
depends:        spl,znvpair,zcommon,zunicode,zavl
vermagic:       4.10.0-22-generic SMP mod_unload
...
root@oddity:~# zpool status
  pool: tank
 state: ONLINE
  scan: scrub canceled on Sun Jun 11 09:52:38 2017
config:

        NAME                                 STATE     READ WRITE CKSUM
        tank                                 ONLINE       0     0     0
          raidz2-0                           ONLINE       0     0     0
            ata-ST4000VN000-1H4168_XXXXXXXX  ONLINE       0     0     0
            ata-ST4000VN000-1H4168_XXXXXXXX  ONLINE       0     0     0
            ata-ST4000VN000-2AH166_XXXXXXXX  ONLINE       0     0     0
            ata-ST4000VN000-1H4168_XXXXXXXX  ONLINE       0     0     0
            ata-ST4000VN000-1H4168_XXXXXXXX  ONLINE       0     0     0
            ata-ST4000VN000-1H4168_XXXXXXXX  ONLINE       0     0     0
            ata-ST4000VN000-1H4168_XXXXXXXX  ONLINE       0     0     0
            ata-ST4000VN000-1H4168_XXXXXXXX  ONLINE       0     0     0

errors: No known data errors

root@oddity:~# zpool get all tank
NAME  PROPERTY                    VALUE                       SOURCE
tank  size                        29T                         -
tank  capacity                    57%                         -
tank  altroot                     -                           default
tank  health                      ONLINE                      -
tank  guid                        18319514605431597227        default
tank  version                     -                           default
tank  bootfs                      -                           default
tank  delegation                  on                          default
tank  autoreplace                 off                         default
tank  cachefile                   -                           default
tank  failmode                    wait                        default
tank  listsnapshots               off                         default
tank  autoexpand                  off                         default
tank  dedupditto                  0                           default
tank  dedupratio                  1.00x                       -
tank  free                        12.3T                       -
tank  allocated                   16.7T                       -
tank  readonly                    off                         -
tank  ashift                      12                          local
tank  comment                     -                           default
tank  expandsize                  -                           -
tank  freeing                     0                           default
tank  fragmentation               5%                          -
tank  leaked                      0                           default
tank  feature@async_destroy       enabled                     local
tank  feature@empty_bpobj         active                      local
tank  feature@lz4_compress        active                      local
tank  feature@spacemap_histogram  active                      local
tank  feature@enabled_txg         active                      local
tank  feature@hole_birth          active                      local
tank  feature@extensible_dataset  active                      local
tank  feature@embedded_data       active                      local
tank  feature@bookmarks           enabled                     local
tank  feature@filesystem_limits   enabled                     local
tank  feature@large_blocks        active                      local

root@oddity:~# zfs get all tank
NAME  PROPERTY              VALUE                  SOURCE
tank  type                  filesystem             -
tank  creation              Fri May 13 19:22 2016  -
tank  used                  11.8T                  -
tank  available             8.13T                  -
tank  referenced            222K                   -
tank  compressratio         1.03x                  -
tank  mounted               yes                    -
tank  quota                 none                   default
tank  reservation           none                   default
tank  recordsize            1M                     local
tank  mountpoint            /tank                  default
tank  sharenfs              off                    default
tank  checksum              on                     default
tank  compression           lz4                    local
tank  atime                 off                    local
tank  devices               on                     default
tank  exec                  on                     default
tank  setuid                on                     default
tank  readonly              off                    default
tank  zoned                 off                    default
tank  snapdir               hidden                 default
tank  aclinherit            passthrough            local
tank  canmount              on                     default
tank  xattr                 sa                     local
tank  copies                1                      default
tank  version               5                      -
tank  utf8only              off                    -
tank  normalization         none                   -
tank  casesensitivity       mixed                  -
tank  vscan                 off                    default
tank  nbmand                off                    default
tank  sharesmb              off                    default
tank  refquota              none                   default
tank  refreservation        none                   default
tank  primarycache          all                    default
tank  secondarycache        all                    default
tank  usedbysnapshots       154K                   -
tank  usedbydataset         222K                   -
tank  usedbychildren        11.8T                  -
tank  usedbyrefreservation  0                      -
tank  logbias               latency                default
tank  dedup                 off                    default
tank  mlslabel              none                   default
tank  sync                  standard               default
tank  refcompressratio      1.00x                  -
tank  written               0                      -
tank  logicalused           12.8T                  -
tank  logicalreferenced     41K                    -
tank  filesystem_limit      none                   default
tank  snapshot_limit        none                   default
tank  filesystem_count      none                   default
tank  snapshot_count        none                   default
tank  snapdev               hidden                 default
tank  acltype               posixacl               local
tank  context               none                   default
tank  fscontext             none                   default
tank  defcontext            none                   default
tank  rootcontext           none                   default
tank  relatime              off                    default
tank  redundant_metadata    all                    default
tank  overlay               off                    default

The target pool:

root@ubackup:~# modinfo zfs
filename:       /lib/modules/4.10.0-22-generic/kernel/zfs/zfs/zfs.ko
version:        0.6.5.9-2
license:        CDDL
author:         OpenZFS on Linux
description:    ZFS
srcversion:     42C4AB70887EA26A9970936
depends:        spl,znvpair,zcommon,zunicode,zavl
vermagic:       4.10.0-22-generic SMP mod_unload
...
root@ubackup:~# zpool status
  pool: btank
 state: ONLINE
  scan: scrub repaired 0 in 3h36m with 0 errors on Tue Jun 13 13:34:08 2017
config:

        NAME                                          STATE     READ WRITE CKSUM
        btank                                         ONLINE       0     0     0
          raidz1-0                                    ONLINE       0     0     0
            ata-WDC_WD30EZRX-00MMMB0_WD-XXXXXXXXXXXX  ONLINE       0     0     0
            ata-WDC_WD30EZRX-00MMMB0_WD-XXXXXXXXXXXX  ONLINE       0     0     0
            ata-WDC_WD30EZRX-00MMMB0_WD-XXXXXXXXXXXX  ONLINE       0     0     0
            ata-WDC_WD30EZRX-00MMMB0_WD-XXXXXXXXXXXX  ONLINE       0     0     0
            ata-WDC_WD30EZRX-00MMMB0_WD-XXXXXXXXXXXX  ONLINE       0     0     0
            ata-WDC_WD30EZRX-00MMMB0_WD-XXXXXXXXXXXX  ONLINE       0     0     0
            ata-WDC_WD30EZRX-00MMMB0_WD-XXXXXXXXXXXX  ONLINE       0     0     0
            ata-WDC_WD30EZRX-00MMMB0_WD-XXXXXXXXXXXX  ONLINE       0     0     0
            ata-WDC_WD30EZRX-00MMMB0_WD-XXXXXXXXXXXX  ONLINE       0     0     0

errors: No known data errors
root@ubackup:~# zpool get all btank
NAME   PROPERTY                    VALUE                       SOURCE
btank  size                        24.5T                       -
btank  capacity                    23%                         -
btank  altroot                     -                           default
btank  health                      ONLINE                      -
btank  guid                        14601555808903550874        default
btank  version                     -                           default
btank  bootfs                      -                           default
btank  delegation                  on                          default
btank  autoreplace                 off                         default
btank  cachefile                   -                           default
btank  failmode                    wait                        default
btank  listsnapshots               off                         default
btank  autoexpand                  off                         default
btank  dedupditto                  0                           default
btank  dedupratio                  1.00x                       -
btank  free                        18.9T                       -
btank  allocated                   5.64T                       -
btank  readonly                    off                         -
btank  ashift                      12                          local
btank  comment                     -                           default
btank  expandsize                  -                           -
btank  freeing                     0                           default
btank  fragmentation               9%                          -
btank  leaked                      0                           default
btank  feature@async_destroy       enabled                     local
btank  feature@empty_bpobj         active                      local
btank  feature@lz4_compress        active                      local
btank  feature@spacemap_histogram  active                      local
btank  feature@enabled_txg         active                      local
btank  feature@hole_birth          active                      local
btank  feature@extensible_dataset  active                      local
btank  feature@embedded_data       active                      local
btank  feature@bookmarks           enabled                     local
btank  feature@filesystem_limits   enabled                     local
btank  feature@large_blocks        active                      local
root@ubackup:~# zfs get all btank
NAME   PROPERTY              VALUE                  SOURCE
btank  type                  filesystem             -
btank  creation              Mon Jun 12 18:41 2017  -
btank  used                  5.01T                  -
btank  available             16.1T                  -
btank  referenced            171K                   -
btank  compressratio         1.03x                  -
btank  mounted               yes                    -
btank  quota                 none                   default
btank  reservation           none                   default
btank  recordsize            1M                     local
btank  mountpoint            /btank                 default
btank  sharenfs              off                    default
btank  checksum              on                     default
btank  compression           lz4                    local
btank  atime                 off                    local
btank  devices               on                     default
btank  exec                  on                     default
btank  setuid                on                     default
btank  readonly              off                    default
btank  zoned                 off                    default
btank  snapdir               hidden                 default
btank  aclinherit            passthrough            local
btank  canmount              on                     default
btank  xattr                 sa                     local
btank  copies                1                      default
btank  version               5                      -
btank  utf8only              on                     -
btank  normalization         formD                  -
btank  casesensitivity       mixed                  -
btank  vscan                 off                    default
btank  nbmand                off                    default
btank  sharesmb              off                    default
btank  refquota              none                   default
btank  refreservation        none                   default
btank  primarycache          all                    default
btank  secondarycache        all                    default
btank  usedbysnapshots       0                      -
btank  usedbydataset         171K                   -
btank  usedbychildren        5.01T                  -
btank  usedbyrefreservation  0                      -
btank  logbias               latency                default
btank  dedup                 off                    default
btank  mlslabel              none                   default
btank  sync                  disabled               local
btank  refcompressratio      1.00x                  -
btank  written               171K                   -
btank  logicalused           5.18T                  -
btank  logicalreferenced     40K                    -
btank  filesystem_limit      none                   default
btank  snapshot_limit        none                   default
btank  filesystem_count      none                   default
btank  snapshot_count        none                   default
btank  snapdev               hidden                 default
btank  acltype               posixacl               local
btank  context               none                   default
btank  fscontext             none                   default
btank  defcontext            none                   default
btank  rootcontext           none                   default
btank  relatime              off                    default
btank  redundant_metadata    all                    default
btank  overlay               off                    default

While the issue was observed with multiple datasets, I'll focus on a smaller one. First, the source (some uneventful snapshots omitted):

root@oddity:~# zfs list -o space -r tank/dataz/Backup
NAME               AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
tank/dataz/Backup  8.13T   126G      461K    126G              0          0
root@oddity:~# zfs list -t snapshot -r tank/dataz/Backup
NAME                                                      USED  AVAIL  REFER  MOUNTPOINT
tank/dataz/Backup@zfs-auto-snap_monthly-2017-06-01-0100      0      -   125G  -
tank/dataz/Backup@zfs-auto-snap_hourly-2017-06-12-1900       0      -   125G  -
tank/dataz/Backup@zfs-auto-snap_hourly-2017-06-12-2000    205K      -   125G  -
tank/dataz/Backup@zfs-auto-snap_hourly-2017-06-12-2100       0      -   126G  -
tank/dataz/Backup@zfs-auto-snap_hourly-2017-06-13-0900       0      -   126G  -

The initial send to ubackup has been performed with the -L and -e flags (not sure if this is relevant). Then performing an incremental send with the -L flag produces the expected result (size differences are due to different pool geometry):

root@oddity:~# zfs send -L -e -I tank/dataz/Backup@zfs-auto-snap_monthly-2017-06-01-0100 tank/dataz/Backup@zfs-auto-snap_hourly-2017-06-13-0900 | ssh ubackup "zfs receive btank/oddity/tank/dataz/Backup"

root@ubackup:~# zfs list -o space -r btank/oddity/tank/dataz/Backup
NAME                            AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
btank/oddity/tank/dataz/Backup  16.0T   133G      754K    133G              0          0
root@ubackup:~# zfs list -t snapshot -r btank/oddity/tank/dataz/Backup
NAME                                                                   USED  AVAIL  REFER  MOUNTPOINT
btank/oddity/tank/dataz/Backup@zfs-auto-snap_monthly-2017-06-01-0100  14.2K      -   131G  -
btank/oddity/tank/dataz/Backup@zfs-auto-snap_hourly-2017-06-12-1900   14.2K      -   131G  -
btank/oddity/tank/dataz/Backup@zfs-auto-snap_hourly-2017-06-12-2000    156K      -   131G  -
btank/oddity/tank/dataz/Backup@zfs-auto-snap_hourly-2017-06-12-2100   14.2K      -   133G  -
btank/oddity/tank/dataz/Backup@zfs-auto-snap_hourly-2017-06-13-0900       0      -   133G  -

But repeating the same send without the -L flag results in corruption:

root@ubackup:~# zfs rollback -r btank/oddity/tank/dataz/Backup@zfs-auto-snap_monthly-2017-06-01-0100

root@oddity:~# zfs send -e -I tank/dataz/Backup@zfs-auto-snap_monthly-2017-06-01-0100 tank/dataz/Backup@zfs-auto-snap_hourly-2017-06-13-0900 | ssh ubackup "zfs receive btank/oddity/tank/dataz/Backup"

root@ubackup:~# zfs list -o space -r btank/oddity/tank/dataz/Backup
NAME                            AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
btank/oddity/tank/dataz/Backup  16.0T   133G     19.0G    114G              0          0
root@ubackup:~# zfs list -t snapshot -r btank/oddity/tank/dataz/Backup
NAME                                                                   USED  AVAIL  REFER  MOUNTPOINT
btank/oddity/tank/dataz/Backup@zfs-auto-snap_monthly-2017-06-01-0100  14.2K      -   131G  -
btank/oddity/tank/dataz/Backup@zfs-auto-snap_hourly-2017-06-12-1900   14.2K      -   131G  -
btank/oddity/tank/dataz/Backup@zfs-auto-snap_hourly-2017-06-12-2000    156K      -   112G  -
btank/oddity/tank/dataz/Backup@zfs-auto-snap_hourly-2017-06-12-2100   14.2K      -   114G  -
btank/oddity/tank/dataz/Backup@zfs-auto-snap_hourly-2017-06-13-0900       0      -   114G  -

Notice how the REFER size drops and the USEDSNAP increase between the two runs. I looked around to find where the data had gone missing and found several random files whose reported sizes on disk were reduced to 512 bytes. Reading these files seems to return the full original size but with content all zeros.

I repeated the whole process multiple times, also recreating the target pool, with the same result. The affected files are always the same, but I haven't found a common characteristic among them. Scrubbing both pools finds no errors. Syslog shows nothing uncommon. I have never noted something similar before. The only notable change I made recently was that I started using sa based xattrs in May, but that might be unrelated.

I am happy to provide more info as far as I'm able.

[zrav]

I think I found the unifying characteristic of all corrupted files: All of them are in directories that have been updated via file creation or deletion between snapshots. Which of the files in the directory get corrupted seems to be random, though.

[zrav]

I'll try to put something together. I'm busy for the next week though, so it'll take a while.

[zrav]

I can reproduce the issue consistently with this script, tested on two different machines (0.6.5.6 and 0.6.5.9):

#!/bin/bash

BASE_DS="tank/issue6224"

zfs create -o recordsize=1M -o compression=lz4 $BASE_DS
zfs create $BASE_DS/source
cd /$BASE_DS/source
wget https://github.com/zfsonlinux/zfs/releases/download/zfs-0.6.5.10/zfs-0.6.5.10.tar.gz
zfs snapshot $BASE_DS/source@snap1
zfs send -L -e $BASE_DS/source@snap1 | zfs receive $BASE_DS/target
cp zfs-0.6.5.10.tar.gz zfs-0.6.5.10.tar.gz.1
zfs snapshot $BASE_DS/source@snap2
zfs send -e -i $BASE_DS/source@snap1 $BASE_DS/source@snap2 | zfs receive $BASE_DS/target

After running, check the size on disk of the files in the target dataset using du or ncdu. Note that the first file is only 512 bytes and all zero content.

NAME                   AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
tank/issue6224         8.13T  10.5M         0    205K              0      10.3M
tank/issue6224/source  8.13T  5.10M      136K   4.97M              0          0
tank/issue6224/target  8.13T  5.22M     2.52M   2.70M              0          0

[loli10K]

@zrav thank you for providing a working reproducer!

I've not had the time to debug this properly yet, but after a cursory reading of the code i think we're seeing the result of the following block in receive_object()

  2134          /*
  2135           * If we are losing blkptrs or changing the block size this must
  2136           * be a new file instance.  We must clear out the previous file
  2137           * contents before we can change this type of metadata in the dnode.
  2138           */
  2139          if (err == 0) {
  2140                  int nblkptr;
  2141  
  2142                  nblkptr = deduce_nblkptr(drro->drr_bonustype,
  2143                      drro->drr_bonuslen);
  2144  
  2145                  if (drro->drr_blksz != doi.doi_data_block_size ||
  2146                      nblkptr < doi.doi_nblkptr) {
  2147                          err = dmu_free_long_range(rwa->os, drro->drr_object,
  2148                              0, DMU_OBJECT_END);
  2149                          if (err != 0)
  2150                                  return (SET_ERROR(EINVAL));
  2151                  }
  2152          }

When we send the incremental stream our DRR OBJECT = inumof($corrupted_file) is sent with a block size that is different from the one already on disk (which was sent with -L the first time), so we "clear out the previous file contents":

First send (blksz = 1M)

[root@centos ~]# zfs send -L $POOLNAME/source@snap1 | zstreamdump -vv | grep "object = `inum /mnt/$POOLNAME/source/file1.dat`"
OBJECT object = 2 type = 19 bonustype = 44 blksz = 1048576 bonuslen = 176
FREE object = 2 offset = 5242880 length = -1
WRITE object = 2 type = 19 checksum type = 7 compression type = 0
WRITE object = 2 type = 19 checksum type = 7 compression type = 0
WRITE object = 2 type = 19 checksum type = 7 compression type = 0
WRITE object = 2 type = 19 checksum type = 7 compression type = 0
WRITE object = 2 type = 19 checksum type = 7 compression type = 0
FREE object = 2 offset = 5242880 length = 1068498944

Incremental send (blksz = 128K)

[root@centos ~]# zfs send -i $POOLNAME/source@snap1 $POOLNAME/source@snap2 | zstreamdump -vv | grep "object = `inum /mnt/$POOLNAME/source/file1.dat`"
OBJECT object = 2 type = 19 bonustype = 44 blksz = 131072 bonuslen = 176
FREE object = 2 offset = 5242880 length = -1

The resulting file on disk is:

[root@centos ~]# zdb -ddddddd -bbbbbbb $POOLNAME/target `inum /mnt/$POOLNAME/target/file1.dat`
Dataset testpool/target [ZPL], ID 76, cr_txg 18, 5.03M, 12 objects, rootbp DVA[0]=<0:1527000:200> DVA[1]=<0:1527200:200> [L0 DMU objset] fletcher4 lz4 LE contiguous unique double size=800L/200P birth=62L/62P fill=12 cksum=dd1da4093:545ff5a1b20:10a4e0a522bd7:2427c83816b784

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
         2    2   128K   128K      0     512   128K    0.00  ZFS plain file (K=inherit) (Z=inherit)
                                               176   bonus  System attributes
	dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED 
	dnode maxblkid: 0
	path	/file1.dat
	uid     0
	gid     0
	atime	Sat Jun 24 07:26:16 2017
	mtime	Sat Jun 12 07:26:16 2018
	ctime	Sat Jun 12 07:26:16 2018
	crtime	Sat Jun 12 07:26:15 2018
	gen	11
	mode	100644
	size	5242880
	parent	34
	links	1
	pflags	40800000004
	xattr	3
Indirect blocks:
               0 L1  HOLE [L1 ZFS plain file] size=20000L birth=43L

The code i was mentioning earlier was integrated with 6c59307 (Illumos 3693 - restore_object uses at least two transactions to restore an object), even before f1512ee (Illumos 5027 - zfs large block support): if all this is true i think every ZFS version since Illumos 5027 is affected, possibly on other platfroms too.

[loli10K]

This reproduces on current master and Illumos, which is quite troubling. Cross-platform reproducer here https://gist.github.com/loli10K/af70fb0f1aae7b174822aa5657e07c28.

ZoL:

root@linux:~# cat /sys/module/zfs/version 
0.7.0-rc4_67_g7e35ea783
root@linux:~# bash -x ./issue-6224.sh 
+ POOLNAME=testpool
+ is_linux
++ uname
+ [[ Linux == \L\i\n\u\x ]]
+ return 0
+ TMPDIR=/var/tmp
+ mountpoint -q /var/tmp
+ zpool destroy testpool
+ fallocate -l 128m /var/tmp/zpool.dat
+ zpool create -O mountpoint=/mnt/testpool testpool /var/tmp/zpool.dat
+ zfs create -o recordsize=1M testpool/source
+ dd if=/dev/urandom of=/mnt/testpool/source/file1.dat bs=1M count=5
5+0 records in
5+0 records out
5242880 bytes (5.2 MB) copied, 0.552689 s, 9.5 MB/s
+ zfs snapshot testpool/source@snap1
+ zfs send -L testpool/source@snap1
+ cat /var/tmp/full.dat
++ inum /mnt/testpool/source/file1.dat
++ stat -c %i /mnt/testpool/source/file1.dat
+ zstreamdump -v
+ grep 'object = 2'
OBJECT object = 2 type = 19 bonustype = 44 blksz = 1048576 bonuslen = 168
FREE object = 2 offset = 5242880 length = -1
WRITE object = 2 type = 19 checksum type = 7 compression type = 0
WRITE object = 2 type = 19 checksum type = 7 compression type = 0
WRITE object = 2 type = 19 checksum type = 7 compression type = 0
WRITE object = 2 type = 19 checksum type = 7 compression type = 0
WRITE object = 2 type = 19 checksum type = 7 compression type = 0
FREE object = 2 offset = 5242880 length = 1068498944
+ cat /var/tmp/full.dat
+ zfs receive -F testpool/target
+ cp /mnt/testpool/source/file1.dat /mnt/testpool/source/full.dat
+ du -sh /mnt/testpool/source/file1.dat /mnt/testpool/source/full.dat /mnt/testpool/target/file1.dat
5.1M	/mnt/testpool/source/file1.dat
5.1M	/mnt/testpool/source/full.dat
5.1M	/mnt/testpool/target/file1.dat
+ zfs snapshot testpool/source@snap2
+ zfs send -i testpool/source@snap1 testpool/source@snap2
+ cat /var/tmp/incr.dat
++ inum /mnt/testpool/source/file1.dat
++ stat -c %i /mnt/testpool/source/file1.dat
+ grep 'object = 2'
+ zstreamdump -v
OBJECT object = 2 type = 19 bonustype = 44 blksz = 131072 bonuslen = 168
FREE object = 2 offset = 5242880 length = -1
+ cat /var/tmp/incr.dat
+ zfs receive -F testpool/target
+ du -sh /mnt/testpool/source/file1.dat /mnt/testpool/source/full.dat /mnt/testpool/target/file1.dat /mnt/testpool/target/full.dat
5.1M	/mnt/testpool/source/file1.dat
5.1M	/mnt/testpool/source/full.dat
512	/mnt/testpool/target/file1.dat
5.1M	/mnt/testpool/target/full.dat
++ inum /mnt/testpool/target/file1.dat
++ stat -c %i /mnt/testpool/target/file1.dat
+ zdb -ddddddd -bbbbbbb testpool/target 2
Dataset testpool/target [ZPL], ID 268, cr_txg 16, 5.03M, 8 objects, rootbp DVA[0]=<0:14fa000:200> DVA[1]=<0:14fa200:200> [L0 DMU objset] fletcher4 lz4 LE contiguous unique double size=800L/200P birth=50L/50P fill=8 cksum=db27241b5:541d80cfdbd:10b6c0498b32d:2497961710b23a

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
         2    2   128K   128K      0     512   128K    0.00  ZFS plain file (K=inherit) (Z=inherit)
                                               168   bonus  System attributes
	dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED 
	dnode maxblkid: 0
	path	/file1.dat
	uid     0
	gid     0
	atime	Mon Jun 26 19:38:50 2017
	mtime	Mon Jun 26 19:38:49 2017
	ctime	Mon Jun 26 19:38:49 2017
	crtime	Mon Jun 26 19:38:49 2017
	gen	9
	mode	100644
	size	5242880
	parent	34
	links	1
	pflags	40800000004
Indirect blocks:
               0 L1  HOLE [L1 ZFS plain file] size=20000L birth=35L


root@linux:~# 

Illumos (SmartOS):

[root@52-54-00-d3-7a-01 ~]# uname -v
joyent_20170622T212149Z
[root@52-54-00-d3-7a-01 ~]# bash -x ./issue-6224.sh
+ POOLNAME=testpool
+ is_linux
++ uname
+ [[ SunOS == \L\i\n\u\x ]]
+ return 1
+ TMPDIR=/tmp
+ zpool destroy testpool
+ mkfile 128m /tmp/zpool.dat
+ zpool create -O mountpoint=/mnt/testpool testpool /tmp/zpool.dat
+ zfs create -o recordsize=1M testpool/source
+ dd if=/dev/urandom of=/mnt/testpool/source/file1.dat bs=1M count=5
5+0 records in
5+0 records out
5242880 bytes transferred in 0.078813 secs (66523111 bytes/sec)
+ zfs snapshot testpool/source@snap1
+ zfs send -L testpool/source@snap1
+ cat /tmp/full.dat
+ zstreamdump -v
++ inum /mnt/testpool/source/file1.dat
++ stat -c %i /mnt/testpool/source/file1.dat
+ grep 'object = 8'
OBJECT object = 8 type = 19 bonustype = 44 blksz = 1048576 bonuslen = 168
FREE object = 8 offset = 5242880 length = -1
WRITE object = 8 type = 19 checksum type = 7 compression type = 0
WRITE object = 8 type = 19 checksum type = 7 compression type = 0
WRITE object = 8 type = 19 checksum type = 7 compression type = 0
WRITE object = 8 type = 19 checksum type = 7 compression type = 0
WRITE object = 8 type = 19 checksum type = 7 compression type = 0
FREE object = 8 offset = 5242880 length = 1068498944
+ cat /tmp/full.dat
+ zfs receive -F testpool/target
+ cp /mnt/testpool/source/file1.dat /mnt/testpool/source/full.dat
+ du -sh /mnt/testpool/source/file1.dat /mnt/testpool/source/full.dat /mnt/testpool/target/file1.dat
 5.0M   /mnt/testpool/source/file1.dat
 5.0M   /mnt/testpool/source/full.dat
 5.0M   /mnt/testpool/target/file1.dat
+ zfs snapshot testpool/source@snap2
+ zfs send -i testpool/source@snap1 testpool/source@snap2
+ cat /tmp/incr.dat
+ zstreamdump -v
++ inum /mnt/testpool/source/file1.dat
++ stat -c %i /mnt/testpool/source/file1.dat
+ grep 'object = 8'
OBJECT object = 8 type = 19 bonustype = 44 blksz = 131072 bonuslen = 168
FREE object = 8 offset = 5242880 length = -1
+ cat /tmp/incr.dat
+ zfs receive -F testpool/target
+ du -sh /mnt/testpool/source/file1.dat /mnt/testpool/source/full.dat /mnt/testpool/target/file1.dat /mnt/testpool/target/full.dat
 5.0M   /mnt/testpool/source/file1.dat
 5.0M   /mnt/testpool/source/full.dat
   0K   /mnt/testpool/target/file1.dat
 5.0M   /mnt/testpool/target/full.dat
++ inum /mnt/testpool/target/file1.dat
++ stat -c %i /mnt/testpool/target/file1.dat
+ zdb -ddddddd -bbbbbbb testpool/target 8
Dataset testpool/target [ZPL], ID 51, cr_txg 15, 5.03M, 9 objects, rootbp DVA[0]=<0:14f8200:200> DVA[1]=<0:14f8400:200> [L0 DMU objset] fletcher4 lz4 LE contiguous unique double size=800L/200P birth=49L/49P fill=9 cksum=b47a0b8ce:44a5957f452:d88c20f8e067:1d7addab9d69db

    Object  lvl   iblk   dblk  dsize  lsize   %full  type
         8    2   128K   128K      0   128K    0.00  ZFS plain file (K=inherit) (Z=inherit)
                                        168   bonus  System attributes
        dnode flags: USED_BYTES USERUSED_ACCOUNTED 
        dnode maxblkid: 0
        path    /file1.dat
        uid     0
        gid     0
        atime   Mon Jun 26 17:40:50 2017
        mtime   Mon Jun 26 17:40:50 2017
        ctime   Mon Jun 26 17:40:50 2017
        crtime  Mon Jun 26 17:40:50 2017
        gen     8
        mode    100644
        size    5242880
        parent  4
        links   1
        pflags  40800000004
Indirect blocks:
               0 L1  HOLE [L1 ZFS plain file] size=20000L birth=34L


[root@52-54-00-d3-7a-01 ~]# 

[rottegift]

FWIW, although it says above that there is a cross-platform that reproduces on Illumos, @lundman notes that it hasn't made the slack and wonders if @ahrens has seen this.

(Reproduces in openzfsonosx too, and annoyingly I have a few affected files in backups that thankfully I haven't had to restore from.)

[zrav]

@behlendorf any particular reason this wasn't revisited?

[behlendorf]

Available developer resources, I'll add it to the 0.8 milestone so we don't loose track of it.

[loli10K]

Just for reference, a similar issue was reported:

• https://www.illumos.org/issues/9952
• 9952 Block size change during zfs receive drops spill block openzfs#705

[ahrens]

@loli10K that issue is related but different. I would guess the problem here is with turning -L on/off between incrementals. The incremental will report the drr_blksz in the OBJECT record as different from what is on the receiving system, so the recv assumes that the object must have been freed and reallocated, because the block size can't change on the sending system (if there are multiple blocks).

The solution might be a bit tricky, but I think it will have to involve disallowing some kinds of zfs receive.