silent data corruption on ARM64?

[shaofeng66]

Describe the bug
writing file using command dd with variable bs paramters, all commands report success, but the md5 checksum verify fail for those written with bs > 512K.

the file:

# pwd
/mismatch
ls -lh 0-1M.dat
-rw-r--r-- 1 root root 1.0M May 27 11:07 0-1M.dat
# md5sum 0-1M.dat 
d547eae310098883cbdac2e6355218e4  0-1M.dat

dd with 256k,512k,1m,4m bs parameter:

# dd if=0-1M.dat of=/mnt/GB3669L8/checksum-mismatch/0-1M.dat.bs256K bs=256K
4+0 records in
4+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 21.7534 s, 48.2 kB/s
# dd if=0-1M.dat of=/mnt/GB3669L8/checksum-mismatch/0-1M.dat.bs512K bs=512K
2+0 records in
2+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00469052 s, 224 MB/s
# dd if=0-1M.dat of=/mnt/GB3669L8/checksum-mismatch/0-1M.dat.bs1M bs=1M
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00227947 s, 460 MB/s
# dd if=0-1M.dat of=/mnt/GB3669L8/checksum-mismatch/0-1M.dat.bs4M bs=4M
0+1 records in
0+1 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00223605 s, 469 MB/s

md5

# md5sum 0-1M.dat /mnt/GB3669L8/checksum-mismatch/0-1M.dat.bs*
d547eae310098883cbdac2e6355218e4  0-1M.dat
b227c070d2eeffd07825f10e89925a09  /mnt/GB3669L8/checksum-mismatch/0-1M.dat.bs1M
d547eae310098883cbdac2e6355218e4  /mnt/GB3669L8/checksum-mismatch/0-1M.dat.bs256K
b227c070d2eeffd07825f10e89925a09  /mnt/GB3669L8/checksum-mismatch/0-1M.dat.bs4M
d547eae310098883cbdac2e6355218e4  /mnt/GB3669L8/checksum-mismatch/0-1M.dat.bs512K

# ltfs -V
LTFS version 2.4.5.1 (Prelim).
LTFS Format Specification version 2.4.0.

# ps -ef | grep ltfs
root       16519       1  0 13:20 ?        00:00:00 ltfs -o devname=/dev/sg3 /mnt/GB3669L8 -o sync_type=time@60 -o trace

# uname -r
4.19.90-89.25.v2401.ky10.aarch64

# getconf PAGESIZE
65536

# lscpu
Architecture:            aarch64
  CPU op-mode(s):        32-bit, 64-bit
  Byte Order:            Little Endian
CPU(s):                  64
  On-line CPU(s) list:   0-63
Vendor ID:               ARM
  BIOS Vendor ID:        Hisilicon
  Model name:            Cortex-A72
    BIOS Model name:     Hi1616
    Model:               2
    Thread(s) per core:  1
    Core(s) per cluster: 16
    Socket(s):           2
    Cluster(s):          4
    Stepping:            r0p2
    BogoMIPS:            100.00
    Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
NUMA:                    
  NUMA node(s):          4
  NUMA node0 CPU(s):     0-15
  NUMA node1 CPU(s):     16-31
  NUMA node2 CPU(s):     32-47
  NUMA node3 CPU(s):     48-63
Vulnerabilities:         
  Gather data sampling:  Not affected
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec store bypass:     Vulnerable
  Spectre v1:            Mitigation; __user pointer sanitization
  Spectre v2:            Vulnerable
  Srbds:                 Not affected
  Tsx async abort:       Not affected

[shaofeng66]

In contrast, on AMD64:

# pwd
/mismatch
# ls -lh 0-1M.dat    
-rw-r--r-- 1 root root 1.0M May 27 13:50 0-1M.dat
# md5sum 0-1M.dat 
d547eae310098883cbdac2e6355218e4  0-1M.dat


# dd if=0-1M.dat of=/mnt/GB3984L8/checksum-mismatch/0-1M.dat.bs256K bs=256K
4+0 records in
4+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 61.2455 s, 17.1 kB/s
# dd if=0-1M.dat of=/mnt/GB3669L8/checksum-mismatch/0-1M.dat.bs512K bs=512K
dd: failed to open '/mnt/GB3669L8/checksum-mismatch/0-1M.dat.bs512K': No such file or directory
# dd if=0-1M.dat of=/mnt/GB3984L8/checksum-mismatch/0-1M.dat.bs512K bs=512K
2+0 records in
2+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544616 s, 19.3 MB/s
# dd if=0-1M.dat of=/mnt/GB3984L8/checksum-mismatch/0-1M.dat.bs1M bs=1M
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00425463 s, 246 MB/s
# dd if=0-1M.dat of=/mnt/GB3984L8/checksum-mismatch/0-1M.dat.bs4M bs=4M
0+1 records in
0+1 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043244 s, 242 MB/s

# md5sum 0-1M.dat /mnt/GB3984L8/checksum-mismatch/0-1M.dat.bs*
d547eae310098883cbdac2e6355218e4  0-1M.dat
d547eae310098883cbdac2e6355218e4  /mnt/GB3984L8/checksum-mismatch/0-1M.dat.bs1M
d547eae310098883cbdac2e6355218e4  /mnt/GB3984L8/checksum-mismatch/0-1M.dat.bs256K
d547eae310098883cbdac2e6355218e4  /mnt/GB3984L8/checksum-mismatch/0-1M.dat.bs4M
d547eae310098883cbdac2e6355218e4  /mnt/GB3984L8/checksum-mismatch/0-1M.dat.bs512K

[shaofeng66]

An interesting thing is the size are incorrect:

# ls -l /mnt/GB3669L8/checksum-mismatch/0-1M.dat.bs*
-rwxrwxrwx 1 root root  524288 May 27 13:42 /mnt/GB3669L8/checksum-mismatch/0-1M.dat.bs1M
-rwxrwxrwx 1 root root 1048576 May 27 13:41 /mnt/GB3669L8/checksum-mismatch/0-1M.dat.bs256K
-rwxrwxrwx 1 root root  524288 May 27 13:42 /mnt/GB3669L8/checksum-mismatch/0-1M.dat.bs4M
-rwxrwxrwx 1 root root 1048576 May 27 13:42 /mnt/GB3669L8/checksum-mismatch/0-1M.dat.bs512K

[madjesc]

Hello @shaofeng66
The size mismatch comes from amd64? or from arm64?
Besides that the difference on sizes seems to be stemming from 2 sides.

First fuse implementation, if you use the latest fuse3 version the problem is present on amd64, fuse sends the write request as the block size provided by dd command, if you use fuse2 the request is spliced into hardcoded 128Kb block sizes. Can you share the fuse version that are you using in both architectures? It can also be a fuse specific issue with different architectures on the same version.

Second is the custom ltfs ioscheduler, if you use the simple fcfs scheduler, using -o iosched_backend=fcfs, the problem is gone in both architectures and fuse versions. The problem is a miscalculated page cache request so the data is trimmed down into the cartridge block size (by default 512Kb and configurable in mkltfs).

Thanks for finding this issue, we will change the code accordingly so ltfs wont fail on bigger block size requests.

[shaofeng66]

The size mismatch comes from amd64? or from arm64?

it come from arm64.

First fuse implementation, if you use the latest fuse3 version the problem is present on amd64, fuse sends the write request as the block size provided by dd command, if you use fuse2 the request is spliced into hardcoded 128Kb block sizes. Can you share the fuse version that are you using in both architectures? It can also be a fuse specific issue with different architectures on the same version.

how to check the fuse version? I have below information:

on arm64:

# fusermount -V
fusermount version: 2.9.9

# modinfo fuse
filename:       /lib/modules/4.19.90-89.25.v2401.ky10.aarch64/kernel/fs/fuse/fuse.ko.xz
alias:          devname:fuse
alias:          char-major-10-229
alias:          fs-fuseblk
alias:          fs-fuse
license:        GPL
description:    Filesystem in Userspace
author:         Miklos Szeredi <miklos@szeredi.hu>
alias:          fs-fusectl
srcversion:     A852AF66507D33AF1E22813
depends:        
intree:         Y
name:           fuse
vermagic:       4.19.90-89.25.v2401.ky10.aarch64 SMP mod_unload modversions aarch64
sig_id:         PKCS#7
signer:         Kylin Enterprise Linux kernel signing key
sig_key:        17:CF:D9:76:C4:05:4B:A7:58:6D:94:E2:32:B0:B5:00:1B:56:FD:62
sig_hashalgo:   sha256
signature:      86:30:AF:B1:15:78:64:0A:79:D6:CB:2D:15:FA:36:08:FF:41:E9:C3:
                .....................................
                85:E6:23:66:29:70:FF:A4:F9:3F:64:B7
parm:           max_user_bgreq:Global limit for the maximum number of backgrounded requests an unprivileged user can set (uint)
parm:           max_user_congthresh:Global limit for the maximum congestion threshold an unprivileged user can set (uint)

on amd64:

# fusermount -V
fusermount version: 2.9.9

# modinfo fuse
filename:       /lib/modules/5.14.0-503.14.1.el9_5.x86_64/kernel/fs/fuse/fuse.ko.xz
alias:          devname:fuse
alias:          char-major-10-229
alias:          fs-fuseblk
alias:          fs-fuse
license:        GPL
description:    Filesystem in Userspace
author:         Miklos Szeredi <miklos@szeredi.hu>
alias:          fs-fusectl
rhelversion:    9.5
srcversion:     05B110AE2596A5E5CDB747E
depends:        
retpoline:      Y
intree:         Y
name:           fuse
vermagic:       5.14.0-503.14.1.el9_5.x86_64 SMP preempt mod_unload modversions 
sig_id:         PKCS#7
signer:         Rocky kernel signing key
sig_key:        6B:67:95:A3:78:7D:44:87:0A:18:07:7A:C1:74:3D:9F:44:DE:EA:E0
sig_hashalgo:   sha256
signature:      23:1F:CB:E5:21:3F:7A:28:D7:79:94:AB:78:1F:AE:B7:4F:F6:1E:E3:
                ..............................................
                59:2A:AC:BC
parm:           max_user_bgreq:Global limit for the maximum number of backgrounded requests an unprivileged user can set (uint)
parm:           max_user_congthresh:Global limit for the maximum congestion threshold an unprivileged user can set (uint)

[shaofeng66]

Second is the custom ltfs ioscheduler, if you use the simple fcfs scheduler, using -o iosched_backend=fcfs, the problem is gone in both architectures and fuse versions. The problem is a miscalculated page cache request so the data is trimmed down into the cartridge block size (by default 512Kb and configurable in mkltfs).

After switching to fcfs, the problem is gone.

Thanks for fixing this bug. Can you pls give more detail on code that cause this problem and why on AMD64 there was no such problem?

[shaofeng66]

looks like the return value of _unified_insert_new_request() at https://github.com/LinearTapeFileSystem/ltfs/blob/v2.4.8.3-10521/src/iosched/unified.c#L1793 should be (ssize_t)copy_count?

[madjesc]

looks like the return value of _unified_insert_new_request() at https://github.com/LinearTapeFileSystem/ltfs/blob/v2.4.8.3-10521/src/iosched/unified.c#L1793 should be (ssize_t)copy_count?

Yes, also the _unified_update_request return value, and we need to ensure that the caller is adding the remaining blocks.

how to check the fuse version? I have below information

Seems that both versions are the same, but for some reason in AMD fuse is splitting bs=1M to 8 blocks of 128KB and the ARM one does not, my machine is an AMD64 and it actually splits the request, I have a hunch that is something specific to fuse ARM

[madjesc]

@shaofeng66 its seems that the fuse difference is coming from your kernel module version and not from your libfuse version. It's odd because your kernel ARM version should be splitting the request into 128KB blocks according to the kernel history, and the AMD one should not be splitting and having this problem.

Could be that the Kylin kernel contains this patch and the Rocky one does not? Besides that this should not be happening and ltfs should handle these block sizes.

[shaofeng66]

@madjesc I have pagesize of 64KiB on Kylin kernel, while on Rocky it is 4KiB. With it multipled by FUSE_MAX_PAGES_PER_REQ=32, the max size of requested block would be 2MiB on Kylin?

# on Kylin
# getconf PAGESIZE
65536

[mrichmon]

This may be an issue with your HBA driver. I've seen some HBA drivers that assume the data block size is always 512KB. In these cases, the additional data beyond 512KB is silently dropped.

It is worthwhile documenting the HBA model and driver version on both platforms. If possible, try a different brand of HBA to see if the problem is localized to one HBA type.