pblk: corrupted read LBA

[HongweiQin]

Hi,

When I run FEMU as a write box, instantiate pblk, and mkfs. Dmesg warns me "corrupted read LBA".

This happens each time when pblk tries to read some data from the device.
It seems that the rqd didn't return the LBA metadata correctly.
Same configuration under qemu-nvme may not print same warnings.

Note that although dmesg warns me, the mkfs command returns successfully and I can mount the device subsequently.

Did I mis-reconfigured?

Guest OS :
CentOS 7
Kernel version 4.16.0

VM startup scripts:

/root/qhw/femu/build-femu/x86_64-softmmu/qemu-system-x86_64 \
        -name "qhwVM" \
        -m 32G \
        -smp 32 \
        --enable-kvm \
        -net nic,macaddr=52:54:00:17:21:72 -net tap,ifname=tap0,script=/var/lib/libvirt/images/qemu-ifup.sh,downscript=no \
        -device virtio-scsi-pci,id=scsi0 \
        -hda /mnt/sdc/qhw/images/qhwImage.qcow2 \
        -hdb /home/qhw/VMimages/backdrive.raw \
        -drive file=/home/qhw/VMimages/vssd1.raw,if=none,aio=threads,format=raw,id=id0 \
        -device nvme,drive=id0,serial=serial0,id=nvme0,namespaces=1,lver=1,lmetasize=16,ll2pmode=0,nlbaf=5,lba_index=3,mdts=10,lnum_ch=8,lnum_lun=4,lnum_pln=2,lsec_size=4096,lsecs_per_pg=4,lpgs_per_blk=512,ldebug=0,femu_mode=0 \
        -qmp unix:./qmp-sock,server,nowait

Warning messages look like this:

May 17 22:54:15 qhwVirt kernel: ------------[ cut here ]------------
May 17 22:54:15 qhwVirt kernel: pblk: corrupted read LBA
May 17 22:54:15 qhwVirt kernel: WARNING: CPU: 26 PID: 8489 at /usr/src/kernels/linux/drivers/lightnvm/pblk-read.c:129 __pblk_end_io_read+0x1a0/0x210 [pblk]
May 17 22:54:15 qhwVirt kernel: Modules linked in: rwGenerator(OE) pblk(OE) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ppdev parport_pc parport pcspkr i2c_piix4 sg ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod ata_generic pata_acpi bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm virtio_scsi ata_piix libata e1000 virtio_pci i2c_core virtio_ring serio_raw floppy virtio dm_mirror dm_region_hash dm_log
May 17 22:54:15 qhwVirt kernel: dm_mod
May 17 22:54:15 qhwVirt kernel: CPU: 26 PID: 8489 Comm: bash Tainted: G           OE    4.16.00adb32858_DisableNVMeDoorbellWriteForFemu+ #6
May 17 22:54:15 qhwVirt kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014
May 17 22:54:15 qhwVirt kernel: RIP: 0010:__pblk_end_io_read+0x1a0/0x210 [pblk]
May 17 22:54:15 qhwVirt kernel: RSP: 0018:ffff9cf8df483cd0 EFLAGS: 00010086
May 17 22:54:15 qhwVirt kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000006
May 17 22:54:15 qhwVirt kernel: RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff9cf8df496970
May 17 22:54:15 qhwVirt kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000000002c8
May 17 22:54:15 qhwVirt kernel: R10: 0000000000000004 R11: 0000000000000000 R12: ffff9cf8ddfc2c40
May 17 22:54:15 qhwVirt kernel: R13: ffff9cf8d2e63e00 R14: 0000000000000001 R15: ffff9cf8dd3b9800
May 17 22:54:15 qhwVirt kernel: FS:  00007fe0895df740(0000) GS:ffff9cf8df480000(0000) knlGS:0000000000000000
May 17 22:54:15 qhwVirt kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 17 22:54:15 qhwVirt kernel: CR2: 0000000000f6d3c8 CR3: 00000006c6adc000 CR4: 00000000000006e0
May 17 22:54:15 qhwVirt kernel: Call Trace:
May 17 22:54:15 qhwVirt kernel: <IRQ>
May 17 22:54:15 qhwVirt kernel: nvme_nvm_end_io+0x2a/0x40
May 17 22:54:15 qhwVirt kernel: __blk_mq_complete_request+0xb9/0x180
May 17 22:54:15 qhwVirt kernel: blk_mq_complete_request+0x63/0xb0
May 17 22:54:15 qhwVirt kernel: nvme_process_cq+0xdd/0x1a0
May 17 22:54:15 qhwVirt kernel: nvme_irq+0x1e/0x50
May 17 22:54:15 qhwVirt kernel: __handle_irq_event_percpu+0x3a/0x1b0
May 17 22:54:15 qhwVirt kernel: handle_irq_event_percpu+0x30/0x70
May 17 22:54:15 qhwVirt kernel: handle_irq_event+0x3d/0x60
May 17 22:54:15 qhwVirt kernel: handle_edge_irq+0x8a/0x190
May 17 22:54:15 qhwVirt kernel: handle_irq+0xa7/0x130
May 17 22:54:15 qhwVirt kernel: ? kvm_clock_read+0x21/0x30
May 17 22:54:15 qhwVirt kernel: ? sched_clock+0x5/0x10
May 17 22:54:15 qhwVirt kernel: do_IRQ+0x43/0xc0
May 17 22:54:15 qhwVirt kernel: common_interrupt+0xf/0xf
May 17 22:54:15 qhwVirt kernel: RIP: 0010:__do_softirq+0x6f/0x288
May 17 22:54:15 qhwVirt kernel: RSP: 0018:ffff9cf8df483f78 EFLAGS: 00000206 ORIG_RAX: ffffffffffffffde
May 17 22:54:15 qhwVirt kernel: RAX: ffff9cf8d61a5800 RBX: ffff9cf8df495f40 RCX: 0000000000000002
May 17 22:54:15 qhwVirt kernel: RDX: 0000000000000000 RSI: 000000000000ee44 RDI: 0000000000000838
May 17 22:54:15 qhwVirt kernel: RBP: 0000000000000000 R08: 00000000e8ba357a R09: 0000000000000001
May 17 22:54:15 qhwVirt kernel: R10: 0000000000000004 R11: 0000000000000005 R12: 0000000000000000
May 17 22:54:15 qhwVirt kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
May 17 22:54:15 qhwVirt kernel: ? common_interrupt+0xa/0xf
May 17 22:54:15 qhwVirt kernel: irq_exit+0xd5/0xe0
May 17 22:54:15 qhwVirt kernel: smp_apic_timer_interrupt+0x60/0x140
May 17 22:54:15 qhwVirt kernel: apic_timer_interrupt+0xf/0x20
May 17 22:54:15 qhwVirt kernel: </IRQ>
May 17 22:54:15 qhwVirt kernel: RIP: 0010:get_signal+0x2da/0x6a0
May 17 22:54:15 qhwVirt kernel: RSP: 0018:ffffa8a64e34fd70 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff12
May 17 22:54:15 qhwVirt kernel: RAX: 0000000000000000 RBX: 0000000000000011 RCX: ffffa8a64e34fe28
May 17 22:54:15 qhwVirt kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff9cf8d4c920c8
May 17 22:54:15 qhwVirt kernel: RBP: ffffa8a64e34fe48 R08: 0000000000026e10 R09: ffffffff9b09b0b7
May 17 22:54:15 qhwVirt kernel: R10: ffff9cf8df4a6e10 R11: ffffce47a0539f00 R12: ffff9cf8d4c91ac8
May 17 22:54:15 qhwVirt kernel: R13: 0000000000000010 R14: ffff9cf8d4c91ac0 R15: ffff9cf8d61a5800
May 17 22:54:15 qhwVirt kernel: ? __dequeue_signal+0x177/0x240
May 17 22:54:15 qhwVirt kernel: do_signal+0x36/0x650
May 17 22:54:15 qhwVirt kernel: exit_to_usermode_loop+0x45/0x8f
May 17 22:54:15 qhwVirt kernel: do_syscall_64+0x172/0x1a0
May 17 22:54:15 qhwVirt kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2
May 17 22:54:15 qhwVirt kernel: RIP: 0033:0x7fe088c11480
May 17 22:54:15 qhwVirt kernel: RSP: 002b:00007ffd52f190b8 EFLAGS: 00000246 ORIG_RAX: 000000000000000e
May 17 22:54:15 qhwVirt kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007fe088c11480
May 17 22:54:15 qhwVirt kernel: RDX: 0000000000000000 RSI: 00007ffd52f19140 RDI: 0000000000000002
May 17 22:54:15 qhwVirt kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000f5b590
May 17 22:54:15 qhwVirt kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000
May 17 22:54:15 qhwVirt kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000f78760
May 17 22:54:15 qhwVirt kernel: Code: 4c 89 ff e8 33 76 ff ff e9 d3 fe ff ff 31 c0 e9 46 ff ff ff 48 c7 c7 48 81 4a c0 31 c0 48 89 54 24 08 48 89 0c 24 e8 50 d8 be da <0f> 0b 48 8b 54 24 08 48 8b 0c 24 e9 db fe ff ff 80 3d f4 ed 00
May 17 22:54:15 qhwVirt kernel: ---[ end trace c99d7929276443cd ]---

[huaicheng]

Hi,

When I run FEMU as a write box, instantiate pblk, and mkfs. Dmesg warns me "corrupted read LBA".

This happens each time when pblk tries to read some data from the device.
It seems that the rqd didn't return the LBA metadata correctly.
Same configuration under qemu-nvme may not print same warnings.

Thanks for reporting this.

We're aware of this bug. We're looking into this. However, this doesn't affect you to use LightNVM. It's just a warning that there is a metadata mismatch between LightNVMs' version and FEMU's. You can still use file systems on top of pblk targets and the data are still correct.

For now, we suggest to comment out the WARN() statement in function pblk_read_check()

[HongweiQin]

Thanks for your reply~