Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Corrupt mmap()ed data when reading in parallel on clone #13608

Closed
TheUbuntuGuy opened this issue Jun 29, 2022 · 26 comments
Closed

Corrupt mmap()ed data when reading in parallel on clone #13608

TheUbuntuGuy opened this issue Jun 29, 2022 · 26 comments
Labels
Component: Memory Management kernel memory management Type: Defect Incorrect behavior (e.g. crash, hang)

Comments

@TheUbuntuGuy
Copy link

System information

Type Version/Name
Distribution Name Ubuntu
Distribution Version 20.04.4
Kernel Version 5.16.28 (tested back to 5.3)
Architecture amd64
OpenZFS Version 2.1.4 (tested back to 0.8.2)

Summary

Sometimes after creating a clone and having multiple processes read the same file(s) from that clone, one process will receive corrupt data from ZFS. The corruption is transient and in-memory only. The pool is always consistent as verified by scrubs, and re-reading the corrupt file succeeds. This is not caused by hardware, as it has been reproduced on at least 5 servers. More details on my testing and how to reproduce follow.

Symptoms from Read Corruption

The easiest way to reproduce is to perform a code compile with make running multiple parallel jobs within a Docker container on ZFS, as Docker uses clones (this is detailed below). This reads the gcc binary and its shared libraries multiple times. When a file is corrupted on read, it usually manifests as one of the following errors:

g++: error while loading shared libraries: /lib/x86_64-linux-gnu/librt.so.1: invalid ELF header

g++: fatal error: cannot execute '/usr/lib/gcc/x86_64-linux-gnu/9/cc1plus': execv: Permission denied

arm-none-eabi-gcc: fatal error: cannot execute '/usr/bin/../lib/gcc/arm-none-eabi/10.2.1/cc1': execv: Exec format error

g++: internal compiler error: Segmentation fault signal terminated program cc1plus

/bin/sh: 1: g++: Input/output error

ZFS Versions Tested

I have reproduced on all versions of ZFS between v0.8.2 and v2.1.4 and on Linux v5.3 through v5.16. This also happens on the Ubuntu 20.04.4 stock kernel and distribution-provided ZFS modules, so it is not unique to my kernel configuration.

HW Reproduced On

I have reproduced on at least 5 different machines, all running different hardware, from Intel 4th gen Core, to AMD 3rd gen Threadripper. All the systems use SSD-backed pools except for one with a 10k RPM HDD, so it appears to be possible to reproduce on slower storage. All systems use either a single vdev, or multiple top-level single disk vdevs. It can also be reproduced in a VM using libvirt/KVM.

Reproduction with Docker

The easiest way to reproduce (and the way this was first discovered) is using Docker to compile C/C++ with make. However Docker is not required to reproduce as I will explain shortly.

This was originally discovered with my company's codebase, but I have been able to reproduce with open source projects, of which I will provide 2 examples.

Prerequisites:

  • Having Docker running using the ZFS graphdriver.
  • I assume a non-root user named user exists with uid 1000, and can be substituted as needed.

Create a Docker image to build the code:

mkdir -p /home/user/buildimage && cd /home/user/buildimage
echo -e "FROM ubuntu:20.04\nRUN apt-get update && apt-get install -y libncursesw5-dev autotools-dev autoconf build-essential binutils automake libtool" > Dockerfile
docker build . -t buildimage:v1

Example 1: htop

Prepare for build:

cd /home/user
git clone https://github.com/htop-dev/htop.git htop
docker run --rm --init --user 1000 -it --mount type=bind,source=/home/user/htop,target=/home/user/htop -w /home/user/htop buildimage:v1 bash -c "./autogen.sh && ./configure"

The actual reproducer:

while :; do docker run --rm --init --user 1000 -it --mount type=bind,source=/home/user/htop,target=/home/user/htop -w /home/user/htop buildimage:v1 bash -c "make -j $(nproc) clean && make -j $(nproc)" || break; done

This will build the project over and over again until it fails. Unfortunately due to this being an intermittent issue, you may have to wait a while. Sometimes it fails within only a few iterations in under a minute. Other times it takes hours. I feel that my company's build system is faster at reproducing, however I cannot share that unfortunately.

Eventually, a file from the cloned dataset that Docker creates will be read incorrectly and will result in a failure. It usually manifests as one of the errors shown above.

Example 2: cpputest

Prepare for build:

cd /home/user
git clone https://github.com/cpputest/cpputest.git cpputest
mkdir cpputest/cpputest_build
docker run --rm --init --user 1000 -it --mount type=bind,source=/home/user/cpputest,target=/home/user/cpputest -w /home/user/cpputest/cpputest_build buildimage:v1 bash -c "autoreconf .. -i && ../configure"

The reproducer:

while :; do docker run --rm --init --user 1000 -it --mount type=bind,source=/home/user/cpputest,target=/home/user/cpputest -w /home/user/cpputest/cpputest_build buildimage:v1 bash -c "make -j $(nproc) clean && make -j $(nproc)" || break; done

Other Cases Tested

This does not appear to be a race condition with the creation of the clone. I inserted a 3 second delay between creation and starting make, and the issue is still reproducible.

Compiling with only 1 process (make -j 1) does not appear to trigger this behaviour. It is possible that it can happen and has simply not happened yet in my limited testing of this case. The minimum parallel job count which has successfully reproduced this is 4. It appears that the more cores the system has, the easier the issue is to reproduce. I have not tested more make jobs than cores.

Running the same Docker-based test on the same hardware using ext4 and overlayfs does not exhibit this issue, so the issue lies with ZFS.

It is also worth noting that I have used several versions of GCC, including versions for cross-compiling, so I do not see the version of GCC being important. I also expect this to be possible to reproduce without GCC, but I have not explored this yet.

Reproduction without Docker

To rule out Docker, I unpacked a Docker image (containing a root filesystem) into a dataset and iterated the following:

  1. Cloned the dataset
  2. chroot into the clone
  3. Run the compile job
  4. Destroy the clone

I was able to reproduce using this method, so this issue is not dependent on Docker. I have used datasets with both legacy and "normal" mountpoints.

To "prove" this is bad data from the filesystem, on one occasion a source code file was read incorrectly and I managed to get a look at the beginning of the file, which looked like this. This is clearly not data from another location in the same file.

                                                                                       �b� W     H���                                                                                               
               �� W     A���                                                c� W     y*Z                                                                                                              
c� W     �8�2                                               @y� W     �{g                                                                                                              
0c� W     Y��>                                                                                                              Hc� W     ����                                                                                                              
�� W     ��m�                                                                                                              �y� W    ��`�                                                                                                              
�y� W    ���                                                                                                              �y� W    �b�                                                                                                              
�y� W    ���                                                                                                              �y� W    �a�                                                                                                              
�y� W    ˽�                                                                                                               � W     ���                                               `� W     
��)�                                                                                                             �� W     �                                                                                                                                                                             
z� W  


                                                                                                                                                             �;%�                                                
z� W      J��                                               0z� W     �߶�                                               Pz� W  

I have a system setup to test this right now and I would be happy to run tests, try code changes, or provide debug output.

@TheUbuntuGuy TheUbuntuGuy added the Type: Defect Incorrect behavior (e.g. crash, hang) label Jun 29, 2022
@rincebrain
Copy link
Contributor

rincebrain commented Jun 29, 2022

So, this is quite strange, because the data should be checksummed on every actual disk read, and shouldn't really be mutated for its lifetime thereafter - that is, you shouldn't get an inconsistent view of the world.

Once it produces incorrect data reading, does it continue to do that again forever, or is it entirely transient?

It might be interesting to build ZFS with --enable-debug and ensure the zfs_flags module parameter has 16 (ZFS_DEBUG_MODIFY) OR'd into it - that checksums ARC buffers as they're in use, and should catch if the contents mutates between when it's supposed to. (To be clear, ZFS_DEBUG_MODIFY does not require a debug build to use, I just would be curious if the additional assertions from --enable-debug would trip as well.)

@TheUbuntuGuy
Copy link
Author

That sounds like a great test. I appreciate your suggestion. Despite my use of ZFS for almost a decade I am not very familiar with the internals. I will reproduce on both a debug build and with the ZFS_DEBUG_MODIFY flag enabled and report back tomorrow.

@TheUbuntuGuy
Copy link
Author

TheUbuntuGuy commented Jun 30, 2022

I tested the existing (non-debug) kernel and set ZFS_DEBUG_MODIFY and no abnormal messages were emitted in dbgmsg.

I compiled master with --enable-debug and gcc (cc1) triggered this panic:

Jun 29 20:53:20 iotestvm kernel: VERIFY(PageUptodate(pp)) failed
Jun 29 20:53:20 iotestvm kernel: PANIC at zfs_vnops_os.c:296:mappedread()
Jun 29 20:53:20 iotestvm kernel: Showing stack for process 26516
Jun 29 20:53:20 iotestvm kernel: CPU: 15 PID: 26516 Comm: cc1 Tainted: P           OE     5.16.18 #0
Jun 29 20:53:20 iotestvm kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Jun 29 20:53:20 iotestvm kernel: Call Trace:
Jun 29 20:53:20 iotestvm kernel:  <TASK>
Jun 29 20:53:20 iotestvm kernel:  dump_stack_lvl+0x48/0x5e
Jun 29 20:53:20 iotestvm kernel:  spl_panic+0xd1/0xe9 [spl]
Jun 29 20:53:20 iotestvm kernel:  ? preempt_count_add+0x68/0xa0
Jun 29 20:53:20 iotestvm kernel:  ? __filemap_get_folio+0x120/0x510
Jun 29 20:53:20 iotestvm kernel:  mappedread+0x6d/0x1c0 [zfs]
Jun 29 20:53:20 iotestvm kernel:  zfs_read+0x189/0x3d0 [zfs]
Jun 29 20:53:20 iotestvm kernel:  zpl_iter_read+0xdf/0x1c0 [zfs]
Jun 29 20:53:20 iotestvm kernel:  new_sync_read+0x11b/0x1a0
Jun 29 20:53:20 iotestvm kernel:  vfs_read+0xf0/0x190
Jun 29 20:53:20 iotestvm kernel:  ksys_read+0x5f/0xe0
Jun 29 20:53:20 iotestvm kernel:  do_syscall_64+0x3b/0xc0
Jun 29 20:53:20 iotestvm kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Jun 29 20:53:20 iotestvm kernel: RIP: 0033:0x7fe76f24f028
Jun 29 20:53:20 iotestvm kernel: Code: 44 24 c8 eb cc 0f 1f 44 00 00 f7 d8 89 05 30 01 01 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 08 c3 0f 1f 80 00 00 00 00 f7 d8 89 05 00 01
Jun 29 20:53:20 iotestvm kernel: RSP: 002b:00007ffde2b72438 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Jun 29 20:53:20 iotestvm kernel: RAX: ffffffffffffffda RBX: 00007ffde2b724ef RCX: 00007fe76f24f028
Jun 29 20:53:20 iotestvm kernel: RDX: 0000000000000340 RSI: 00007ffde2b72508 RDI: 0000000000000003
Jun 29 20:53:20 iotestvm kernel: RBP: 00007ffde2b72490 R08: 0000000000080000 R09: 00007fe76f22b4e0
Jun 29 20:53:20 iotestvm kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000003
Jun 29 20:53:20 iotestvm kernel: R13: 00007fe76f22b4e0 R14: 00007ffde2b72500 R15: 0000000000000340
Jun 29 20:53:20 iotestvm kernel:  </TASK>

Looking at the code it appears that ZFS keeps 2 copies of mmaped pages. If somehow multiple accesses can cause the 2 copies to become desynchronized, then this could completely explain the issue.

@rincebrain
Copy link
Contributor

...huh. Yeah, there it goes, on git (482505f).

The handwavey parts of the stack trace make me suspicious, let me go try a patch I have...

[270108.416880] VERIFY(PageUptodate(pp)) failed
[270108.416896] PANIC at zfs_vnops_os.c:296:mappedread()
[270108.416908] Showing stack for process 1836930
[270108.416919] CPU: 17 PID: 1836930 Comm: g++ Kdump: loaded Tainted: P           OE     5.10.0-15-amd64 #1 Debian 5.10.120-1
[270108.416945] Hardware name: Micro-Star International Co., Ltd. MS-7D50/MEG X570S ACE MAX (MS-7D50), BIOS 1.32 03/02/2022
[270108.416975] Call Trace:
[270108.416989]  dump_stack+0x6b/0x83
[270108.417007]  spl_panic+0xd4/0xfc [spl]
[270108.417066]  ? dbuf_rele_and_unlock+0x7b5/0x820 [zfs]
[270108.417108]  ? dmu_buf_hold_array_by_dnode+0x4b0/0x700 [zfs]
[270108.417146]  ? dmu_read_uio_dnode+0x13b/0x170 [zfs]
[270108.417160]  ? wait_on_page_bit_common+0x167/0x3b0
[270108.417171]  ? trace_event_raw_event_file_check_and_advance_wb_err+0xf0/0xf0
[270108.417187]  ? pagecache_get_page+0xbe/0x380
[270108.417228]  mappedread+0x6d/0x1d0 [zfs]
[270108.417268]  zfs_read+0x189/0x3d0 [zfs]
[270108.417305]  zpl_iter_read+0xdf/0x1c0 [zfs]
[270108.417317]  __kernel_read+0x149/0x2d0
[270108.417328]  load_elf_binary+0x188/0x15e0
[270108.417338]  ? tomoyo_find_next_domain+0x268/0x860
[270108.417351]  bprm_execve+0x2ed/0x6b0
[270108.417360]  do_execveat_common+0x1e5/0x220
[270108.417370]  __x64_sys_execve+0x39/0x50
[270108.417380]  do_syscall_64+0x33/0x80
[270108.417390]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[270108.417402] RIP: 0033:0x7feb34c806c7
[270108.417411] Code: ff ff 76 e7 f7 d8 64 41 89 00 eb df 0f 1f 80 00 00 00 00 f7 d8 64 41 89 00 eb dc 0f 1f 84 00 00 00 00 00 b8 3b 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 99 27 0f 00 f7 d8 64 89 01 48
[270108.417452] RSP: 002b:00007fff451f5f98 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
[270108.417469] RAX: ffffffffffffffda RBX: 00007feb34d7b530 RCX: 00007feb34c806c7
[270108.418064] RDX: 0000000001b94de0 RSI: 0000000001b95578 RDI: 0000000001b95870
[270108.418642] RBP: 0000000001b958f0 R08: 0000000000000000 R09: 0000000000000000
[270108.419200] R10: fffffffffffff28c R11: 0000000000000202 R12: 0000000000000001
[270108.419769] R13: 0000000000000002 R14: 0000000001b95870 R15: 0000000000000001

@TheUbuntuGuy TheUbuntuGuy changed the title Corrupt data read from ZFS when reading in parallel on clone Corrupt mmap()ed data when reading in parallel Jun 30, 2022
@rincebrain
Copy link
Contributor

You changed the title - in experimenting, I only saw it with clones, are you saying you've reproduced it without that?

@TheUbuntuGuy
Copy link
Author

My apologies, my testing did show it only with memory mapped files, but I have not definitively reproduced without a clone. I was just trying to make the title less unwieldy. For brevity I will put that back.

@TheUbuntuGuy TheUbuntuGuy changed the title Corrupt mmap()ed data when reading in parallel Corrupt mmap()ed data when reading in parallel on clone Jul 1, 2022
@behlendorf behlendorf added the Component: Memory Management kernel memory management label Sep 19, 2022
@ryao
Copy link
Contributor

ryao commented Sep 19, 2022

I realize that it is late to say this, but could you rebuild with --enable-debug --enable-debuginfo --enable-asan --enable-ubsan and try to reproduce this.

@TheUbuntuGuy
Copy link
Author

I'll try and do a test with those options some time this week.

@ryao
Copy link
Contributor

ryao commented Sep 19, 2022

I just realized that --enable-asan --enable-ubsan only applies to user space. You would need to recompile the kernel with those (and debuginfo) set in menuconfig and then rebuild ZFS with --enable-debug to get the full effect. :/

@TheUbuntuGuy
Copy link
Author

Good to note. I run my own kernel build so it shouldn't be an issue to get the sanitizer support built-in on my test bench.

@ryao
Copy link
Contributor

ryao commented Sep 19, 2022

Great. Keep in mind that the sanitizers have performance and memory overhead, so things will run somewhat slowly (although not as slowly as if you had somehow used valgrind on the kernel) and system memory usage will be a few times more than normal.

@rincebrain
Copy link
Contributor

rincebrain commented Sep 20, 2022

You'd also probably need a couple patches I've given up on upstreaming to make it not flood the logs with useless messages about lz4 and the crypto code using uninited memory, at least that's my recollection from the last time I broke out kASAN.

#13208

@ryao
Copy link
Contributor

ryao commented Sep 20, 2022

You'd also probably need a couple patches I've given up on upstreaming to make it not flood the logs with useless messages about lz4 and the crypto code using uninited memory, at least that's my recollection from the last time I broke out kASAN.

#13208

The last time I used kASAN was before we had LZ4 or encryption in ZFS, so I had been unaware of that.

@rincebrain
Copy link
Contributor

Sure, I wasn't remarking on it as a criticism or something, just to save a round trip of "hey I'm out of space because syslog flooded with these :C"

@TheUbuntuGuy
Copy link
Author

I compiled 5.18.14 with CONFIG_KASAN=y and CONFIG_DEBUG_INFO=y, and configured ZFS 2.1.5 release (with the patches from #13208) with --enable-asan --enable-systemd --enable-debug --enable-debuginfo.

After over an hour I did get another crash, but surprisingly there were no other log messages from the sanitizers. The only time they output anything at all was when running zpool import in the initramfs due to some memory leaks.

The assert is the same as before, but the stack is slightly different:

kernel: VERIFY(PageUptodate(pp)) failed
kernel: PANIC at zfs_vnops_os.c:296:mappedread()
kernel: Showing stack for process 1319467
kernel: CPU: 1 PID: 1319467 Comm: g++ Tainted: P           OE     5.18.14-with-kasan #0
kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
kernel: Call Trace:
kernel:  <TASK>
kernel:  dump_stack_lvl+0x45/0x5a
kernel:  spl_panic+0x12c/0x155 [spl]
kernel:  ? spl_dumpstack+0x33/0x33 [spl]
kernel:  ? __x64_sys_execve+0x52/0x60
kernel:  ? kfree+0xa2/0x360
kernel:  ? xas_start+0x7f/0x1c0
kernel:  ? xas_load+0xcf/0x150
kernel:  ? __rcu_read_unlock+0x48/0x70
kernel:  ? __filemap_get_folio+0x1d6/0x4e0
kernel:  ? generic_file_read_iter+0x280/0x280
kernel:  mappedread+0x15f/0x170 [zfs]
kernel:  zfs_read+0x2ab/0x540 [zfs]
kernel:  zpl_iter_read+0x13e/0x1e0 [zfs]
kernel:  ? zpl_uio_init.constprop.0+0x10/0x10 [zfs]
kernel:  ? __aa_label_next_not_in_set+0x1a0/0x1a0
kernel:  ? aa_file_perm+0x204/0x6b0
kernel:  ? __ia32_compat_sys_pwritev64+0x190/0x190
kernel:  __kernel_read+0x22f/0x4e0
kernel:  ? __ia32_compat_sys_pwritev64+0x190/0x190
kernel:  ? fsnotify_perm.part.0+0xa0/0x250
kernel:  ? fsnotify_perm.part.0+0xa0/0x250
kernel:  load_elf_binary+0x2ec/0x2380
kernel:  ? tomoyo_find_next_domain+0x4e4/0xd90
kernel:  ? load_elf_library+0x3b0/0x3b0
kernel:  ? ima_bprm_check+0x108/0x140
kernel:  ? ima_file_mprotect+0x280/0x280
kernel:  ? __srcu_read_unlock+0x1f/0x50
kernel:  ? load_script+0x25/0x310
kernel:  bprm_execve+0x3be/0xa00
kernel:  ? open_exec+0x50/0x50
kernel:  do_execveat_common.isra.0+0x260/0x310
kernel:  __x64_sys_execve+0x52/0x60
kernel:  do_syscall_64+0x31/0x50
kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
kernel: RIP: 0033:0x7fd3b094f17b
kernel: Code: 41 89 01 eb da 66 2e 0f 1f 84 00 00 00 00 00 f7 d8 64 41 89 01 eb d6 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 3b 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d e5 8c 10 00 f7 d8 64 89 01 48
kernel: RSP: 002b:00007ffddc8c07c8 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
kernel: RAX: ffffffffffffffda RBX: 00007fd3b0a5f570 RCX: 00007fd3b094f17b
kernel: RDX: 0000000000e72200 RSI: 0000000000e72918 RDI: 0000000000e72c80
kernel: RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000
kernel: R10: 000000000040132e R11: 0000000000000202 R12: 0000000000000000
kernel: R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000e72d00
kernel:  </TASK>

@ryao
Copy link
Contributor

ryao commented Oct 17, 2022

The sanitizers only applied to the userspace code. You need to enable them in a custom kernel build's make menuconfig to use them inside the kernel.

I wish that you had tested using 2.1.6. There are some undefined behavior fixes in that. Confirming it with those fixes in place would at least let us not need to worry about them.

@TheUbuntuGuy
Copy link
Author

TheUbuntuGuy commented Oct 17, 2022

Correct me if I am wrong, but are these Linux config options not what is required? Are there more options I am missing?

CONFIG_KASAN=y
CONFIG_KASAN_GENERIC=y
CONFIG_KASAN_OUTLINE=y
CONFIG_KASAN_STACK=y

I can re-test with 2.1.6 or mainline, but before I do I want to ensure my kernel is configured correctly.

Edit: I have missed CONFIG_KASAN_VMALLOC which I will enable and retest.

@ryao
Copy link
Contributor

ryao commented Oct 17, 2022

Those would enable KASAN inside the kernel. I noticed that you said that you used --enable-asan, so I thought that you had not compiled your kernel to use it.

You might want to also enable CONFIG_UBSAN=y, although that has a bigger performance impact than KASAN.

@TheUbuntuGuy
Copy link
Author

Turns out I also had that enabled as well (I configured this kernel weeks ago and just got to running it now):

CONFIG_UBSAN=y
CONFIG_UBSAN_BOUNDS=y
CONFIG_UBSAN_ONLY_BOUNDS=y
CONFIG_UBSAN_SHIFT=y
CONFIG_UBSAN_BOOL=y
CONFIG_UBSAN_ENUM=y
CONFIG_UBSAN_SANITIZE_ALL=y

@ryao
Copy link
Contributor

ryao commented Oct 17, 2022

There appears to be one more sanitizer that you can turn on if your kernel and toolchain are new enough:

https://www.kernel.org/doc/html/latest/dev-tools/kcsan.html?highlight=sanitizer

@rincebrain
Copy link
Contributor

Someone should poke @behlendorf, he and I looked at this for a few minutes because I had a convenient reproducer set up already, but I forget what came of it other than "well that didn't help" a few times.

@TheUbuntuGuy
Copy link
Author

TheUbuntuGuy commented Oct 18, 2022

Some more test results. I re-tested with CONFIG_KASAN_VMALLOC=y and ZFS mainline as of ab49df4 and the results were the same - no output from any sanitizer, trace as follows:

kernel: VERIFY(PageUptodate(pp)) failed
kernel: PANIC at zfs_vnops_os.c:298:mappedread()
kernel: Showing stack for process 524308
kernel: CPU: 16 PID: 524308 Comm: sed Tainted: P           OE     5.18.14-with-kasan2 #1
kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
kernel: Call Trace:
kernel:  <TASK>
kernel:  dump_stack_lvl+0x45/0x5a
kernel:  spl_panic+0x12c/0x155 [spl]
kernel:  ? spl_dumpstack+0x33/0x33 [spl]
kernel:  ? zpl_iter_read+0x13e/0x1e0 [zfs]
kernel:  ? do_syscall_64+0x31/0x50
kernel:  ? entry_SYSCALL_64_after_hwframe+0x44/0xae
kernel:  ? __d_path+0x100/0x100
kernel:  ? aa_audit_file+0x110/0x2a0
kernel:  ? update_file_ctx+0xc0/0xc0
kernel:  ? xas_start+0x7f/0x1c0
kernel:  ? xas_load+0xcf/0x150
kernel:  ? __rcu_read_unlock+0x48/0x70
kernel:  ? __filemap_get_folio+0x1d6/0x4e0
kernel:  ? generic_file_read_iter+0x280/0x280
kernel:  ? zfs_rangelock_enter_reader+0x1c7/0x2e0 [zfs]
kernel:  spl_assert+0x2f/0x30 [zfs]
kernel:  mappedread+0x15d/0x170 [zfs]
kernel:  zfs_read+0x271/0x430 [zfs]
kernel:  zpl_iter_read+0x13e/0x1e0 [zfs]
kernel:  ? zpl_uio_init.constprop.0+0x10/0x10 [zfs]
kernel:  ? __rcu_read_unlock+0x48/0x70
kernel:  ? aa_file_perm+0x204/0x6b0
kernel:  ? cp_old_stat+0x340/0x340
kernel:  new_sync_read+0x208/0x310
kernel:  ? __ia32_sys_llseek+0x1e0/0x1e0
kernel:  ? zpl_getattr_impl.constprop.0+0x121/0x170 [zfs]
kernel:  ? fsnotify_perm.part.0+0xa0/0x250
kernel:  vfs_read+0x1f0/0x250
kernel:  __x64_sys_pread64+0x117/0x140
kernel:  ? vfs_read+0x250/0x250
kernel:  do_syscall_64+0x31/0x50
kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
kernel: RIP: 0033:0x7f821901d05e
kernel: Code: 00 00 00 00 f7 d8 89 05 00 01 01 00 48 c7 c0 ff ff ff ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 49 89 ca b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 0a c3 66 0f 1f 84 00 00 00 00 00 f7 d8 89 05
kernel: RSP: 002b:00007ffe8d8683b8 EFLAGS: 00000206 ORIG_RAX: 0000000000000011
kernel: RAX: ffffffffffffffda RBX: 00007ffe8d8683c0 RCX: 00007f821901d05e
kernel: RDX: 0000000000000044 RSI: 00007ffe8d8683c0 RDI: 0000000000000003
kernel: RBP: 00007ffe8d868450 R08: 0000000000000003 R09: 0000000000000338
kernel: R10: 0000000000000338 R11: 0000000000000206 R12: 0000000000000044
kernel: R13: 00007f8218ffa9f0 R14: 00007ffe8d8688f8 R15: 00007ffe8d868ab8
kernel:  </TASK>

Now I tried using KCSAN (which requires disabling KASAN) using the same ZFS commit. There is some noise from this sanitizer as just idling the system prints a benign message every few seconds. I was still able to trigger the assert.

Not sure how much of the log is relevant, so here is the sanitizer output for the 12 seconds leading up to and just after the panic (at 01:25:12). I can provide a complete dump of the syslog for the system if anyone wants it.

Oct 18 01:25:00 zfstest kernel: ==================================================================
Oct 18 01:25:00 zfstest kernel: BUG: KCSAN: data-race in zrl_add_impl [zfs] / zrl_remove [zfs]
Oct 18 01:25:00 zfstest kernel:
Oct 18 01:25:00 zfstest kernel: write to 0xffffa0b94bc06450 of 8 bytes by task 222322 on cpu 11:
Oct 18 01:25:00 zfstest kernel:  zrl_add_impl+0x11b/0x1a0 [zfs]
Oct 18 01:25:00 zfstest kernel:
Oct 18 01:25:00 zfstest kernel: read to 0xffffa0b94bc06450 of 8 bytes by task 222327 on cpu 0:
Oct 18 01:25:00 zfstest kernel:  zrl_remove+0x19/0xb0 [zfs]
Oct 18 01:25:00 zfstest kernel:  dbuf_verify+0x414/0xbf0 [zfs]
Oct 18 01:25:00 zfstest kernel:  dbuf_hold_impl+0x23b/0xdf0 [zfs]
Oct 18 01:25:00 zfstest kernel:  dbuf_hold_level+0x2b/0x60 [zfs]
Oct 18 01:25:00 zfstest kernel:  dbuf_hold+0x12/0x20 [zfs]
Oct 18 01:25:00 zfstest kernel:  dmu_buf_hold_array_by_dnode+0x164/0x720 [zfs]
Oct 18 01:25:00 zfstest kernel:  dmu_read_uio_dnode+0x65/0x1b0 [zfs]
Oct 18 01:25:00 zfstest kernel:  dmu_read_uio_dbuf+0x64/0x90 [zfs]
Oct 18 01:25:00 zfstest kernel:  zfs_read+0x1ff/0x440 [zfs]
Oct 18 01:25:00 zfstest kernel:  zpl_iter_read+0xf3/0x180 [zfs]
Oct 18 01:25:00 zfstest kernel:  new_sync_read+0x1d9/0x2a0
Oct 18 01:25:00 zfstest kernel:  vfs_read+0x22a/0x290
Oct 18 01:25:00 zfstest kernel:  ksys_read+0x7f/0x110
Oct 18 01:25:00 zfstest kernel:  __x64_sys_read+0x3d/0x50
Oct 18 01:25:00 zfstest kernel:  do_syscall_64+0x2f/0x50
Oct 18 01:25:00 zfstest kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 18 01:25:00 zfstest kernel:
Oct 18 01:25:00 zfstest kernel: value changed: 0xffffa0b9e8db8000 -> 0x0000000000000000
Oct 18 01:25:00 zfstest kernel:
Oct 18 01:25:00 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:00 zfstest kernel: CPU: 0 PID: 222327 Comm: cc1plus Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:00 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:00 zfstest kernel: ==================================================================
Oct 18 01:25:00 zfstest kernel: ==================================================================
Oct 18 01:25:00 zfstest kernel: BUG: KCSAN: data-race in generic_fillattr / touch_atime
Oct 18 01:25:00 zfstest kernel:
Oct 18 01:25:00 zfstest kernel: write to 0xffffa0b95c904dc8 of 16 bytes by task 222310 on cpu 4:
Oct 18 01:25:00 zfstest kernel:  touch_atime+0x217/0x280
Oct 18 01:25:00 zfstest kernel:  file_accessed+0x1e/0x20 [zfs]
Oct 18 01:25:00 zfstest kernel:  zpl_file_accessed+0x78/0x90 [zfs]
Oct 18 01:25:00 zfstest kernel:  zpl_iter_read+0x152/0x180 [zfs]
Oct 18 01:25:00 zfstest kernel:  new_sync_read+0x1d9/0x2a0
Oct 18 01:25:00 zfstest kernel:  vfs_read+0x22a/0x290
Oct 18 01:25:00 zfstest kernel:  ksys_read+0x7f/0x110
Oct 18 01:25:00 zfstest kernel:  __x64_sys_read+0x3d/0x50
Oct 18 01:25:00 zfstest kernel:  do_syscall_64+0x2f/0x50
Oct 18 01:25:00 zfstest kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 18 01:25:00 zfstest kernel:
Oct 18 01:25:00 zfstest kernel: read to 0xffffa0b95c904dc8 of 16 bytes by task 222325 on cpu 13:
Oct 18 01:25:00 zfstest kernel:  generic_fillattr+0x1e8/0x2c0
Oct 18 01:25:00 zfstest kernel:  zfs_getattr_fast+0xc6/0x350 [zfs]
Oct 18 01:25:00 zfstest kernel:  zpl_getattr_impl.constprop.0+0x5c/0x1f0 [zfs]
Oct 18 01:25:00 zfstest kernel:  zpl_getattr+0xa/0x10 [zfs]
Oct 18 01:25:00 zfstest kernel:  vfs_getattr_nosec+0x17d/0x1b0
Oct 18 01:25:00 zfstest kernel:  vfs_fstat+0x45/0x80
Oct 18 01:25:00 zfstest kernel:  __do_sys_newfstat+0x26/0x60
Oct 18 01:25:00 zfstest kernel:  __x64_sys_newfstat+0x2b/0x30
Oct 18 01:25:00 zfstest kernel:  do_syscall_64+0x2f/0x50
Oct 18 01:25:00 zfstest kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 18 01:25:00 zfstest kernel:
Oct 18 01:25:00 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:00 zfstest kernel: CPU: 13 PID: 222325 Comm: cc1plus Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:00 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:00 zfstest kernel: ==================================================================
Oct 18 01:25:00 zfstest kernel: ==================================================================
Oct 18 01:25:00 zfstest kernel: BUG: KCSAN: data-race in osq_lock / osq_lock
Oct 18 01:25:00 zfstest kernel:
Oct 18 01:25:00 zfstest kernel: write (marked) to 0xffffa0bcaf8b1c00 of 8 bytes by task 222306 on cpu 9:
Oct 18 01:25:00 zfstest kernel:  osq_lock+0x2a0/0x2e0
Oct 18 01:25:00 zfstest kernel:  __mutex_lock.constprop.0+0x1ab/0xa40
Oct 18 01:25:00 zfstest kernel:  __mutex_lock_slowpath+0xf/0x10
Oct 18 01:25:00 zfstest kernel:  mutex_lock+0x93/0xa0
Oct 18 01:25:00 zfstest kernel:  zfs_getattr_fast+0xaf/0x350 [zfs]
Oct 18 01:25:00 zfstest kernel:  zpl_getattr_impl.constprop.0+0x5c/0x1f0 [zfs]
Oct 18 01:25:00 zfstest kernel:  zpl_getattr+0xa/0x10 [zfs]
Oct 18 01:25:00 zfstest kernel:  vfs_getattr_nosec+0x17d/0x1b0
Oct 18 01:25:00 zfstest kernel:  vfs_statx+0xbc/0x1d0
Oct 18 01:25:00 zfstest kernel:  vfs_fstatat+0x54/0x70
Oct 18 01:25:00 zfstest kernel:  __do_sys_newlstat+0x33/0x70
Oct 18 01:25:00 zfstest kernel:  __x64_sys_newlstat+0x2c/0x30
Oct 18 01:25:00 zfstest kernel:  do_syscall_64+0x2f/0x50
Oct 18 01:25:00 zfstest kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 18 01:25:00 zfstest kernel:
Oct 18 01:25:00 zfstest kernel: read to 0xffffa0bcaf8b1c00 of 8 bytes by task 222319 on cpu 2:
Oct 18 01:25:00 zfstest kernel:  osq_lock+0x218/0x2e0
Oct 18 01:25:00 zfstest kernel:  __mutex_lock.constprop.0+0x1ab/0xa40
Oct 18 01:25:00 zfstest kernel:  __mutex_lock_slowpath+0xf/0x10
Oct 18 01:25:00 zfstest kernel:  mutex_lock+0x93/0xa0
Oct 18 01:25:00 zfstest kernel:  zfs_getattr_fast+0xaf/0x350 [zfs]
Oct 18 01:25:00 zfstest kernel:  zpl_getattr_impl.constprop.0+0x5c/0x1f0 [zfs]
Oct 18 01:25:00 zfstest kernel:  zpl_getattr+0xa/0x10 [zfs]
Oct 18 01:25:00 zfstest kernel:  vfs_getattr_nosec+0x17d/0x1b0
Oct 18 01:25:00 zfstest kernel:  vfs_statx+0xbc/0x1d0
Oct 18 01:25:00 zfstest kernel:  vfs_fstatat+0x54/0x70
Oct 18 01:25:00 zfstest kernel:  __do_sys_newlstat+0x33/0x70
Oct 18 01:25:00 zfstest kernel:  __x64_sys_newlstat+0x2c/0x30
Oct 18 01:25:00 zfstest kernel:  do_syscall_64+0x2f/0x50
Oct 18 01:25:00 zfstest kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 18 01:25:00 zfstest kernel:
Oct 18 01:25:00 zfstest kernel: value changed: 0x0000000000000000 -> 0xffffa0bcafb31c00
Oct 18 01:25:00 zfstest kernel:
Oct 18 01:25:00 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:00 zfstest kernel: CPU: 2 PID: 222319 Comm: cc1plus Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:00 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:00 zfstest kernel: ==================================================================
Oct 18 01:25:01 zfstest kernel: ==================================================================
Oct 18 01:25:01 zfstest kernel: BUG: KCSAN: data-race in folio_mark_accessed / pagevec_lru_move_fn
Oct 18 01:25:01 zfstest kernel:
Oct 18 01:25:01 zfstest kernel: write (marked) to 0xffffe5d645519c80 of 8 bytes by task 222309 on cpu 1:
Oct 18 01:25:01 zfstest kernel:  folio_mark_accessed+0x3e/0x340
Oct 18 01:25:01 zfstest kernel:  mark_page_accessed+0x24/0x80
Oct 18 01:25:01 zfstest kernel:  unmap_page_range+0x85c/0x17a0
Oct 18 01:25:01 zfstest kernel:  unmap_single_vma+0xbf/0x160
Oct 18 01:25:01 zfstest kernel:  unmap_vmas+0xe7/0x1a0
Oct 18 01:25:01 zfstest kernel:  exit_mmap+0xde/0x230
Oct 18 01:25:01 zfstest kernel:  mmput+0xb5/0x1e0
Oct 18 01:25:01 zfstest kernel:  do_exit+0x549/0x1260
Oct 18 01:25:01 zfstest kernel:  do_group_exit+0x53/0x100
Oct 18 01:25:01 zfstest kernel:  __x64_sys_exit_group+0x21/0x30
Oct 18 01:25:01 zfstest kernel:  do_syscall_64+0x2f/0x50
Oct 18 01:25:01 zfstest kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 18 01:25:01 zfstest kernel:
Oct 18 01:25:01 zfstest kernel: read to 0xffffe5d645519c80 of 8 bytes by task 222324 on cpu 7:
Oct 18 01:25:01 zfstest kernel:  pagevec_lru_move_fn+0xf3/0x3b0
Oct 18 01:25:01 zfstest kernel:  folio_mark_accessed+0x2e4/0x340
Oct 18 01:25:01 zfstest kernel:  mark_page_accessed+0x24/0x80
Oct 18 01:25:01 zfstest kernel:  unmap_page_range+0x85c/0x17a0
Oct 18 01:25:01 zfstest kernel:  unmap_single_vma+0xbf/0x160
Oct 18 01:25:01 zfstest kernel:  unmap_vmas+0xe7/0x1a0
Oct 18 01:25:01 zfstest kernel:  exit_mmap+0xde/0x230
Oct 18 01:25:01 zfstest kernel:  mmput+0xb5/0x1e0
Oct 18 01:25:01 zfstest kernel:  do_exit+0x549/0x1260
Oct 18 01:25:01 zfstest kernel:  do_group_exit+0x53/0x100
Oct 18 01:25:01 zfstest kernel:  __x64_sys_exit_group+0x21/0x30
Oct 18 01:25:01 zfstest kernel:  do_syscall_64+0x2f/0x50
Oct 18 01:25:01 zfstest kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 18 01:25:01 zfstest kernel:
Oct 18 01:25:01 zfstest kernel: value changed: 0x0017ffffc0000004 -> 0x0017ffffc0000006
Oct 18 01:25:01 zfstest kernel:
Oct 18 01:25:01 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:01 zfstest kernel: CPU: 7 PID: 222324 Comm: cc1plus Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:01 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:01 zfstest kernel: ==================================================================
Oct 18 01:25:01 zfstest kernel: ==================================================================
Oct 18 01:25:01 zfstest kernel: BUG: KCSAN: data-race in pagevec_lru_move_fn / workingset_activation
Oct 18 01:25:01 zfstest kernel:
Oct 18 01:25:01 zfstest kernel: read-write (marked) to 0xffffe5d645dfa700 of 8 bytes by task 222328 on cpu 16:
Oct 18 01:25:01 zfstest kernel:  pagevec_lru_move_fn+0xbe/0x3b0
Oct 18 01:25:01 zfstest kernel:  folio_mark_accessed+0x2e4/0x340
Oct 18 01:25:01 zfstest kernel:  mark_page_accessed+0x24/0x80
Oct 18 01:25:01 zfstest kernel:  unmap_page_range+0x85c/0x17a0
Oct 18 01:25:01 zfstest kernel:  unmap_single_vma+0xbf/0x160
Oct 18 01:25:01 zfstest kernel:  unmap_vmas+0xe7/0x1a0
Oct 18 01:25:01 zfstest kernel:  exit_mmap+0xde/0x230
Oct 18 01:25:01 zfstest kernel:  mmput+0xb5/0x1e0
Oct 18 01:25:01 zfstest kernel:  do_exit+0x549/0x1260
Oct 18 01:25:01 zfstest kernel:  do_group_exit+0x53/0x100
Oct 18 01:25:01 zfstest kernel:  __x64_sys_exit_group+0x21/0x30
Oct 18 01:25:01 zfstest kernel:  do_syscall_64+0x2f/0x50
Oct 18 01:25:01 zfstest kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 18 01:25:01 zfstest kernel:
Oct 18 01:25:01 zfstest kernel: read to 0xffffe5d645dfa700 of 8 bytes by task 222309 on cpu 1:
Oct 18 01:25:01 zfstest kernel:  workingset_activation+0xf5/0x1b0
Oct 18 01:25:01 zfstest kernel:  folio_mark_accessed+0x1ff/0x340
Oct 18 01:25:01 zfstest kernel:  mark_page_accessed+0x24/0x80
Oct 18 01:25:01 zfstest kernel:  unmap_page_range+0x85c/0x17a0
Oct 18 01:25:01 zfstest kernel:  unmap_single_vma+0xbf/0x160
Oct 18 01:25:01 zfstest kernel:  unmap_vmas+0xe7/0x1a0
Oct 18 01:25:01 zfstest kernel:  exit_mmap+0xde/0x230
Oct 18 01:25:01 zfstest kernel:  mmput+0xb5/0x1e0
Oct 18 01:25:01 zfstest kernel:  do_exit+0x549/0x1260
Oct 18 01:25:01 zfstest kernel:  do_group_exit+0x53/0x100
Oct 18 01:25:01 zfstest kernel:  __x64_sys_exit_group+0x21/0x30
Oct 18 01:25:01 zfstest kernel:  do_syscall_64+0x2f/0x50
Oct 18 01:25:01 zfstest kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 18 01:25:01 zfstest kernel:
Oct 18 01:25:01 zfstest kernel: value changed: 0x0017ffffc0000014 -> 0x0017ffffc0000034
Oct 18 01:25:01 zfstest kernel:
Oct 18 01:25:01 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:01 zfstest kernel: CPU: 1 PID: 222309 Comm: cc1plus Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:01 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:01 zfstest kernel: ==================================================================
Oct 18 01:25:02 zfstest kernel: ==================================================================
Oct 18 01:25:02 zfstest kernel: BUG: KCSAN: data-race in arc_tempreserve_space [zfs] / arc_tempreserve_space [zfs]
Oct 18 01:25:02 zfstest kernel:
Oct 18 01:25:02 zfstest kernel: read-write (marked) to 0xffffffffc0c4b910 of 8 bytes by task 222593 on cpu 22:
Oct 18 01:25:02 zfstest kernel:  arc_tempreserve_space+0x198/0x380 [zfs]
Oct 18 01:25:02 zfstest kernel:  dsl_dir_tempreserve_space+0xcf/0x200 [zfs]
Oct 18 01:25:02 zfstest kernel:
Oct 18 01:25:02 zfstest kernel: read to 0xffffffffc0c4b910 of 8 bytes by task 222603 on cpu 12:
Oct 18 01:25:02 zfstest kernel:  arc_tempreserve_space+0xe1/0x380 [zfs]
Oct 18 01:25:02 zfstest kernel:  dsl_dir_tempreserve_space+0xcf/0x200 [zfs]
Oct 18 01:25:02 zfstest kernel:
Oct 18 01:25:02 zfstest kernel: value changed: 0x000000000000959c -> 0x000000000000b390
Oct 18 01:25:02 zfstest kernel:
Oct 18 01:25:02 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:02 zfstest kernel: CPU: 12 PID: 222603 Comm: as Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:02 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:02 zfstest kernel: ==================================================================
Oct 18 01:25:03 zfstest kernel: ==================================================================
Oct 18 01:25:03 zfstest kernel: BUG: KCSAN: data-race in zrl_add_impl [zfs] / zrl_remove [zfs]
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: write to 0xffffa0b94bdc0310 of 8 bytes by task 222804 on cpu 14:
Oct 18 01:25:03 zfstest kernel:  zrl_remove+0x61/0xb0 [zfs]
Oct 18 01:25:03 zfstest kernel:  dbuf_verify+0x414/0xbf0 [zfs]
Oct 18 01:25:03 zfstest kernel:  dbuf_rele_and_unlock+0x5e/0x7e0 [zfs]
Oct 18 01:25:03 zfstest kernel:  dbuf_rele+0x55/0x80 [zfs]
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: read to 0xffffa0b94bdc0310 of 8 bytes by task 222811 on cpu 16:
Oct 18 01:25:03 zfstest kernel:  zrl_add_impl+0x100/0x1a0 [zfs]
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: value changed: 0xffffa0b9e56acd40 -> 0x0000000000000000
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:03 zfstest kernel: CPU: 16 PID: 222811 Comm: cc1 Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:03 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:03 zfstest kernel: ==================================================================
Oct 18 01:25:03 zfstest kernel: ==================================================================
Oct 18 01:25:03 zfstest kernel: BUG: KCSAN: data-race in __mod_lruvec_page_state / filemap_fault
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: read-write (marked) to 0xffffe5d644f5f980 of 8 bytes by task 222811 on cpu 16:
Oct 18 01:25:03 zfstest kernel:  filemap_fault+0x12c/0xdc0
Oct 18 01:25:03 zfstest kernel:  __do_fault+0x76/0x1e0
Oct 18 01:25:03 zfstest kernel:  __handle_mm_fault+0x1983/0x21f0
Oct 18 01:25:03 zfstest kernel:  handle_mm_fault+0x104/0x3a0
Oct 18 01:25:03 zfstest kernel:  do_user_addr_fault+0x1cc/0x6a0
Oct 18 01:25:03 zfstest kernel:  exc_page_fault+0x71/0x170
Oct 18 01:25:03 zfstest kernel:  asm_exc_page_fault+0x21/0x30
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: read to 0xffffe5d644f5f980 of 8 bytes by task 222788 on cpu 21:
Oct 18 01:25:03 zfstest kernel:  __mod_lruvec_page_state+0x42/0x220
Oct 18 01:25:03 zfstest kernel:  __filemap_add_folio+0x5be/0x930
Oct 18 01:25:03 zfstest kernel:  filemap_add_folio+0x65/0x100
Oct 18 01:25:03 zfstest kernel:  __filemap_get_folio+0x326/0x530
Oct 18 01:25:03 zfstest kernel:  filemap_fault+0x1d0/0xdc0
Oct 18 01:25:03 zfstest kernel:  __do_fault+0x76/0x1e0
Oct 18 01:25:03 zfstest kernel:  __handle_mm_fault+0x1983/0x21f0
Oct 18 01:25:03 zfstest kernel:  handle_mm_fault+0x104/0x3a0
Oct 18 01:25:03 zfstest kernel:  do_user_addr_fault+0x1cc/0x6a0
Oct 18 01:25:03 zfstest kernel:  exc_page_fault+0x71/0x170
Oct 18 01:25:03 zfstest kernel:  asm_exc_page_fault+0x21/0x30
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: value changed: 0x0017ffffc0000001 -> 0x0017ffffc0000081
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:03 zfstest kernel: CPU: 21 PID: 222788 Comm: cc1 Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:03 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:03 zfstest kernel: ==================================================================
Oct 18 01:25:03 zfstest kernel: ==================================================================
Oct 18 01:25:03 zfstest kernel: BUG: KCSAN: data-race in arc_tempreserve_clear [zfs] / arc_tempreserve_clear [zfs]
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: read-write (marked) to 0xffffffffc0c4b910 of 8 bytes by task 222807 on cpu 10:
Oct 18 01:25:03 zfstest kernel:  arc_tempreserve_clear+0x22/0x70 [zfs]
Oct 18 01:25:03 zfstest kernel:  dsl_dir_tempreserve_clear+0x1ae/0x240 [zfs]
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: read to 0xffffffffc0c4b910 of 8 bytes by task 222792 on cpu 22:
Oct 18 01:25:03 zfstest kernel:  arc_tempreserve_clear+0x36/0x70 [zfs]
Oct 18 01:25:03 zfstest kernel:  dsl_dir_tempreserve_clear+0x1ae/0x240 [zfs]
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: value changed: 0x0000000000016800 -> 0x000000000000fc00
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:03 zfstest kernel: CPU: 22 PID: 222792 Comm: cc1 Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:03 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:03 zfstest kernel: ==================================================================
Oct 18 01:25:03 zfstest kernel: ==================================================================
Oct 18 01:25:03 zfstest kernel: BUG: KCSAN: data-race in folio_mark_accessed / workingset_activation
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: write (marked) to 0xffffe5d645a15180 of 8 bytes by task 222810 on cpu 7:
Oct 18 01:25:03 zfstest kernel:  folio_mark_accessed+0x3e/0x340
Oct 18 01:25:03 zfstest kernel:  mark_page_accessed+0x24/0x80
Oct 18 01:25:03 zfstest kernel:  unmap_page_range+0x85c/0x17a0
Oct 18 01:25:03 zfstest kernel:  unmap_single_vma+0xbf/0x160
Oct 18 01:25:03 zfstest kernel:  unmap_vmas+0xe7/0x1a0
Oct 18 01:25:03 zfstest kernel:  exit_mmap+0xde/0x230
Oct 18 01:25:03 zfstest kernel:  mmput+0xb5/0x1e0
Oct 18 01:25:03 zfstest kernel:  do_exit+0x549/0x1260
Oct 18 01:25:03 zfstest kernel:  do_group_exit+0x53/0x100
Oct 18 01:25:03 zfstest kernel:  __x64_sys_exit_group+0x21/0x30
Oct 18 01:25:03 zfstest kernel:  do_syscall_64+0x2f/0x50
Oct 18 01:25:03 zfstest kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: read to 0xffffe5d645a15180 of 8 bytes by task 222801 on cpu 6:
Oct 18 01:25:03 zfstest kernel:  workingset_activation+0xf5/0x1b0
Oct 18 01:25:03 zfstest kernel:  folio_mark_accessed+0x1ff/0x340
Oct 18 01:25:03 zfstest kernel:  mark_page_accessed+0x24/0x80
Oct 18 01:25:03 zfstest kernel:  unmap_page_range+0x85c/0x17a0
Oct 18 01:25:03 zfstest kernel:  unmap_single_vma+0xbf/0x160
Oct 18 01:25:03 zfstest kernel:  unmap_vmas+0xe7/0x1a0
Oct 18 01:25:03 zfstest kernel:  exit_mmap+0xde/0x230
Oct 18 01:25:03 zfstest kernel:  mmput+0xb5/0x1e0
Oct 18 01:25:03 zfstest kernel:  do_exit+0x549/0x1260
Oct 18 01:25:03 zfstest kernel:  do_group_exit+0x53/0x100
Oct 18 01:25:03 zfstest kernel:  __x64_sys_exit_group+0x21/0x30
Oct 18 01:25:03 zfstest kernel:  do_syscall_64+0x2f/0x50
Oct 18 01:25:03 zfstest kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: value changed: 0x0017ffffc0000014 -> 0x0017ffffc0000016
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:03 zfstest kernel: CPU: 6 PID: 222801 Comm: cc1 Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:03 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:03 zfstest kernel: ==================================================================
Oct 18 01:25:03 zfstest kernel: ==================================================================
Oct 18 01:25:03 zfstest kernel: BUG: KCSAN: data-race in osq_lock / osq_lock
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: write (marked) to 0xffffa0bcafc31c08 of 8 bytes by task 222851 on cpu 6:
Oct 18 01:25:03 zfstest kernel:  osq_lock+0x294/0x2e0
Oct 18 01:25:03 zfstest kernel:  rwsem_down_write_slowpath+0x10b/0xaf0
Oct 18 01:25:03 zfstest kernel:  down_write+0xc3/0xd0
Oct 18 01:25:03 zfstest kernel:  path_openat+0x563/0x1ac0
Oct 18 01:25:03 zfstest kernel:  do_filp_open+0x14d/0x240
Oct 18 01:25:03 zfstest kernel:  do_sys_openat2+0xa8/0x200
Oct 18 01:25:03 zfstest kernel:  __x64_sys_openat+0xab/0x100
Oct 18 01:25:03 zfstest kernel:  do_syscall_64+0x2f/0x50
Oct 18 01:25:03 zfstest kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: read to 0xffffa0bcafc31c08 of 8 bytes by task 222862 on cpu 16:
Oct 18 01:25:03 zfstest kernel:  osq_lock+0x10f/0x2e0
Oct 18 01:25:03 zfstest kernel:  rwsem_down_write_slowpath+0x10b/0xaf0
Oct 18 01:25:03 zfstest kernel:  down_write+0xc3/0xd0
Oct 18 01:25:03 zfstest kernel:  path_openat+0x563/0x1ac0
Oct 18 01:25:03 zfstest kernel:  do_filp_open+0x14d/0x240
Oct 18 01:25:03 zfstest kernel:  do_sys_openat2+0xa8/0x200
Oct 18 01:25:03 zfstest kernel:  __x64_sys_openat+0xab/0x100
Oct 18 01:25:03 zfstest kernel:  do_syscall_64+0x2f/0x50
Oct 18 01:25:03 zfstest kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: value changed: 0xffffa0bcaf9b1c00 -> 0xffffa0bcafab1c00
Oct 18 01:25:03 zfstest kernel:
Oct 18 01:25:03 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:03 zfstest kernel: CPU: 16 PID: 222862 Comm: as Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:03 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:03 zfstest kernel: ==================================================================
Oct 18 01:25:04 zfstest kernel: ==================================================================
Oct 18 01:25:04 zfstest kernel: BUG: KCSAN: data-race in spa_config_enter [zfs] / spa_config_held [zfs]
Oct 18 01:25:04 zfstest kernel:
Oct 18 01:25:04 zfstest kernel: write to 0xffffa0b9670762bc of 4 bytes by task 223325 on cpu 10:
Oct 18 01:25:04 zfstest kernel:  spa_config_enter+0x16a/0x290 [zfs]
Oct 18 01:25:04 zfstest kernel:  metaslab_alloc+0x8d/0x700 [zfs]
Oct 18 01:25:04 zfstest kernel:  zio_dva_allocate+0x27f/0x800 [zfs]
Oct 18 01:25:04 zfstest kernel:  zio_execute+0x18c/0x2a0 [zfs]
Oct 18 01:25:04 zfstest kernel:
Oct 18 01:25:04 zfstest kernel: read to 0xffffa0b9670762bc of 4 bytes by task 588 on cpu 5:
Oct 18 01:25:04 zfstest kernel:  spa_config_held+0x5e/0xb0 [zfs]
Oct 18 01:25:04 zfstest kernel:  vdev_lookup_top+0x34/0xd0 [zfs]
Oct 18 01:25:04 zfstest kernel:  metaslab_check_free+0x173/0x330 [zfs]
Oct 18 01:25:04 zfstest kernel:  zio_free_sync+0x7c/0x1b0 [zfs]
Oct 18 01:25:04 zfstest kernel:  zio_free+0xd7/0x110 [zfs]
Oct 18 01:25:04 zfstest kernel:  dsl_free+0x25/0x30 [zfs]
Oct 18 01:25:04 zfstest kernel:  dsl_dataset_block_kill+0x301/0xc30 [zfs]
Oct 18 01:25:04 zfstest kernel:  free_blocks+0xd5/0x3e0 [zfs]
Oct 18 01:25:04 zfstest kernel:  free_children+0x579/0x690 [zfs]
Oct 18 01:25:04 zfstest kernel:  dnode_sync_free_range_impl+0x15f/0x3f0 [zfs]
Oct 18 01:25:04 zfstest kernel:  dnode_sync_free_range+0x78/0xe0 [zfs]
Oct 18 01:25:04 zfstest kernel:  range_tree_walk+0x70/0xb0 [zfs]
Oct 18 01:25:04 zfstest kernel:  dnode_sync+0x7cc/0x1370 [zfs]
Oct 18 01:25:04 zfstest kernel:  dmu_objset_sync_dnodes+0xe5/0x170 [zfs]
Oct 18 01:25:04 zfstest kernel:
Oct 18 01:25:04 zfstest kernel: value changed: 0x00000000 -> 0x00000002
Oct 18 01:25:04 zfstest kernel:
Oct 18 01:25:04 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:04 zfstest kernel: CPU: 5 PID: 588 Comm: dp_sync_taskq Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:04 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:04 zfstest kernel: ==================================================================
Oct 18 01:25:04 zfstest kernel: ==================================================================
Oct 18 01:25:04 zfstest kernel: BUG: KCSAN: data-race in spa_config_exit [zfs] / spa_config_held [zfs]
Oct 18 01:25:04 zfstest kernel:
Oct 18 01:25:04 zfstest kernel: write to 0xffffa0b96707643c of 4 bytes by task 563 on cpu 22:
Oct 18 01:25:04 zfstest kernel:  spa_config_exit+0xa4/0x1a0 [zfs]
Oct 18 01:25:04 zfstest kernel:  metaslab_free+0x1be/0x330 [zfs]
Oct 18 01:25:04 zfstest kernel:  zio_free_sync+0x133/0x1b0 [zfs]
Oct 18 01:25:04 zfstest kernel:  zio_free+0xd7/0x110 [zfs]
Oct 18 01:25:04 zfstest kernel:  dsl_free+0x25/0x30 [zfs]
Oct 18 01:25:04 zfstest kernel:  dsl_dataset_block_kill+0x301/0xc30 [zfs]
Oct 18 01:25:04 zfstest kernel:  dbuf_write_done+0x545/0x9c0 [zfs]
Oct 18 01:25:04 zfstest kernel:
Oct 18 01:25:04 zfstest kernel: read to 0xffffa0b96707643c of 4 bytes by task 223404 on cpu 5:
Oct 18 01:25:04 zfstest kernel:  spa_config_held+0x5e/0xb0 [zfs]
Oct 18 01:25:04 zfstest kernel:  vdev_lookup_top+0x34/0xd0 [zfs]
Oct 18 01:25:04 zfstest kernel:  metaslab_group_alloc_verify+0x125/0x1e0 [zfs]
Oct 18 01:25:04 zfstest kernel:  zio_done+0xa2c/0x2290 [zfs]
Oct 18 01:25:04 zfstest kernel:  zio_execute+0x18c/0x2a0 [zfs]
Oct 18 01:25:04 zfstest kernel:
Oct 18 01:25:04 zfstest kernel: value changed: 0x00000002 -> 0x00000000
Oct 18 01:25:04 zfstest kernel:
Oct 18 01:25:04 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:04 zfstest kernel: CPU: 5 PID: 223404 Comm: z_wr_int_3 Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:04 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:04 zfstest kernel: ==================================================================
Oct 18 01:25:06 zfstest kernel: ==================================================================
Oct 18 01:25:06 zfstest kernel: BUG: KCSAN: data-race in _find_next_bit+0x47/0x110
Oct 18 01:25:06 zfstest kernel:
Oct 18 01:25:06 zfstest kernel: race at unknown origin, with read to 0xffffa0b940066770 of 8 bytes by interrupt on cpu 2:
Oct 18 01:25:06 zfstest kernel:  _find_next_bit+0x47/0x110
Oct 18 01:25:06 zfstest kernel:  cpumask_next_wrap+0x3e/0xa0
Oct 18 01:25:06 zfstest kernel:  _nohz_idle_balance.constprop.0.isra.0+0xe0/0x340
Oct 18 01:25:06 zfstest kernel:  __do_softirq+0xfc/0x363
Oct 18 01:25:06 zfstest kernel:  __irq_exit_rcu+0xb5/0x100
Oct 18 01:25:06 zfstest kernel:  sysvec_call_function_single+0xa2/0xd0
Oct 18 01:25:06 zfstest kernel:  asm_sysvec_call_function_single+0x15/0x20
Oct 18 01:25:06 zfstest kernel:  native_safe_halt+0xb/0x10
Oct 18 01:25:06 zfstest kernel:  default_idle+0xa/0x10
Oct 18 01:25:06 zfstest kernel:  default_idle_call+0x32/0xe0
Oct 18 01:25:06 zfstest kernel:  do_idle+0x1fe/0x270
Oct 18 01:25:06 zfstest kernel:  cpu_startup_entry+0x19/0x20
Oct 18 01:25:06 zfstest kernel:  start_secondary+0x128/0x160
Oct 18 01:25:06 zfstest kernel:  secondary_startup_64_no_verify+0xc3/0xcb
Oct 18 01:25:06 zfstest kernel:
Oct 18 01:25:06 zfstest kernel: value changed: 0x0000000000fdff5e -> 0x0000000000fdff5c
Oct 18 01:25:06 zfstest kernel:
Oct 18 01:25:06 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:06 zfstest kernel: CPU: 2 PID: 0 Comm: swapper/2 Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:06 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:06 zfstest kernel: ==================================================================
Oct 18 01:25:06 zfstest kernel: ==================================================================
Oct 18 01:25:06 zfstest kernel: BUG: KCSAN: data-race in zrl_add_impl [zfs] / zrl_remove [zfs]
Oct 18 01:25:06 zfstest kernel:
Oct 18 01:25:06 zfstest kernel: write to 0xffffa0b963b4f0b0 of 8 bytes by task 702 on cpu 6:
Oct 18 01:25:06 zfstest kernel:  zrl_add_impl+0x11b/0x1a0 [zfs]
Oct 18 01:25:06 zfstest kernel:
Oct 18 01:25:06 zfstest kernel: read to 0xffffa0b963b4f0b0 of 8 bytes by task 223796 on cpu 15:
Oct 18 01:25:06 zfstest kernel:  zrl_remove+0x19/0xb0 [zfs]
Oct 18 01:25:06 zfstest kernel:  dbuf_verify+0x414/0xbf0 [zfs]
Oct 18 01:25:06 zfstest kernel:  dbuf_hold_impl+0x23b/0xdf0 [zfs]
Oct 18 01:25:06 zfstest kernel:  dbuf_hold_level+0x2b/0x60 [zfs]
Oct 18 01:25:06 zfstest kernel:  dbuf_hold+0x12/0x20 [zfs]
Oct 18 01:25:06 zfstest kernel:  dnode_hold_impl+0x54b/0x1130 [zfs]
Oct 18 01:25:06 zfstest kernel:  dnode_hold+0x17/0x20 [zfs]
Oct 18 01:25:06 zfstest kernel:  dmu_bonus_hold+0x33/0xa0 [zfs]
Oct 18 01:25:06 zfstest kernel:  dsl_deadlist_open+0x9a/0x200 [zfs]
Oct 18 01:25:06 zfstest kernel:  dsl_dataset_hold_obj+0x5a3/0xd20 [zfs]
Oct 18 01:25:06 zfstest kernel:  dsl_dataset_hold_obj+0xb55/0xd20 [zfs]
Oct 18 01:25:06 zfstest kernel:  dsl_dataset_hold_obj_flags+0x1f/0x90 [zfs]
Oct 18 01:25:06 zfstest kernel:  dsl_dataset_hold_flags+0xd5/0x3c0 [zfs]
Oct 18 01:25:06 zfstest kernel:  dmu_objset_hold_flags+0x80/0xf0 [zfs]
Oct 18 01:25:06 zfstest kernel:  dmu_objset_hold+0x12/0x20 [zfs]
Oct 18 01:25:06 zfstest kernel:  zfs_ioc_objset_stats+0x2e/0x90 [zfs]
Oct 18 01:25:06 zfstest kernel:  zfsdev_ioctl_common+0x831/0x980 [zfs]
Oct 18 01:25:06 zfstest kernel:  zfsdev_ioctl+0x53/0xe0 [zfs]
Oct 18 01:25:06 zfstest kernel:  __x64_sys_ioctl+0xb9/0xf0
Oct 18 01:25:06 zfstest kernel:  do_syscall_64+0x2f/0x50
Oct 18 01:25:06 zfstest kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 18 01:25:06 zfstest kernel:
Oct 18 01:25:06 zfstest kernel: value changed: 0xffffa0b96d7199c0 -> 0x0000000000000000
Oct 18 01:25:06 zfstest kernel:
Oct 18 01:25:06 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:06 zfstest kernel: CPU: 15 PID: 223796 Comm: zfs Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:06 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:06 zfstest kernel: ==================================================================
Oct 18 01:25:06 zfstest kernel: ==================================================================
Oct 18 01:25:06 zfstest kernel: BUG: KCSAN: data-race in dnode_sync [zfs] / dnode_sync [zfs]
Oct 18 01:25:06 zfstest kernel:
Oct 18 01:25:06 zfstest kernel: write to 0xffffa0b9e5e84c18 of 8 bytes by task 598 on cpu 4:
Oct 18 01:25:06 zfstest kernel:  dnode_sync+0xeef/0x1370 [zfs]
Oct 18 01:25:06 zfstest kernel:  dmu_objset_sync_dnodes+0xe5/0x170 [zfs]
Oct 18 01:25:06 zfstest kernel:
Oct 18 01:25:06 zfstest kernel: read to 0xffffa0b9e5e84c18 of 8 bytes by task 596 on cpu 14:
Oct 18 01:25:06 zfstest kernel:  dnode_sync+0xedc/0x1370 [zfs]
Oct 18 01:25:06 zfstest kernel:  dmu_objset_sync_dnodes+0xe5/0x170 [zfs]
Oct 18 01:25:06 zfstest kernel:
Oct 18 01:25:06 zfstest kernel: value changed: 0x000000000000004c -> 0x000000000000004d
Oct 18 01:25:06 zfstest kernel:
Oct 18 01:25:06 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:06 zfstest kernel: CPU: 14 PID: 596 Comm: dp_sync_taskq Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:06 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:06 zfstest kernel: ==================================================================
Oct 18 01:25:07 zfstest kernel: ==================================================================
Oct 18 01:25:07 zfstest kernel: BUG: KCSAN: data-race in __sbitmap_get_word / _find_next_bit
Oct 18 01:25:07 zfstest kernel:
Oct 18 01:25:07 zfstest kernel: read-write (marked) to 0xffffa0b9648f4180 of 8 bytes by task 224079 on cpu 21:
Oct 18 01:25:07 zfstest kernel:  __sbitmap_get_word+0x6b/0xb0
Oct 18 01:25:07 zfstest kernel:  sbitmap_get_shallow+0x12f/0x300
Oct 18 01:25:07 zfstest kernel:  sbitmap_queue_get_shallow+0x1f/0x40
Oct 18 01:25:07 zfstest kernel:  __blk_mq_get_tag+0x5a/0x230
Oct 18 01:25:07 zfstest kernel:  blk_mq_get_tag+0x4bf/0x5b0
Oct 18 01:25:07 zfstest kernel:  __blk_mq_alloc_requests+0x341/0x5c0
Oct 18 01:25:07 zfstest kernel:  blk_mq_submit_bio+0x6b4/0xd50
Oct 18 01:25:07 zfstest kernel:  __submit_bio+0x1ce/0x2f0
Oct 18 01:25:07 zfstest kernel:  submit_bio_noacct_nocheck+0x4d9/0x540
Oct 18 01:25:07 zfstest kernel:  submit_bio_noacct+0x339/0xa20
Oct 18 01:25:07 zfstest kernel:  submit_bio+0x7b/0xf0
Oct 18 01:25:07 zfstest kernel:  vdev_submit_bio_impl+0x5/0x10 [zfs]
Oct 18 01:25:07 zfstest kernel:  vdev_submit_bio+0x43/0x60 [zfs]
Oct 18 01:25:07 zfstest kernel:  __vdev_disk_physio.constprop.0+0x347/0x410 [zfs]
Oct 18 01:25:07 zfstest kernel:  vdev_disk_io_start+0xec/0x4f0 [zfs]
Oct 18 01:25:07 zfstest kernel:  zio_vdev_io_start+0x235/0x6d0 [zfs]
Oct 18 01:25:07 zfstest kernel:  zio_execute+0x18c/0x2a0 [zfs]
Oct 18 01:25:07 zfstest kernel:
Oct 18 01:25:07 zfstest kernel: read to 0xffffa0b9648f4180 of 8 bytes by task 565 on cpu 13:
Oct 18 01:25:07 zfstest kernel:  _find_next_bit+0x47/0x110
Oct 18 01:25:07 zfstest kernel:  __sbitmap_get_word+0x3b/0xb0
Oct 18 01:25:07 zfstest kernel:  sbitmap_get_shallow+0x12f/0x300
Oct 18 01:25:07 zfstest kernel:  sbitmap_queue_get_shallow+0x1f/0x40
Oct 18 01:25:07 zfstest kernel:  __blk_mq_get_tag+0x5a/0x230
Oct 18 01:25:07 zfstest kernel:  blk_mq_get_tag+0x4bf/0x5b0
Oct 18 01:25:07 zfstest kernel:  __blk_mq_alloc_requests+0x341/0x5c0
Oct 18 01:25:07 zfstest kernel:  blk_mq_submit_bio+0x6b4/0xd50
Oct 18 01:25:07 zfstest kernel:  __submit_bio+0x1ce/0x2f0
Oct 18 01:25:07 zfstest kernel:  submit_bio_noacct_nocheck+0x4d9/0x540
Oct 18 01:25:07 zfstest kernel:  submit_bio_noacct+0x339/0xa20
Oct 18 01:25:07 zfstest kernel:  submit_bio+0x7b/0xf0
Oct 18 01:25:07 zfstest kernel:  vdev_submit_bio_impl+0x5/0x10 [zfs]
Oct 18 01:25:07 zfstest kernel:  vdev_submit_bio+0x43/0x60 [zfs]
Oct 18 01:25:07 zfstest kernel:  __vdev_disk_physio.constprop.0+0x347/0x410 [zfs]
Oct 18 01:25:07 zfstest kernel:  vdev_disk_io_start+0xec/0x4f0 [zfs]
Oct 18 01:25:07 zfstest kernel:  zio_vdev_io_start+0x235/0x6d0 [zfs]
Oct 18 01:25:07 zfstest kernel:  zio_nowait+0x1b0/0x350 [zfs]
Oct 18 01:25:07 zfstest kernel:
Oct 18 01:25:07 zfstest kernel: value changed: 0x0000000000200000 -> 0x1000000000200000
Oct 18 01:25:07 zfstest kernel:
Oct 18 01:25:07 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:07 zfstest kernel: CPU: 13 PID: 565 Comm: z_wr_int_3 Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:07 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:07 zfstest kernel: ==================================================================
Oct 18 01:25:07 zfstest kernel: ==================================================================
Oct 18 01:25:07 zfstest kernel: BUG: KCSAN: data-race in spa_config_exit [zfs] / spa_config_held [zfs]
Oct 18 01:25:07 zfstest kernel:
Oct 18 01:25:07 zfstest kernel: write to 0xffffa0b9670762bc of 4 bytes by task 224135 on cpu 23:
Oct 18 01:25:07 zfstest kernel:  spa_config_exit+0xa4/0x1a0 [zfs]
Oct 18 01:25:07 zfstest kernel:  metaslab_alloc+0x246/0x700 [zfs]
Oct 18 01:25:07 zfstest kernel:  zio_dva_allocate+0x27f/0x800 [zfs]
Oct 18 01:25:07 zfstest kernel:  zio_execute+0x18c/0x2a0 [zfs]
Oct 18 01:25:07 zfstest kernel:
Oct 18 01:25:07 zfstest kernel: read to 0xffffa0b9670762bc of 4 bytes by task 564 on cpu 12:
Oct 18 01:25:07 zfstest kernel:  spa_config_held+0x5e/0xb0 [zfs]
Oct 18 01:25:07 zfstest kernel:  dva_get_dsize_sync+0x3d/0xd0 [zfs]
Oct 18 01:25:07 zfstest kernel:  bp_get_dsize_sync+0xaa/0x1f0 [zfs]
Oct 18 01:25:07 zfstest kernel:  dsl_dataset_block_kill+0x4a/0xc30 [zfs]
Oct 18 01:25:07 zfstest kernel:  dbuf_write_done+0x545/0x9c0 [zfs]
Oct 18 01:25:07 zfstest kernel:
Oct 18 01:25:07 zfstest kernel: value changed: 0x00000001 -> 0x00000000
Oct 18 01:25:07 zfstest kernel:
Oct 18 01:25:07 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:07 zfstest kernel: CPU: 12 PID: 564 Comm: z_wr_int_2 Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:07 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:07 zfstest kernel: ==================================================================
Oct 18 01:25:07 zfstest systemd-networkd[1062]: docker0: Lost carrier
Oct 18 01:25:08 zfstest kernel: ==================================================================
Oct 18 01:25:08 zfstest kernel: BUG: KCSAN: data-race in spa_config_enter [zfs] / spa_config_held [zfs]
Oct 18 01:25:08 zfstest kernel:
Oct 18 01:25:08 zfstest kernel: write to 0xffffa0b96707637c of 4 bytes by task 224272 on cpu 16:
Oct 18 01:25:08 zfstest kernel:  spa_config_enter+0x16a/0x290 [zfs]
Oct 18 01:25:08 zfstest kernel:  zio_vdev_io_start+0x666/0x6d0 [zfs]
Oct 18 01:25:08 zfstest kernel:  zio_execute+0x18c/0x2a0 [zfs]
Oct 18 01:25:08 zfstest kernel:
Oct 18 01:25:08 zfstest kernel: read to 0xffffa0b96707637c of 4 bytes by task 224271 on cpu 8:
Oct 18 01:25:08 zfstest kernel:  spa_config_held+0x5e/0xb0 [zfs]
Oct 18 01:25:08 zfstest kernel:  zio_create+0x7ac/0x890 [zfs]
Oct 18 01:25:08 zfstest kernel:  zio_vdev_child_io+0x215/0x420 [zfs]
Oct 18 01:25:08 zfstest kernel:  vdev_mirror_io_start+0x178/0x540 [zfs]
Oct 18 01:25:08 zfstest kernel:  zio_vdev_io_start+0x539/0x6d0 [zfs]
Oct 18 01:25:08 zfstest kernel:  zio_execute+0x18c/0x2a0 [zfs]
Oct 18 01:25:08 zfstest kernel:
Oct 18 01:25:08 zfstest kernel: value changed: 0x00000004 -> 0x00000005
Oct 18 01:25:08 zfstest kernel:
Oct 18 01:25:08 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:08 zfstest kernel: CPU: 8 PID: 224271 Comm: z_wr_iss Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:08 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:08 zfstest kernel: ==================================================================
Oct 18 01:25:08 zfstest kernel: ==================================================================
Oct 18 01:25:08 zfstest kernel: BUG: KCSAN: data-race in zrl_add_impl [zfs] / zrl_add_impl [zfs]
Oct 18 01:25:08 zfstest kernel:
Oct 18 01:25:08 zfstest kernel: write to 0xffffa0b96d2941d0 of 8 bytes by task 224316 on cpu 20:
Oct 18 01:25:08 zfstest kernel:  zrl_add_impl+0x11b/0x1a0 [zfs]
Oct 18 01:25:08 zfstest kernel:
Oct 18 01:25:08 zfstest kernel: read to 0xffffa0b96d2941d0 of 8 bytes by task 224315 on cpu 7:
Oct 18 01:25:08 zfstest kernel:  zrl_add_impl+0x100/0x1a0 [zfs]
Oct 18 01:25:08 zfstest kernel:
Oct 18 01:25:08 zfstest kernel: value changed: 0x0000000000000000 -> 0xffffa0b9de778000
Oct 18 01:25:08 zfstest kernel:
Oct 18 01:25:08 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:08 zfstest kernel: CPU: 7 PID: 224315 Comm: zfs Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:08 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:08 zfstest kernel: ==================================================================
Oct 18 01:25:10 zfstest kernel: ==================================================================
Oct 18 01:25:10 zfstest kernel: BUG: KCSAN: data-race in osq_lock / osq_lock
Oct 18 01:25:10 zfstest kernel:
Oct 18 01:25:10 zfstest kernel: write (marked) to 0xffffa0bcafcb1c08 of 8 bytes by task 593 on cpu 1:
Oct 18 01:25:10 zfstest kernel:  osq_lock+0x294/0x2e0
Oct 18 01:25:10 zfstest kernel:  __mutex_lock.constprop.0+0x1ab/0xa40
Oct 18 01:25:10 zfstest kernel:  __mutex_lock_slowpath+0xf/0x10
Oct 18 01:25:10 zfstest kernel:  mutex_lock+0x93/0xa0
Oct 18 01:25:10 zfstest kernel:  do_userquota_cacheflush+0x1c5/0x3f0 [zfs]
Oct 18 01:25:10 zfstest kernel:  userquota_updates_task+0x42e/0x690 [zfs]
Oct 18 01:25:10 zfstest kernel:  taskq_thread+0x389/0x650 [spl]
Oct 18 01:25:10 zfstest kernel:  kthread+0x172/0x1a0
Oct 18 01:25:10 zfstest kernel:  ret_from_fork+0x22/0x30
Oct 18 01:25:10 zfstest kernel:
Oct 18 01:25:10 zfstest kernel: read to 0xffffa0bcafcb1c08 of 8 bytes by task 591 on cpu 18:
Oct 18 01:25:10 zfstest kernel:  osq_lock+0x10f/0x2e0
Oct 18 01:25:10 zfstest kernel:  __mutex_lock.constprop.0+0x1ab/0xa40
Oct 18 01:25:10 zfstest kernel:  __mutex_lock_slowpath+0xf/0x10
Oct 18 01:25:10 zfstest kernel:  mutex_lock+0x93/0xa0
Oct 18 01:25:10 zfstest kernel:  do_userquota_cacheflush+0x310/0x3f0 [zfs]
Oct 18 01:25:10 zfstest kernel:  userquota_updates_task+0x42e/0x690 [zfs]
Oct 18 01:25:10 zfstest kernel:  taskq_thread+0x389/0x650 [spl]
Oct 18 01:25:10 zfstest kernel:  kthread+0x172/0x1a0
Oct 18 01:25:10 zfstest kernel:  ret_from_fork+0x22/0x30
Oct 18 01:25:10 zfstest kernel:
Oct 18 01:25:10 zfstest kernel: value changed: 0xffffa0bcaf871c00 -> 0xffffa0bcaf971c00
Oct 18 01:25:10 zfstest kernel:
Oct 18 01:25:10 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:10 zfstest kernel: CPU: 18 PID: 591 Comm: dp_sync_taskq Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:10 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:10 zfstest kernel: ==================================================================
Oct 18 01:25:11 zfstest kernel: ==================================================================
Oct 18 01:25:11 zfstest kernel: BUG: KCSAN: data-race in spa_config_exit [zfs] / spa_config_held [zfs]
Oct 18 01:25:11 zfstest kernel:
Oct 18 01:25:11 zfstest kernel: write to 0xffffa0b96707637c of 4 bytes by task 562 on cpu 6:
Oct 18 01:25:11 zfstest kernel:  spa_config_exit+0xa4/0x1a0 [zfs]
Oct 18 01:25:11 zfstest kernel:  zio_vdev_io_assess+0x371/0x4d0 [zfs]
Oct 18 01:25:11 zfstest kernel:  zio_execute+0x18c/0x2a0 [zfs]
Oct 18 01:25:11 zfstest kernel:
Oct 18 01:25:11 zfstest kernel: read to 0xffffa0b96707637c of 4 bytes by task 225144 on cpu 10:
Oct 18 01:25:11 zfstest kernel:  spa_config_held+0x5e/0xb0 [zfs]
Oct 18 01:25:11 zfstest kernel:  vdev_lookup_top+0x34/0xd0 [zfs]
Oct 18 01:25:11 zfstest kernel:  metaslab_group_alloc_decrement+0x24/0xa0 [zfs]
Oct 18 01:25:11 zfstest kernel:  zio_dva_throttle_done+0x257/0x560 [zfs]
Oct 18 01:25:11 zfstest kernel:
Oct 18 01:25:11 zfstest kernel: value changed: 0x00000004 -> 0x00000003
Oct 18 01:25:11 zfstest kernel:
Oct 18 01:25:11 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:11 zfstest kernel: CPU: 10 PID: 225144 Comm: z_wr_int_2 Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:11 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:11 zfstest kernel: ==================================================================
Oct 18 01:25:11 zfstest kernel: ==================================================================
Oct 18 01:25:11 zfstest kernel: BUG: KCSAN: data-race in zrl_add_impl [zfs] / zrl_remove [zfs]
Oct 18 01:25:11 zfstest kernel:
Oct 18 01:25:11 zfstest kernel: write to 0xffffa0b9596841d0 of 8 bytes by task 225101 on cpu 15:
Oct 18 01:25:11 zfstest kernel:  zrl_add_impl+0x11b/0x1a0 [zfs]
Oct 18 01:25:11 zfstest kernel:
Oct 18 01:25:11 zfstest kernel: read to 0xffffa0b9596841d0 of 8 bytes by task 225104 on cpu 12:
Oct 18 01:25:11 zfstest kernel:  zrl_remove+0x19/0xb0 [zfs]
Oct 18 01:25:11 zfstest kernel:  dmu_read_uio_dbuf+0x78/0x90 [zfs]
Oct 18 01:25:11 zfstest kernel:  zfs_read+0x1ff/0x440 [zfs]
Oct 18 01:25:11 zfstest kernel:  zpl_iter_read+0xf3/0x180 [zfs]
Oct 18 01:25:11 zfstest kernel:  new_sync_read+0x1d9/0x2a0
Oct 18 01:25:11 zfstest kernel:  vfs_read+0x22a/0x290
Oct 18 01:25:11 zfstest kernel:  ksys_read+0x7f/0x110
Oct 18 01:25:11 zfstest kernel:  __x64_sys_read+0x3d/0x50
Oct 18 01:25:11 zfstest kernel:  do_syscall_64+0x2f/0x50
Oct 18 01:25:11 zfstest kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 18 01:25:11 zfstest kernel:
Oct 18 01:25:11 zfstest kernel: value changed: 0xffffa0b9e72dcd40 -> 0x0000000000000000
Oct 18 01:25:11 zfstest kernel:
Oct 18 01:25:11 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:11 zfstest kernel: CPU: 12 PID: 225104 Comm: cc1plus Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:11 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:11 zfstest kernel: ==================================================================
Oct 18 01:25:12 zfstest kernel: ==================================================================
Oct 18 01:25:12 zfstest kernel: BUG: KCSAN: data-race in zrl_add_impl [zfs] / zrl_add_impl [zfs]
Oct 18 01:25:12 zfstest kernel:
Oct 18 01:25:12 zfstest kernel: write to 0xffffa0b9e79381d0 of 8 bytes by task 225089 on cpu 6:
Oct 18 01:25:12 zfstest kernel:  zrl_add_impl+0x11b/0x1a0 [zfs]
Oct 18 01:25:12 zfstest kernel:
Oct 18 01:25:12 zfstest kernel: read to 0xffffa0b9e79381d0 of 8 bytes by task 225087 on cpu 3:
Oct 18 01:25:12 zfstest kernel:  zrl_add_impl+0x100/0x1a0 [zfs]
Oct 18 01:25:12 zfstest kernel:
Oct 18 01:25:12 zfstest kernel: value changed: 0xffffa0b9de77cd40 -> 0x0000000000000000
Oct 18 01:25:12 zfstest kernel:
Oct 18 01:25:12 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:12 zfstest kernel: CPU: 3 PID: 225087 Comm: cc1plus Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:12 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:12 zfstest kernel: ==================================================================
Oct 18 01:25:12 zfstest kernel: ==================================================================
Oct 18 01:25:12 zfstest kernel: BUG: KCSAN: data-race in _find_next_bit+0x47/0x110
Oct 18 01:25:12 zfstest kernel:
Oct 18 01:25:12 zfstest kernel: race at unknown origin, with read to 0xffffa0b940066770 of 8 bytes by interrupt on cpu 0:
Oct 18 01:25:12 zfstest kernel:  _find_next_bit+0x47/0x110
Oct 18 01:25:12 zfstest kernel:  cpumask_next_wrap+0x3e/0xa0
Oct 18 01:25:12 zfstest kernel:  _nohz_idle_balance.constprop.0.isra.0+0xe0/0x340
Oct 18 01:25:12 zfstest kernel:  __do_softirq+0xfc/0x363
Oct 18 01:25:12 zfstest kernel:  __irq_exit_rcu+0xb5/0x100
Oct 18 01:25:12 zfstest kernel:  sysvec_call_function_single+0xa2/0xd0
Oct 18 01:25:12 zfstest kernel:  asm_sysvec_call_function_single+0x15/0x20
Oct 18 01:25:12 zfstest kernel:  native_safe_halt+0xb/0x10
Oct 18 01:25:12 zfstest kernel:  default_idle+0xa/0x10
Oct 18 01:25:12 zfstest kernel:  default_idle_call+0x32/0xe0
Oct 18 01:25:12 zfstest kernel:  do_idle+0x1fe/0x270
Oct 18 01:25:12 zfstest kernel:  cpu_startup_entry+0x19/0x20
Oct 18 01:25:12 zfstest kernel:  rest_init+0x110/0x140
Oct 18 01:25:12 zfstest kernel:  arch_call_rest_init+0xa/0x10
Oct 18 01:25:12 zfstest kernel:  start_kernel+0xc72/0xc99
Oct 18 01:25:12 zfstest kernel:  secondary_startup_64_no_verify+0xc3/0xcb
Oct 18 01:25:12 zfstest kernel:
Oct 18 01:25:12 zfstest kernel: value changed: 0x00000000007fdfdf -> 0x00000000007fcfdf
Oct 18 01:25:12 zfstest kernel:
Oct 18 01:25:12 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:12 zfstest kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:12 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:12 zfstest kernel: ==================================================================
Oct 18 01:25:12 zfstest kernel: VERIFY(PageUptodate(pp)) failed
Oct 18 01:25:12 zfstest kernel: PANIC at zfs_vnops_os.c:298:mappedread()
Oct 18 01:25:14 zfstest kernel: Showing stack for process 225384
Oct 18 01:25:14 zfstest kernel: CPU: 5 PID: 225384 Comm: sed Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:14 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:14 zfstest kernel: Call Trace:
Oct 18 01:25:14 zfstest kernel:  <TASK>
Oct 18 01:25:14 zfstest kernel:  dump_stack_lvl+0x45/0x5a
Oct 18 01:25:14 zfstest kernel:  dump_stack+0xc/0xd
Oct 18 01:25:14 zfstest kernel:  spl_dumpstack+0x32/0x34 [spl]
Oct 18 01:25:14 zfstest kernel:  spl_panic+0xdd/0x10b [spl]
Oct 18 01:25:14 zfstest kernel:  ? xas_start+0x7f/0x1c0
Oct 18 01:25:14 zfstest kernel:  ? __rcu_read_unlock+0x53/0x70
Oct 18 01:25:14 zfstest kernel:  ? __filemap_get_folio+0x22a/0x530
Oct 18 01:25:14 zfstest kernel:  spl_assert+0x17/0x20 [zfs]
Oct 18 01:25:14 zfstest kernel:  mappedread+0x158/0x160 [zfs]
Oct 18 01:25:14 zfstest kernel:  zfs_read+0x285/0x440 [zfs]
Oct 18 01:25:14 zfstest kernel:  zpl_iter_read+0xf3/0x180 [zfs]
Oct 18 01:25:14 zfstest kernel:  ? populate_seccomp_data+0x1a7/0x240
Oct 18 01:25:14 zfstest kernel:  new_sync_read+0x1d9/0x2a0
Oct 18 01:25:14 zfstest kernel:  vfs_read+0x22a/0x290
Oct 18 01:25:14 zfstest kernel:  __x64_sys_pread64+0xc9/0x100
Oct 18 01:25:14 zfstest kernel:  do_syscall_64+0x2f/0x50
Oct 18 01:25:14 zfstest kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 18 01:25:14 zfstest kernel: RIP: 0033:0x7f20d5ddc05e
Oct 18 01:25:14 zfstest kernel: Code: 00 00 00 00 f7 d8 89 05 00 01 01 00 48 c7 c0 ff ff ff ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 49 89 ca b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 0a c3 66 0f 1f 84 00 00 00 00 00 f7 d8 89 05
Oct 18 01:25:14 zfstest kernel: RSP: 002b:00007ffcc12dee68 EFLAGS: 00000202 ORIG_RAX: 0000000000000011
Oct 18 01:25:14 zfstest kernel: RAX: ffffffffffffffda RBX: 00007ffcc12df188 RCX: 00007f20d5ddc05e
Oct 18 01:25:14 zfstest kernel: RDX: 0000000000000044 RSI: 00007ffcc12dee70 RDI: 0000000000000003
Oct 18 01:25:14 zfstest kernel: RBP: 00007ffcc12def10 R08: 00007ffcc12defc8 R09: 0000000000000000
Oct 18 01:25:14 zfstest kernel: R10: 0000000000000338 R11: 0000000000000202 R12: 0000000000000003
Oct 18 01:25:14 zfstest kernel: R13: 00007f20d5db99c0 R14: 00007ffcc12def80 R15: 00007ffcc12dee70
Oct 18 01:25:14 zfstest kernel:  </TASK>
Oct 18 01:25:14 zfstest kernel: ==================================================================
Oct 18 01:25:14 zfstest kernel: BUG: KCSAN: data-race in folio_mark_accessed / workingset_activation
Oct 18 01:25:14 zfstest kernel:
Oct 18 01:25:14 zfstest kernel: write (marked) to 0xffffe5d6457b5bc0 of 8 bytes by task 225107 on cpu 5:
Oct 18 01:25:14 zfstest kernel:  folio_mark_accessed+0x3e/0x340
Oct 18 01:25:14 zfstest kernel:  mark_page_accessed+0x24/0x80
Oct 18 01:25:14 zfstest kernel:  unmap_page_range+0x85c/0x17a0
Oct 18 01:25:14 zfstest kernel:  unmap_single_vma+0xbf/0x160
Oct 18 01:25:14 zfstest kernel:  unmap_vmas+0xe7/0x1a0
Oct 18 01:25:14 zfstest kernel:  exit_mmap+0xde/0x230
Oct 18 01:25:14 zfstest kernel:  mmput+0xb5/0x1e0
Oct 18 01:25:14 zfstest kernel:  do_exit+0x549/0x1260
Oct 18 01:25:14 zfstest kernel:  do_group_exit+0x53/0x100
Oct 18 01:25:14 zfstest kernel:  __x64_sys_exit_group+0x21/0x30
Oct 18 01:25:14 zfstest kernel:  do_syscall_64+0x2f/0x50
Oct 18 01:25:14 zfstest kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 18 01:25:14 zfstest kernel:
Oct 18 01:25:14 zfstest kernel: read to 0xffffe5d6457b5bc0 of 8 bytes by task 225101 on cpu 22:
Oct 18 01:25:14 zfstest kernel:  workingset_activation+0xf5/0x1b0
Oct 18 01:25:14 zfstest kernel:  folio_mark_accessed+0x1ff/0x340
Oct 18 01:25:14 zfstest kernel:  mark_page_accessed+0x24/0x80
Oct 18 01:25:14 zfstest kernel:  unmap_page_range+0x85c/0x17a0
Oct 18 01:25:14 zfstest kernel:  unmap_single_vma+0xbf/0x160
Oct 18 01:25:14 zfstest kernel:  unmap_vmas+0xe7/0x1a0
Oct 18 01:25:14 zfstest kernel:  exit_mmap+0xde/0x230
Oct 18 01:25:14 zfstest kernel:  mmput+0xb5/0x1e0
Oct 18 01:25:14 zfstest kernel:  do_exit+0x549/0x1260
Oct 18 01:25:14 zfstest kernel:  do_group_exit+0x53/0x100
Oct 18 01:25:14 zfstest kernel:  __x64_sys_exit_group+0x21/0x30
Oct 18 01:25:14 zfstest kernel:  do_syscall_64+0x2f/0x50
Oct 18 01:25:14 zfstest kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 18 01:25:14 zfstest kernel:
Oct 18 01:25:14 zfstest kernel: value changed: 0x0017ffffc0000014 -> 0x0017ffffc0000016
Oct 18 01:25:14 zfstest kernel:
Oct 18 01:25:14 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:14 zfstest kernel: CPU: 22 PID: 225101 Comm: cc1plus Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:14 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:14 zfstest kernel: ==================================================================
Oct 18 01:25:15 zfstest kernel: ==================================================================
Oct 18 01:25:15 zfstest kernel: BUG: KCSAN: data-race in metaslab_class_throttle_reserve [zfs] / metaslab_group_increment_qdepth [zfs]
Oct 18 01:25:15 zfstest kernel:
Oct 18 01:25:15 zfstest kernel: read-write (marked) to 0xffffa0b963b4c2d0 of 8 bytes by task 565 on cpu 12:
Oct 18 01:25:15 zfstest kernel:  metaslab_group_increment_qdepth+0xc5/0xe0 [zfs]
Oct 18 01:25:15 zfstest kernel:  metaslab_group_alloc_decrement+0x90/0xa0 [zfs]
Oct 18 01:25:15 zfstest kernel:  zio_dva_throttle_done+0x257/0x560 [zfs]
Oct 18 01:25:15 zfstest kernel:
Oct 18 01:25:15 zfstest kernel: read to 0xffffa0b963b4c2d0 of 8 bytes by task 225190 on cpu 1:
Oct 18 01:25:15 zfstest kernel:  metaslab_class_throttle_reserve+0x37/0x110 [zfs]
Oct 18 01:25:15 zfstest kernel:  zio_io_to_allocate+0xde/0x1c0 [zfs]
Oct 18 01:25:15 zfstest kernel:
Oct 18 01:25:15 zfstest kernel: value changed: 0x0000000000000041 -> 0x0000000000000042
Oct 18 01:25:15 zfstest kernel:
Oct 18 01:25:15 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:15 zfstest kernel: CPU: 1 PID: 225190 Comm: z_wr_iss Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:15 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:15 zfstest kernel: ==================================================================
Oct 18 01:25:15 zfstest kernel: ==================================================================
Oct 18 01:25:15 zfstest kernel: BUG: KCSAN: data-race in spa_config_exit [zfs] / spa_config_held [zfs]
Oct 18 01:25:15 zfstest kernel:
Oct 18 01:25:15 zfstest kernel: write to 0xffffa0b9670762bc of 4 bytes by task 225546 on cpu 20:
Oct 18 01:25:15 zfstest kernel:  spa_config_exit+0xa4/0x1a0 [zfs]
Oct 18 01:25:15 zfstest kernel:  metaslab_alloc+0x246/0x700 [zfs]
Oct 18 01:25:15 zfstest kernel:  zio_dva_allocate+0x27f/0x800 [zfs]
Oct 18 01:25:15 zfstest kernel:  zio_execute+0x18c/0x2a0 [zfs]
Oct 18 01:25:15 zfstest kernel:
Oct 18 01:25:15 zfstest kernel: read to 0xffffa0b9670762bc of 4 bytes by task 225564 on cpu 14:
Oct 18 01:25:15 zfstest kernel:  spa_config_held+0x5e/0xb0 [zfs]
Oct 18 01:25:15 zfstest kernel:  vdev_lookup_top+0x34/0xd0 [zfs]
Oct 18 01:25:15 zfstest kernel:  metaslab_check_free+0x173/0x330 [zfs]
Oct 18 01:25:15 zfstest kernel:  zio_free_sync+0x7c/0x1b0 [zfs]
Oct 18 01:25:15 zfstest kernel:  zio_free+0xd7/0x110 [zfs]
Oct 18 01:25:15 zfstest kernel:  dsl_free+0x25/0x30 [zfs]
Oct 18 01:25:15 zfstest kernel:  dsl_dataset_block_kill+0x301/0xc30 [zfs]
Oct 18 01:25:15 zfstest kernel:  dbuf_write_done+0x545/0x9c0 [zfs]
Oct 18 01:25:15 zfstest kernel:
Oct 18 01:25:15 zfstest kernel: value changed: 0x00000001 -> 0x00000000
Oct 18 01:25:15 zfstest kernel:
Oct 18 01:25:15 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:15 zfstest kernel: CPU: 14 PID: 225564 Comm: z_wr_int_3 Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:15 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:15 zfstest kernel: ==================================================================
Oct 18 01:25:15 zfstest kernel: ==================================================================
Oct 18 01:25:15 zfstest kernel: BUG: KCSAN: data-race in spa_config_enter [zfs] / spa_config_held [zfs]
Oct 18 01:25:15 zfstest kernel:
Oct 18 01:25:15 zfstest kernel: write to 0xffffa0b9670762bc of 4 bytes by task 225539 on cpu 8:
Oct 18 01:25:15 zfstest kernel:  spa_config_enter+0x16a/0x290 [zfs]
Oct 18 01:25:15 zfstest kernel:  metaslab_alloc+0x8d/0x700 [zfs]
Oct 18 01:25:15 zfstest kernel:  zio_dva_allocate+0x27f/0x800 [zfs]
Oct 18 01:25:15 zfstest kernel:  zio_execute+0x18c/0x2a0 [zfs]
Oct 18 01:25:15 zfstest kernel:
Oct 18 01:25:15 zfstest kernel: read to 0xffffa0b9670762bc of 4 bytes by task 225575 on cpu 2:
Oct 18 01:25:15 zfstest kernel:  spa_config_held+0x5e/0xb0 [zfs]
Oct 18 01:25:15 zfstest kernel:  dva_get_dsize_sync+0x3d/0xd0 [zfs]
Oct 18 01:25:15 zfstest kernel:  bp_get_dsize_sync+0xaa/0x1f0 [zfs]
Oct 18 01:25:15 zfstest kernel:  dsl_dataset_block_kill+0x4a/0xc30 [zfs]
Oct 18 01:25:15 zfstest kernel:  dbuf_write_done+0x545/0x9c0 [zfs]
Oct 18 01:25:15 zfstest kernel:
Oct 18 01:25:15 zfstest kernel: value changed: 0x00000000 -> 0x00000001
Oct 18 01:25:15 zfstest kernel:
Oct 18 01:25:15 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:15 zfstest kernel: CPU: 2 PID: 225575 Comm: z_wr_int_0 Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:15 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:15 zfstest kernel: ==================================================================
Oct 18 01:25:16 zfstest kernel: ==================================================================
Oct 18 01:25:16 zfstest kernel: BUG: KCSAN: data-race in zrl_add_impl [zfs] / zrl_remove [zfs]
Oct 18 01:25:16 zfstest kernel:
Oct 18 01:25:16 zfstest kernel: write to 0xffffa0b963b4f0b0 of 8 bytes by task 755 on cpu 6:
Oct 18 01:25:16 zfstest kernel:  zrl_add_impl+0x11b/0x1a0 [zfs]
Oct 18 01:25:16 zfstest kernel:
Oct 18 01:25:16 zfstest kernel: read to 0xffffa0b963b4f0b0 of 8 bytes by task 702 on cpu 2:
Oct 18 01:25:16 zfstest kernel:  zrl_remove+0x19/0xb0 [zfs]
Oct 18 01:25:16 zfstest kernel:  dbuf_verify+0x414/0xbf0 [zfs]
Oct 18 01:25:16 zfstest kernel:  dbuf_rele_and_unlock+0x5e/0x7e0 [zfs]
Oct 18 01:25:16 zfstest kernel:  dnode_rele_and_unlock+0x10b/0x160 [zfs]
Oct 18 01:25:16 zfstest kernel:  dbuf_destroy+0x460/0x930 [zfs]
Oct 18 01:25:16 zfstest kernel:  dbuf_rele_and_unlock+0x2ff/0x7e0 [zfs]
Oct 18 01:25:16 zfstest kernel:  dbuf_rele+0x55/0x80 [zfs]
Oct 18 01:25:16 zfstest kernel:
Oct 18 01:25:16 zfstest kernel: value changed: 0xffffa0b9651f3380 -> 0x0000000000000000
Oct 18 01:25:16 zfstest kernel:
Oct 18 01:25:16 zfstest kernel: Reported by Kernel Concurrency Sanitizer on:
Oct 18 01:25:16 zfstest kernel: CPU: 2 PID: 702 Comm: txg_sync Tainted: P           OE     5.18.14-with-kcsan #2
Oct 18 01:25:16 zfstest kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct 18 01:25:16 zfstest kernel: ==================================================================

@rincebrain
Copy link
Contributor

A note - my limited understanding of the problem is that:

  • ZFS does a lot of buffering to make mmap work on Linux, for reasons I think were partially explained to me once that I've forgotten :/
  • This appears to be you being able to hit a window between when ZFS allocates a buffer for mmap and actually filling it in, thus the assert trip about the page not being up to date on debug builds and the incorrect data without that passes scrubs otherwise (since the buffered copy is just that).

So I don't think you're going to trip the sanitizers (since it's a correctly allocated and sized buffer, and I would not be surprised if things were zeroing the source of the buffer before we got to this point...), since it's an unhandled edge case, not a wild pointer or incorrectly sized item. Technically I imagine if you annotated KCSAN enough to know what was going on you might be able to convince it it's a data race?

But this is based on vague memories of looking at this when it was first posted, so who knows.

@TheUbuntuGuy
Copy link
Author

I'd be surprised if the buffer is zeroed before being filled in. In the one case I managed to see the page contents it was definitely not zeroed so that lends some credence to that. So if that's the case I would expect this to be treated as a uninitialized read by a sanitizer, which is something they should be able to catch (as long as it understands the memory allocator). This is all just speculation from me however - the memory mapped code goes over my head, so I'm just collecting data for someone else to figure it out.

@rincebrain
Copy link
Contributor

My guess would be it doesn't understand the allocator well enough, but I don't know, I've rarely seen sanitizers do useful things with mmap in particular, so my expectations are pretty low.

@allanjude
Copy link
Contributor

We have seen a similar issue / stack from chrome running inside a docker container:

Dec  7 13:16:18 runbot118 kernel: [158150.247007] VERIFY(PageUptodate(pp)) failed
Dec  7 13:16:18 runbot118 kernel: [158150.298042] PANIC at zfs_vnops_os.c:322:mappedread()
Dec  7 13:16:18 runbot118 kernel: [158150.358455] Showing stack for process 529553
Dec  7 13:16:18 runbot118 kernel: [158150.358458] CPU: 0 PID: 529553 Comm: google-chrome Tainted: P           OE     5.15.0-56-generic #62-Ubuntu
Dec  7 13:16:18 runbot118 kernel: [158150.358460] Hardware name: To Be Filled By O.E.M. E3C252D4U-2T/E3C252D4U-2T/OVH, BIOS 1.31.OV02 01/28/2022
Dec  7 13:16:18 runbot118 kernel: [158150.358461] Call Trace:
Dec  7 13:16:18 runbot118 kernel: [158150.358463]  <TASK>
Dec  7 13:16:18 runbot118 kernel: [158150.358465]  show_stack+0x52/0x5c
Dec  7 13:16:18 runbot118 kernel: [158150.358469]  dump_stack_lvl+0x4a/0x63
Dec  7 13:16:18 runbot118 kernel: [158150.358473]  dump_stack+0x10/0x16
Dec  7 13:16:18 runbot118 kernel: [158150.358475]  spl_dumpstack+0x29/0x2f [spl]
Dec  7 13:16:18 runbot118 kernel: [158150.358482]  spl_panic+0xd1/0xe9 [spl]
Dec  7 13:16:18 runbot118 kernel: [158150.358488]  ? prepend_copy+0x1b/0x50
Dec  7 13:16:18 runbot118 kernel: [158150.358490]  ? xas_load+0x17/0xd0
Dec  7 13:16:18 runbot118 kernel: [158150.358492]  ? __cond_resched+0x1a/0x50
Dec  7 13:16:18 runbot118 kernel: [158150.358494]  ? pagecache_get_page+0x28b/0x590
Dec  7 13:16:18 runbot118 kernel: [158150.358496]  ? __kmalloc_node+0x166/0x3a0
Dec  7 13:16:18 runbot118 kernel: [158150.358498]  mappedread+0x148/0x160 [zfs]
Dec  7 13:16:18 runbot118 kernel: [158150.358599]  zfs_read+0x18c/0x3c0 [zfs]
Dec  7 13:16:18 runbot118 kernel: [158150.358679]  zpl_iter_read+0xa3/0x110 [zfs]
Dec  7 13:16:18 runbot118 kernel: [158150.358759]  __kernel_read+0x141/0x2d0
Dec  7 13:16:18 runbot118 kernel: [158150.358763]  kernel_read+0x59/0xa0
Dec  7 13:16:18 runbot118 kernel: [158150.358764]  search_binary_handler+0x77/0x270
Dec  7 13:16:18 runbot118 kernel: [158150.358766]  exec_binprm+0x51/0x180
Dec  7 13:16:18 runbot118 kernel: [158150.358767]  bprm_execve.part.0+0x181/0x220
Dec  7 13:16:18 runbot118 kernel: [158150.358768]  bprm_execve+0x5e/0x90
Dec  7 13:16:18 runbot118 kernel: [158150.358770]  do_execveat_common.isra.0+0x198/0x210
Dec  7 13:16:18 runbot118 kernel: [158150.358771]  __x64_sys_execve+0x37/0x50
Dec  7 13:16:18 runbot118 kernel: [158150.358773]  do_syscall_64+0x59/0xc0
Dec  7 13:16:18 runbot118 kernel: [158150.358777]  ? syscall_exit_to_user_mode+0x27/0x50
Dec  7 13:16:18 runbot118 kernel: [158150.358779]  ? do_syscall_64+0x69/0xc0
Dec  7 13:16:18 runbot118 kernel: [158150.358781]  ? exit_to_user_mode_prepare+0x37/0xb0
Dec  7 13:16:18 runbot118 kernel: [158150.358784]  ? syscall_exit_to_user_mode+0x27/0x50
Dec  7 13:16:18 runbot118 kernel: [158150.358785]  ? do_syscall_64+0x69/0xc0
Dec  7 13:16:18 runbot118 kernel: [158150.358787]  ? exit_to_user_mode_prepare+0x37/0xb0
Dec  7 13:16:18 runbot118 kernel: [158150.358788]  ? irqentry_exit_to_user_mode+0x9/0x20
Dec  7 13:16:18 runbot118 kernel: [158150.358790]  ? irqentry_exit+0x1d/0x30
Dec  7 13:16:18 runbot118 kernel: [158150.358791]  ? exc_page_fault+0x89/0x170
Dec  7 13:16:18 runbot118 kernel: [158150.358793]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  7 13:16:18 runbot118 kernel: [158150.358796] RIP: 0033:0x7f4f3b8f417b
Dec  7 13:16:18 runbot118 kernel: [158150.358798] Code: 41 89 01 eb da 66 2e 0f 1f 84 00 00 00 00 00 f7 d8 64 41 89 01 eb d6 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 3b 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d e5 8c 10 00 f7 d8 64 89 01 48
Dec  7 13:16:18 runbot118 kernel: [158150.358799] RSP: 002b:00007fff0c699a98 EFLAGS: 00000246 ORIG_RAX: 000000000000003b
Dec  7 13:16:18 runbot118 kernel: [158150.358800] RAX: ffffffffffffffda RBX: 000055c299dc6920 RCX: 00007f4f3b8f417b
Dec  7 13:16:18 runbot118 kernel: [158150.358802] RDX: 000055c299dc6940 RSI: 000055c299dc2490 RDI: 000055c299dc6920
Dec  7 13:16:18 runbot118 kernel: [158150.358802] RBP: 000055c299dc6920 R08: 00007fff0c699a70 R09: 0000000000000000
Dec  7 13:16:18 runbot118 kernel: [158150.358803] R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000
Dec  7 13:16:18 runbot118 kernel: [158150.358803] R13: 000055c299dc2490 R14: 000055c299dc6940 R15: 000055c299dc6940
Dec  7 13:16:18 runbot118 kernel: [158150.358805]  </TASK>

@robn robn mentioned this issue Feb 16, 2023
13 tasks
lundman pushed a commit to openzfsonwindows/openzfs that referenced this issue Mar 3, 2023
When a page is faulted in for memory mapped I/O the page lock
may be dropped before it has been read and marked up to date.
If a buffered read encounters such a page in mappedread() it
must wait until the page has been updated. Failure to do so
will result in a panic on debug builds and incorrect data on
production builds.

The critical part of this change is in mappedread() where pages
which are not up to date are now handled. Additionally, it
includes the following simplifications.

- zfs_getpage() and zfs_fillpage() could be passed an array of
  pages. This could be more efficient if it was used but in
  practice only a single page was ever provided. These
  interfaces were simplified to acknowledge that.

- update_pages() was modified to correctly set the PG_error bit
  on a page when it cannot be read by dmu_read().

- Setting PG_error and PG_uptodate was moved to zfs_fillpage()
  from zpl_readpage_common(). This is consistent with the
  handling in update_pages() and mappedread().

- Minor additional refactoring to comments and variable
  declarations to improve readability.

- Add a test case to exercise concurrent buffered, direct,
  and mmap IO to the same file.

- Reduce the mmap_sync test case default run time.

Reviewed-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Reviewed-by: Brian Atkinson <batkinson@lanl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes openzfs#13608 
Closes openzfs#14498
robn pushed a commit to robn/zfs that referenced this issue Apr 15, 2023
When a page is faulted in for memory mapped I/O the page lock
may be dropped before it has been read and marked up to date.
If a buffered read encounters such a page in mappedread() it
must wait until the page has been updated. Failure to do so
will result in a panic on debug builds and incorrect data on
production builds.

The critical part of this change is in mappedread() where pages
which are not up to date are now handled. Additionally, it
includes the following simplifications.

- zfs_getpage() and zfs_fillpage() could be passed an array of
  pages. This could be more efficient if it was used but in
  practice only a single page was ever provided. These
  interfaces were simplified to acknowledge that.

- update_pages() was modified to correctly set the PG_error bit
  on a page when it cannot be read by dmu_read().

- Setting PG_error and PG_uptodate was moved to zfs_fillpage()
  from zpl_readpage_common(). This is consistent with the
  handling in update_pages() and mappedread().

- Minor additional refactoring to comments and variable
  declarations to improve readability.

- Add a test case to exercise concurrent buffered, direct,
  and mmap IO to the same file.

- Reduce the mmap_sync test case default run time.

Reviewed-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Reviewed-by: Brian Atkinson <batkinson@lanl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes openzfs#13608
Closes openzfs#14498
robn pushed a commit to robn/zfs that referenced this issue Apr 15, 2023
When a page is faulted in for memory mapped I/O the page lock
may be dropped before it has been read and marked up to date.
If a buffered read encounters such a page in mappedread() it
must wait until the page has been updated. Failure to do so
will result in a panic on debug builds and incorrect data on
production builds.

The critical part of this change is in mappedread() where pages
which are not up to date are now handled. Additionally, it
includes the following simplifications.

- zfs_getpage() and zfs_fillpage() could be passed an array of
  pages. This could be more efficient if it was used but in
  practice only a single page was ever provided. These
  interfaces were simplified to acknowledge that.

- update_pages() was modified to correctly set the PG_error bit
  on a page when it cannot be read by dmu_read().

- Setting PG_error and PG_uptodate was moved to zfs_fillpage()
  from zpl_readpage_common(). This is consistent with the
  handling in update_pages() and mappedread().

- Minor additional refactoring to comments and variable
  declarations to improve readability.

- Add a test case to exercise concurrent buffered, direct,
  and mmap IO to the same file.

- Reduce the mmap_sync test case default run time.

Reviewed-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Reviewed-by: Brian Atkinson <batkinson@lanl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes openzfs#13608
Closes openzfs#14498
robn pushed a commit to robn/zfs that referenced this issue Apr 20, 2023
When a page is faulted in for memory mapped I/O the page lock
may be dropped before it has been read and marked up to date.
If a buffered read encounters such a page in mappedread() it
must wait until the page has been updated. Failure to do so
will result in a panic on debug builds and incorrect data on
production builds.

The critical part of this change is in mappedread() where pages
which are not up to date are now handled. Additionally, it
includes the following simplifications.

- zfs_getpage() and zfs_fillpage() could be passed an array of
  pages. This could be more efficient if it was used but in
  practice only a single page was ever provided. These
  interfaces were simplified to acknowledge that.

- update_pages() was modified to correctly set the PG_error bit
  on a page when it cannot be read by dmu_read().

- Setting PG_error and PG_uptodate was moved to zfs_fillpage()
  from zpl_readpage_common(). This is consistent with the
  handling in update_pages() and mappedread().

- Minor additional refactoring to comments and variable
  declarations to improve readability.

- Add a test case to exercise concurrent buffered, direct,
  and mmap IO to the same file.

- Reduce the mmap_sync test case default run time.

Reviewed-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Reviewed-by: Brian Atkinson <batkinson@lanl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes openzfs#13608
Closes openzfs#14498
behlendorf added a commit that referenced this issue Apr 21, 2023
When a page is faulted in for memory mapped I/O the page lock
may be dropped before it has been read and marked up to date.
If a buffered read encounters such a page in mappedread() it
must wait until the page has been updated. Failure to do so
will result in a panic on debug builds and incorrect data on
production builds.

The critical part of this change is in mappedread() where pages
which are not up to date are now handled. Additionally, it
includes the following simplifications.

- zfs_getpage() and zfs_fillpage() could be passed an array of
  pages. This could be more efficient if it was used but in
  practice only a single page was ever provided. These
  interfaces were simplified to acknowledge that.

- update_pages() was modified to correctly set the PG_error bit
  on a page when it cannot be read by dmu_read().

- Setting PG_error and PG_uptodate was moved to zfs_fillpage()
  from zpl_readpage_common(). This is consistent with the
  handling in update_pages() and mappedread().

- Minor additional refactoring to comments and variable
  declarations to improve readability.

- Add a test case to exercise concurrent buffered, direct,
  and mmap IO to the same file.

- Reduce the mmap_sync test case default run time.

Reviewed-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Reviewed-by: Brian Atkinson <batkinson@lanl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #13608
Closes #14498
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: Memory Management kernel memory management Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
None yet
Development

No branches or pull requests

5 participants