Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

debug PANIC at dmu_brt_clone(), VERIFY3(nbps == numbufs) failed #15724

Closed
rrevans opened this issue Dec 31, 2023 · 1 comment · Fixed by #15735
Closed

debug PANIC at dmu_brt_clone(), VERIFY3(nbps == numbufs) failed #15724

rrevans opened this issue Dec 31, 2023 · 1 comment · Fixed by #15735
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)

Comments

@rrevans
Copy link
Contributor

rrevans commented Dec 31, 2023

System information

Type Version/Name
Distribution Name Fedora core
Distribution Version 37
Kernel Version 6.5.12-100.fc37
Architecture x86_64
OpenZFS Version zfs-2.2.99-281_g07e95b467 (built at head)

cp is from coreutils-9.1-8.fc37.x86_64

Describe the problem you're observing

PANIC with debug build when cloning a file if source and target files have different recordsize.

In non-debug builds, clone fails with Invalid cross-device link due to block size mismatch.

The assertion in dmu_brt_clone probably needs to be reworked to tolerate dmu_buf_hold_array returning an unexpected number of buffers in this case?

Describe how to reproduce the problem

  1. build 07e95b467 with ./configure --enable-debug
  2. import/create test pool and dataset
  3. create two files with different recordsize
  4. cp --reflink=always <file1> <file2>

Example script:

# replace "test" with name of dataset for cwd 
zfs set recordsize=128k test
dd if=/dev/urandom bs=128k count=2 of=x
zfs set recordsize=256k test
dd if=/dev/urandom bs=128k count=2 of=y
cp --reflink=always x y

Include any warning/errors/backtraces from the system logs

dmesg:

[  961.798837] VERIFY3(nbps == numbufs) failed (4 == 2)
[  961.798862] PANIC at dmu.c:2315:dmu_brt_clone()
[  961.798877] Showing stack for process 4634
[  961.798879] CPU: 9 PID: 4634 Comm: cp Tainted: P           OE      6.5.12-100.fc37.x86_64 #1
[  961.798883] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-1.fc37 04/01/2014
[  961.798885] Call Trace:
[  961.798888]  <TASK>
[  961.798891]  dump_stack_lvl+0x47/0x60
[  961.798901]  spl_panic+0x100/0x120 [spl]
[  961.798927]  ? srso_alias_return_thunk+0x5/0x7f
[  961.798932]  ? dnode_rele_and_unlock+0x73/0x1f0 [zfs]
[  961.799164]  dmu_brt_clone+0x5e6/0x780 [zfs]
[  961.799380]  ? srso_alias_return_thunk+0x5/0x7f
[  961.799385]  ? srso_alias_return_thunk+0x5/0x7f
[  961.799389]  ? dmu_tx_assign+0x3eb/0x830 [zfs]
[  961.799618]  zfs_clone_range+0xb8b/0x1030 [zfs]
[  961.799844]  __zpl_clone_file_range.isra.0+0xf3/0x1b0 [zfs]
[  961.800043]  do_clone_file_range+0x105/0x290
[  961.800050]  vfs_clone_file_range+0x3e/0x140
[  961.800056]  ioctl_file_clone+0x49/0xb0
[  961.800062]  do_vfs_ioctl+0x7a/0x980
[  961.800067]  ? srso_alias_return_thunk+0x5/0x7f
[  961.800073]  __x64_sys_ioctl+0x72/0xd0
[  961.800079]  do_syscall_64+0x5f/0x90
[  961.800084]  ? srso_alias_return_thunk+0x5/0x7f
[  961.800089]  ? srso_alias_return_thunk+0x5/0x7f
[  961.800093]  ? syscall_exit_to_user_mode+0x2b/0x40
[  961.800097]  ? srso_alias_return_thunk+0x5/0x7f
[  961.800101]  ? do_syscall_64+0x6b/0x90
[  961.800105]  ? do_syscall_64+0x6b/0x90
[  961.800109]  ? srso_alias_return_thunk+0x5/0x7f
[  961.800113]  ? srso_alias_return_thunk+0x5/0x7f
[  961.800117]  ? syscall_exit_to_user_mode+0x2b/0x40
[  961.800121]  ? srso_alias_return_thunk+0x5/0x7f
[  961.800125]  ? do_syscall_64+0x6b/0x90
[  961.800128]  ? srso_alias_return_thunk+0x5/0x7f
[  961.800132]  ? syscall_exit_to_user_mode+0x2b/0x40
[  961.800136]  ? srso_alias_return_thunk+0x5/0x7f
[  961.800140]  ? do_syscall_64+0x6b/0x90
[  961.800144]  ? srso_alias_return_thunk+0x5/0x7f
[  961.800148]  ? exc_page_fault+0x77/0x170
[  961.800153]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  961.800158] RIP: 0033:0x7f85468ace0f
[  961.800170] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 18 48 8b 44 24 18 64 48 2b 04 25 28 00 00
[  961.800173] RSP: 002b:00007ffd77a5f5a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  961.800178] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007f85468ace0f
[  961.800181] RDX: 0000000000000003 RSI: 0000000040049409 RDI: 0000000000000004
[  961.800183] RBP: 00007ffd77a6178d R08: 0000000000000002 R09: 00000000000001a4
[  961.800185] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd77a5fa50
[  961.800187] R13: 0000000000000001 R14: 0000000000000014 R15: 00007ffd77a5fe10
[  961.800194]  </TASK>

/proc/spl/kstat/zfs/dbgmsg only reports recordsize changes and syncs

strace says cp is using the FICLONE ioctl to clone the file.
@rrevans rrevans added the Type: Defect Incorrect behavior (e.g. crash, hang) label Dec 31, 2023
@amotin amotin mentioned this issue Jan 3, 2024
Merged
13 tasks
@amotin
Copy link
Member

amotin commented Jan 3, 2024

Thanks for the clear scenario. I've reproduced it on FreeBSD. #15735 should fix it.

behlendorf pushed a commit that referenced this issue Jan 9, 2024
@amotin
Improve block sizes checks during cloning
255741f 
- Fail if source block is smaller than destination.  We can only
grow blocks, not shrink them.
 - Fail if we do not have full znode range lock.  In that case grow
is not even called.  We should improve zfs_rangelock_cb() somehow
to know when cloning needs to grow the block size unlike write.
 - Fail of we tried to resize, but failed.  There are many reasons
for it to fail that we can not predict at this level, so be ready
for them.  Unlike write, that may proceed after growth failure,
block cloning can't and must return error.

This fixes assertion inside dmu_brt_clone() when it sees different
number of blocks held in destination than it got block pointers.
Builds without ZFS_DEBUG returned EXDEV, so are not affected much.

Reviewed-by: Pawel Jakub Dawidek <pawel@dawidek.net>
Reviewed-by: Brian Atkinson <batkinson@lanl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by:	Alexander Motin <mav@FreeBSD.org>
Sponsored by:	iXsystems, Inc.
Closes #15724 
Closes #15735
amotin added a commit to amotin/zfs that referenced this issue Jan 9, 2024
@amotin
Improve block sizes checks during cloning
59e3bf5 
- Fail if source block is smaller than destination.  We can only
grow blocks, not shrink them.
 - Fail if we do not have full znode range lock.  In that case grow
is not even called.  We should improve zfs_rangelock_cb() somehow
to know when cloning needs to grow the block size unlike write.
 - Fail of we tried to resize, but failed.  There are many reasons
for it to fail that we can not predict at this level, so be ready
for them.  Unlike write, that may proceed after growth failure,
block cloning can't and must return error.

This fixes assertion inside dmu_brt_clone() when it sees different
number of blocks held in destination than it got block pointers.
Builds without ZFS_DEBUG returned EXDEV, so are not affected much.

Reviewed-by: Pawel Jakub Dawidek <pawel@dawidek.net>
Reviewed-by: Brian Atkinson <batkinson@lanl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by:	Alexander Motin <mav@FreeBSD.org>
Sponsored by:	iXsystems, Inc.
Closes openzfs#15724 
Closes openzfs#15735
mmatuska pushed a commit to mmatuska/zfs that referenced this issue Jan 10, 2024
@amotin @mmatuska
Improve block sizes checks during cloning
6106a00 
- Fail if source block is smaller than destination.  We can only
grow blocks, not shrink them.
 - Fail if we do not have full znode range lock.  In that case grow
is not even called.  We should improve zfs_rangelock_cb() somehow
to know when cloning needs to grow the block size unlike write.
 - Fail of we tried to resize, but failed.  There are many reasons
for it to fail that we can not predict at this level, so be ready
for them.  Unlike write, that may proceed after growth failure,
block cloning can't and must return error.

This fixes assertion inside dmu_brt_clone() when it sees different
number of blocks held in destination than it got block pointers.
Builds without ZFS_DEBUG returned EXDEV, so are not affected much.

Reviewed-by: Pawel Jakub Dawidek <pawel@dawidek.net>
Reviewed-by: Brian Atkinson <batkinson@lanl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by:	Alexander Motin <mav@FreeBSD.org>
Sponsored by:	iXsystems, Inc.
Closes openzfs#15724 
Closes openzfs#15735
behlendorf pushed a commit that referenced this issue Jan 12, 2024
@amotin @behlendorf
Improve block sizes checks during cloning
152a775 
- Fail if source block is smaller than destination.  We can only
grow blocks, not shrink them.
 - Fail if we do not have full znode range lock.  In that case grow
is not even called.  We should improve zfs_rangelock_cb() somehow
to know when cloning needs to grow the block size unlike write.
 - Fail of we tried to resize, but failed.  There are many reasons
for it to fail that we can not predict at this level, so be ready
for them.  Unlike write, that may proceed after growth failure,
block cloning can't and must return error.

This fixes assertion inside dmu_brt_clone() when it sees different
number of blocks held in destination than it got block pointers.
Builds without ZFS_DEBUG returned EXDEV, so are not affected much.

Reviewed-by: Pawel Jakub Dawidek <pawel@dawidek.net>
Reviewed-by: Brian Atkinson <batkinson@lanl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by:	Alexander Motin <mav@FreeBSD.org>
Sponsored by:	iXsystems, Inc.
Closes #15724 
Closes #15735
lundman pushed a commit to openzfsonwindows/openzfs that referenced this issue Mar 13, 2024
@amotin @lundman
Improve block sizes checks during cloning
05f8f67 
- Fail if source block is smaller than destination.  We can only
grow blocks, not shrink them.
 - Fail if we do not have full znode range lock.  In that case grow
is not even called.  We should improve zfs_rangelock_cb() somehow
to know when cloning needs to grow the block size unlike write.
 - Fail of we tried to resize, but failed.  There are many reasons
for it to fail that we can not predict at this level, so be ready
for them.  Unlike write, that may proceed after growth failure,
block cloning can't and must return error.

This fixes assertion inside dmu_brt_clone() when it sees different
number of blocks held in destination than it got block pointers.
Builds without ZFS_DEBUG returned EXDEV, so are not affected much.

Reviewed-by: Pawel Jakub Dawidek <pawel@dawidek.net>
Reviewed-by: Brian Atkinson <batkinson@lanl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by:	Alexander Motin <mav@FreeBSD.org>
Sponsored by:	iXsystems, Inc.
Closes openzfs#15724 
Closes openzfs#15735
lundman pushed a commit to openzfsonwindows/openzfs that referenced this issue Mar 13, 2024
@amotin @lundman
Improve block sizes checks during cloning
457f69c 
- Fail if source block is smaller than destination.  We can only
grow blocks, not shrink them.
 - Fail if we do not have full znode range lock.  In that case grow
is not even called.  We should improve zfs_rangelock_cb() somehow
to know when cloning needs to grow the block size unlike write.
 - Fail of we tried to resize, but failed.  There are many reasons
for it to fail that we can not predict at this level, so be ready
for them.  Unlike write, that may proceed after growth failure,
block cloning can't and must return error.

This fixes assertion inside dmu_brt_clone() when it sees different
number of blocks held in destination than it got block pointers.
Builds without ZFS_DEBUG returned EXDEV, so are not affected much.

Reviewed-by: Pawel Jakub Dawidek <pawel@dawidek.net>
Reviewed-by: Brian Atkinson <batkinson@lanl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by:	Alexander Motin <mav@FreeBSD.org>
Sponsored by:	iXsystems, Inc.
Closes openzfs#15724 
Closes openzfs#15735
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Improve block sizes checks during cloning amotin/zfs
2 participants
@rrevans @amotin