Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ztest: pthread_mutex_destroy() fails with EBUSY #2027

Closed
Xaseron opened this issue Jan 4, 2014 · 6 comments
Closed

ztest: pthread_mutex_destroy() fails with EBUSY #2027

Xaseron opened this issue Jan 4, 2014 · 6 comments
Labels
Component: Test Suite Indicates an issue with the test framework or a test case
Milestone
0.8.0

Comments

@Xaseron
Copy link

Xaseron commented Jan 4, 2014

zdb: ../../lib/libzpool/kernel.c:263: Assertion `pthread_mutex_destroy(&(mp)->m_lock) == 0 (0x10 == 0x0)' failed.
@behlendorf
Copy link
Contributor

It appears pthread_mutex_destroy() returned EBUSY (16) unexpectedly. According to the man page EBUSY happen when...

EBUSY - The implementation has detected an attempt to destroy the object referenced by mutex while it is locked or referenced (for example, while being used in a pthread_cond_timedwait() or pthread_cond_wait()) by another thread.

So it sounds like there some code path in zdb where we attempt to destroy the mutex while it's still in use. If you can get an backtraces for the core that would be helpful.

@Xaseron
Copy link
Author

Xaseron commented Jan 10, 2014

Program received signal SIGABRT, Aborted.
0x00007ffff6e6e319 in raise () from /usr/lib/libc.so.6
(gdb) bt
#0 0x00007ffff6e6e319 in raise () from /usr/lib/libc.so.6
#1 0x00007ffff6e6f718 in abort () from /usr/lib/libc.so.6
#2 0x00007ffff6e67446 in __assert_fail_base () from /usr/lib/libc.so.6
#3 0x00007ffff6e674f2 in __assert_fail () from /usr/lib/libc.so.6
#4 0x00007ffff6e6756b in __assert () from /usr/lib/libc.so.6
#5 0x00007ffff7666794 in mutex_destroy () from /usr/lib/libzpool.so.1
#6 0x00007ffff7674570 in buf_dest () from /usr/lib/libzpool.so.1
#7 0x00007ffff767180a in arc_buf_destroy () from /usr/lib/libzpool.so.1
#8 0x00007ffff7672dc4 in arc_evict () from /usr/lib/libzpool.so.1
#9 0x00007ffff7676276 in arc_flush () from /usr/lib/libzpool.so.1
#10 0x00007ffff76bb0c1 in dsl_pool_close () from /usr/lib/libzpool.so.1
#11 0x00007ffff76d59d5 in spa_unload () from /usr/lib/libzpool.so.1
#12 0x00007ffff76d9a8b in spa_load () from /usr/lib/libzpool.so.1
#13 0x00007ffff76da6c6 in spa_load_best () from /usr/lib/libzpool.so.1
#14 0x00007ffff76daabc in spa_open_common () from /usr/lib/libzpool.so.1
#15 0x0000000000404b80 in main ()

@behlendorf behlendorf mentioned this issue Mar 5, 2014
Closed
@behlendorf behlendorf mentioned this issue Jul 29, 2014
Closed
@cwedgwood
Copy link
Contributor

Related:

Program received signal SIGABRT, Aborted.
0x00007ffff688f407 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffff688f407 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff68907e8 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007ffff6888526 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007ffff68885d2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007ffff688864b in __assert () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x00007ffff76a61e7 in mutex_destroy (mp=0x631d50) at ../../lib/libzpool/kernel.c:301
#6  0x00007ffff76aeded in buf_dest (vbuf=<optimized out>, unused=<optimized out>) at ../../module/zfs/arc.c:993
#7  0x00007ffff76aceef in umem_cache_free (cp=<optimized out>, cp=<optimized out>, ptr=0x631d40) at ../../lib/libspl/include/umem.h:191
#8  arc_buf_destroy (buf=0x631d40, recycle=<optimized out>, all=B_TRUE) at ../../module/zfs/arc.c:1671
#9  0x00007ffff76ad32d in arc_evict (state=0x7ffff79af5c0 <ARC_mru>, spa=spa@entry=2653086010684638390, bytes=bytes@entry=-1, recycle=recycle@entry=B_FALSE, type=type@entry=ARC_BUFC_METADATA)
    at ../../module/zfs/arc.c:1987
#10 0x00007ffff76b07d6 in arc_flush (spa=0x61fae0) at ../../module/zfs/arc.c:2357
#11 0x00007ffff76de6fb in dsl_pool_close (dp=0x631fc0) at ../../module/zfs/dsl_pool.c:318
#12 0x00007ffff76f857c in spa_unload (spa=spa@entry=0x61fae0) at ../../module/zfs/spa.c:1281
#13 0x00007ffff76fbd27 in spa_load_impl (pool_guid=<optimized out>, ereport=<synthetic pointer>, mosconfig=B_FALSE, type=SPA_IMPORT_EXISTING, state=SPA_LOAD_OPEN, config=<optimized out>, spa=0x61fae0)
    at ../../module/zfs/spa.c:2484
#14 spa_load (spa=spa@entry=0x61fae0, state=state@entry=SPA_LOAD_OPEN, type=type@entry=SPA_IMPORT_EXISTING, mosconfig=mosconfig@entry=B_FALSE) at ../../module/zfs/spa.c:2101
#15 0x00007ffff76fca99 in spa_load_best (spa=spa@entry=0x61fae0, state=state@entry=SPA_LOAD_OPEN, mosconfig=mosconfig@entry=0, max_request=<optimized out>, rewind_flags=2) at ../../module/zfs/spa.c:2823
#16 0x00007ffff76fcded in spa_open_common (pool=pool@entry=0x7fffffffee14 "pool0", spapp=spapp@entry=0x7fffffffea18, tag=tag@entry=0x40f9f5 <__func__.17356>, nvpolicy=<optimized out>, config=config@entry=0x0)
    at ../../module/zfs/spa.c:2946
#17 0x00007ffff76fd085 in spa_open_rewind (name=name@entry=0x7fffffffee14 "pool0", spapp=spapp@entry=0x7fffffffea18, tag=tag@entry=0x40f9f5 <__func__.17356>, policy=<optimized out>, config=config@entry=0x0)
    at ../../module/zfs/spa.c:3024
#18 0x0000000000404e43 in main (argc=1, argv=<optimized out>) at ../../cmd/zdb/zdb.c:3527

@behlendorf behlendorf changed the title zdb -S tank assertion failed Assertion `pthread_mutex_destroy(&(mp)->m_lock) == 0 (0x10 == 0x0)' failed. Aug 6, 2014
@cryptix
Copy link

cryptix commented Oct 4, 2014

same issue when I try to run zdb -C pool but i'ts (now?) at ../../lib/libzpool/kernel.c:301.

is there another way to get the GUID's of a pool? I want to detach a device that is UNAVAIL

backtrace:

Program received signal SIGABRT, Aborted.
0x00007ffff6eb3967 in raise () from /usr/lib/libc.so.6
(gdb) bt
#0  0x00007ffff6eb3967 in raise () from /usr/lib/libc.so.6
#1  0x00007ffff6eb4d3a in abort () from /usr/lib/libc.so.6
#2  0x00007ffff6eac8ad in __assert_fail_base () from /usr/lib/libc.so.6
#3  0x00007ffff6eac962 in __assert_fail () from /usr/lib/libc.so.6
#4  0x00007ffff6eac9db in __assert () from /usr/lib/libc.so.6
#5  0x00007ffff76a62a3 in mutex_destroy () from /usr/lib/libzpool.so.2
#6  0x00007ffff76aefdd in ?? () from /usr/lib/libzpool.so.2
#7  0x00007ffff76ad13f in ?? () from /usr/lib/libzpool.so.2
#8  0x00007ffff76ad59d in ?? () from /usr/lib/libzpool.so.2
#9  0x00007ffff76b09a6 in arc_flush () from /usr/lib/libzpool.so.2
#10 0x00007ffff76dd3aa in dsl_pool_close () from /usr/lib/libzpool.so.2
#11 0x00007ffff76f6222 in ?? () from /usr/lib/libzpool.so.2
#12 0x00007ffff76f9938 in ?? () from /usr/lib/libzpool.so.2
#13 0x00007ffff76fa6f9 in ?? () from /usr/lib/libzpool.so.2
#14 0x00007ffff76faa4d in ?? () from /usr/lib/libzpool.so.2
#15 0x0000000000404c53 in ?? ()
#16 0x00007ffff6ea0040 in __libc_start_main () from /usr/lib/libc.so.6
#17 0x000000000040618c in ?? ()

@behlendorf behlendorf removed this from the 0.6.4 milestone Oct 30, 2014
behlendorf added a commit to behlendorf/zfs that referenced this issue Oct 30, 2014
@behlendorf
Change VERIFY to ASSERT in mutex_destroy()
8ce94e5 
There have been multiple reports of 'zdb' tripping the VERIFY in
mutex_destroy() because pthread_mutex_destroy() returns EBUSY.

Exactly how this can happen still needs to be explained, but this
doesn't strictly need to be fatal for non-debug builds.  Therefore,
this patch converts the VERIFY to an ASSERT until the root cause
is determined and resolved.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue openzfs#2027
@behlendorf behlendorf mentioned this issue Oct 30, 2014
Closed
@ryao
Copy link
Contributor

ryao commented Nov 3, 2014

Pull request #2012 will add zpool status -g, which will provide GUIDs.

behlendorf added a commit to behlendorf/zfs that referenced this issue Jan 21, 2015
@behlendorf
Change VERIFY to ASSERT in mutex_destroy()
c07950c 
There have been multiple reports of 'zdb' tripping the VERIFY in
mutex_destroy() because pthread_mutex_destroy() returns EBUSY.

Exactly how this can happen still needs to be explained, but this
doesn't strictly need to be fatal for non-debug builds.  Therefore,
this patch converts the VERIFY to an ASSERT until the root cause
is determined and resolved.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue openzfs#2027
behlendorf added a commit that referenced this issue Feb 11, 2015
@behlendorf
Change VERIFY to ASSERT in mutex_destroy()
340dfbe 
There have been multiple reports of 'zdb' tripping the VERIFY in
mutex_destroy() because pthread_mutex_destroy() returns EBUSY.

Exactly how this can happen still needs to be explained, but this
doesn't strictly need to be fatal for non-debug builds.  Therefore,
this patch converts the VERIFY to an ASSERT until the root cause
is determined and resolved.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #2027
DeHackEd pushed a commit to DeHackEd/zfs that referenced this issue Apr 4, 2015
@behlendorf @DeHackEd
Change VERIFY to ASSERT in mutex_destroy()
27e9cc3 
There have been multiple reports of 'zdb' tripping the VERIFY in
mutex_destroy() because pthread_mutex_destroy() returns EBUSY.

Exactly how this can happen still needs to be explained, but this
doesn't strictly need to be fatal for non-debug builds.  Therefore,
this patch converts the VERIFY to an ASSERT until the root cause
is determined and resolved.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue openzfs#2027
DeHackEd pushed a commit to DeHackEd/zfs that referenced this issue Apr 5, 2015
@behlendorf @DeHackEd
Change VERIFY to ASSERT in mutex_destroy()
3355110 
There have been multiple reports of 'zdb' tripping the VERIFY in
mutex_destroy() because pthread_mutex_destroy() returns EBUSY.

Exactly how this can happen still needs to be explained, but this
doesn't strictly need to be fatal for non-debug builds.  Therefore,
this patch converts the VERIFY to an ASSERT until the root cause
is determined and resolved.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue openzfs#2027
@behlendorf behlendorf added the Component: Test Suite Indicates an issue with the test framework or a test case label Jul 16, 2016
@behlendorf behlendorf added this to the 0.8.0 milestone Jul 16, 2016
@behlendorf behlendorf changed the title Assertion `pthread_mutex_destroy(&(mp)->m_lock) == 0 (0x10 == 0x0)' failed. ztest: pthread_mutex_destroy() fails with EBUSY Jan 24, 2017
@behlendorf
Copy link
Contributor

Closing as stale.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Component: Test Suite Indicates an issue with the test framework or a test case
Projects
None yet
Milestone
0.8.0
Development

No branches or pull requests
5 participants
@cryptix @behlendorf @cwedgwood @Xaseron @ryao