segfault at error 4 in libtcmalloc #7905
Comments
|
Assuming the fix was elsewhere. Let us know if any change is needed from RocksDB. |
|
@ajkr @huangwei04 I think it should be reopened as the problem still exists. Do I need to provide any more detailed information? Thanks. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When I test CEPH performance whit vdbench, an OSD coredump。
message log
Jan 28 04:13:28 ceph04 kernel: [494544.449442] rocksdb:low16[639204]: segfault at 24 ip 00007f3396e14d63 sp 00007f33791e7648 error 4 in libtcmalloc.so.4.4.5[7f3396ddf000+46000]
core trace
$18 = "\000$ \004"
$19 = "\000$ \004"
Program terminated with signal 11, Segmentation fault.
#0 0x00007f3396e14d63 in tc_malloc () from /lib64/libtcmalloc.so.4
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.176-2.el7.cgslv5.x86_64 elfutils-libs-0.176-2.el7.cgslv5.x86_64 fuse-libs-2.9.2-11.el7.x86_64 glibc-2.17-292.el7.cgslv5.0.1.gf7a2c7a.x86_64 gperftools-libs-2.6.1-1.el7.x86_64 libaio-0.3.109-13.el7.x86_64 libattr-2.4.46-12.el7.x86_64 libblkid-2.23.2-43.el7_4.2.cgslv5.0.3.g36a7632.x86_64 libcap-2.22-9.el7.x86_64 libgcc-4.8.5-28.el7_5.1.x86_64 liboath-2.6.2-1.el7.x86_64 libstdc++-4.8.5-28.el7_5.1.x86_64 libuuid-2.23.2-43.el7_4.2.cgslv5.0.3.g36a7632.x86_64 lz4-1.7.5-2.el7.x86_64 nspr-4.21.0-1.el7.x86_64 nss-3.44.0-7.el7_7.cgslv5.0.1.g3030b5d.x86_64 nss-softokn-3.44.0-8.el7_7.x86_64 nss-softokn-freebl-3.44.0-8.el7_7.x86_64 nss-util-3.44.0-4.el7_7.x86_64 openssl-libs-1.0.2k-19.el7.cgslv5.x86_64 snappy-1.1.0-3.el7.x86_64 sqlite-3.7.17-8.el7.cgslv5.x86_64 systemd-libs-219-67.el7.cgslv5.0.13.gce5793d.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0 0x00007f3396e14d63 in tc_malloc () from /lib64/libtcmalloc.so.4
#1 0x00007f339765e580 in _dl_map_object_deps () from /lib64/ld-linux-x86-64.so.2
#2 0x00007f33976652ab in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#3 0x00007f3397660784 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#4 0x00007f3397664b3b in _dl_open () from /lib64/ld-linux-x86-64.so.2
#5 0x00007f3393e246f2 in do_dlopen () from /lib64/libc.so.6
#6 0x00007f3397660784 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#7 0x00007f3393e247b2 in __libc_dlopen_mode () from /lib64/libc.so.6
#8 0x00007f3393dfb9a5 in init () from /lib64/libc.so.6
#9 0x00007f3394af01cb in __pthread_once_slow () from /lib64/libpthread.so.0
#10 0x00007f3393dfbabc in backtrace () from /lib64/libc.so.6
#11 0x0000565188c79774 in BackTrace (s=1, this=0x7f33791e8620) at /usr/src/debug/ceph-14.2.11/src/common/BackTrace.h:28
#12 handle_fatal_signal(int) () at /usr/src/debug/ceph-14.2.11/src/global/signal_handler.cc:167
#13
#14 0x00007f3396e15ce3 in tc_newarray () from /lib64/libtcmalloc.so.4
#15 0x000056518862fab9 in allocate (this=, __n=)
at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.tcc:995
#16 std::string::_Rep::_S_create (__capacity=39, __old_capacity=, __alloc=...)
at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.tcc:1057
#17 0x000056518862fafb in std::string::_Rep::_M_clone (this=0x5651be729860, __alloc=..., __res=)
at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.tcc:1073
#18 0x000056518862fbc0 in std::string::reserve (this=this@entry=0x5652403d01c8, __res=, __res@entry=39)
at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.tcc:951
#19 0x00005651886d09fe in std::string::append (this=this@entry=0x5652403d01c8, __s=0x565192e34458 "rocksdb.block.based.table.index.type",
_n=n@entry=36) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.tcc:747
#20 0x00005651892b5adc in rocksdb::BlockBuilder::Add (this=0x5652403d01c0, key=..., value=..., delta_value=delta_value@entry=0x0)
at /usr/src/debug/ceph-14.2.11/src/rocksdb/include/rocksdb/slice.h:62
#21 0x000056518920ccf0 in rocksdb::PropertyBlockBuilder::Finish (this=this@entry=0x7f33791eec80)
at /usr/src/debug/ceph-14.2.11/src/rocksdb/table/meta_blocks.cc:125
#22 0x00005651892b0db0 in rocksdb::BlockBasedTableBuilder::WritePropertiesBlock(rocksdb::MetaIndexBuilder*) ()
at /usr/src/debug/ceph-14.2.11/src/rocksdb/table/block_based_table_builder.cc:967
#23 0x00005651892b228e in rocksdb::BlockBasedTableBuilder::Finish() ()
at /usr/src/debug/ceph-14.2.11/src/rocksdb/table/block_based_table_builder.cc:1140
#24 0x000056518927a80d in rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::CompactionRangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*) ()
at /usr/src/debug/ceph-14.2.11/src/rocksdb/db/compaction_job.cc:1281
#25 0x000056518927b507 in rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*) ()
at /usr/src/debug/ceph-14.2.11/src/rocksdb/db/compaction_job.cc:975
#26 0x000056518927c6d4 in rocksdb::CompactionJob::Run() () at /usr/src/debug/ceph-14.2.11/src/rocksdb/db/compaction_job.cc:597
#27 0x00005651890e5c99 in rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority) () at /usr/src/debug/ceph-14.2.11/src/rocksdb/db/db_impl_compaction_flush.cc:2623
#28 0x00005651890ec165 in rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority) ()
at /usr/src/debug/ceph-14.2.11/src/rocksdb/db/db_impl_compaction_flush.cc:2191
#29 0x00005651890ec65a in rocksdb::DBImpl::BGWorkCompaction (arg=)
at /usr/src/debug/ceph-14.2.11/src/rocksdb/db/db_impl_compaction_flush.cc:1972
#30 0x00005651892bb79a in operator() (this=0x7f33791f0b60) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:260
---Type to continue, or q to quit---
#31 rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () at /usr/src/debug/ceph-14.2.11/src/rocksdb/util/threadpool_imp.cc:265
#32 0x00005651892bb92d in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x565192c60ae0)
at /usr/src/debug/ceph-14.2.11/src/rocksdb/util/threadpool_imp.cc:306
#33 0x000056518935f90f in execute_native_thread_routine ()
#34 0x00007f3394af1e65 in start_thread () from /lib64/libpthread.so.0
#35 0x00007f3393de58ad in clone () from /lib64/libc.so.6
#20 0x00005651892b5adc in rocksdb::BlockBuilder::Add (this=0x5652403d01c0, key=..., value=..., delta_value=delta_value@entry=0x0)
at /usr/src/debug/ceph-14.2.11/src/rocksdb/include/rocksdb/slice.h:62
62 const char* data() const { return data; }
(gdb) p shared
$15 = 0
(gdb) p key
$16 = (const rocksdb::Slice &) @0x7f33791eebb0: {data = 0x565192e34458 "rocksdb.block.based.table.index.type", size = 36}
#19 0x00005651886d09fe in std::string::append (this=this@entry=0x5652403d01c8, __s=0x565192e34458 "rocksdb.block.based.table.index.type",
__n=__n@entry=36) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.tcc:747
747 /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.tcc: No such file or directory.
(gdb) p *this
(gdb) down
#18 0x000056518862fbc0 in std::string::reserve (this=this@entry=0x5652403d01c8, __res=, __res@entry=39)
at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.tcc:951
951 /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.tcc: No such file or directory.
(gdb) p *this
(gdb) down
#17 0x000056518862fafb in std::string::_Rep::_M_clone (this=0x5651be729860, __alloc=..., __res=)
at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.tcc:1073
1073 in /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.tcc
(gdb) down
#16 std::string::_Rep::_S_create (__capacity=39, _old_capacity=, alloc=...)
at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.tcc:1057
1057 in /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.tcc
(gdb) dpwn
Undefined command: "dpwn". Try "help".
(gdb) down
#15 0x000056518862fab9 in allocate (this=, n=)
at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.tcc:995
995 in /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.tcc
#21 0x000056518920ccf0 in rocksdb::PropertyBlockBuilder::Finish (this=this@entry=0x7f33791eec80)
at /usr/src/debug/ceph-14.2.11/src/rocksdb/table/meta_blocks.cc:125
125 properties_block->Add(prop.first, prop.second);
(gdb) p prop
$16 = (const std::pair<std::basic_string<char, std::char_traits, std::allocator > const, std::basic_string<char, std::char_traits, std::allocator > > &) @0x5651d90c2d40: {<std::pair_base<std::basic_string<char, std::char_traits, std::allocator > const, std::basic_string<char, std::char_traits, std::allocator > >> = {}, first = "rocksdb.block.based.table.index.type",
second = "\000\000\000"}
(gdb) p (BlockBuilder)properties_block
$17 = {block_restart_interval = 1077739968, use_delta_encoding = 82, use_value_delta_encoding = 86, buffer = "",
restarts = std::vector of length -11236540451328, capacity -11236871566520 = {
Dump of assembler code for function tc_newarray:
0x00007f3396e15c80 <+0>: mov 0x211239(%rip),%rax # 0x7f3397026ec0
0x00007f3396e15c87 <+7>: test %rax,%rax
0x00007f3396e15c8a <+10>: jne 0x7f3396e15d19 <tc_newarray+153>
0x00007f3396e15c90 <+16>: mov 0x210001(%rip),%rax # 0x7f3397025c98
0x00007f3396e15c97 <+23>: mov %fs:(%rax),%rcx
0x00007f3396e15c9b <+27>: test %rcx,%rcx
0x00007f3396e15c9e <+30>: je 0x7f3396e15d19 <tc_newarray+153>
0x00007f3396e15ca0 <+32>: cmp $0x400,%rdi
0x00007f3396e15ca7 <+39>: ja 0x7f3396e15d1e <tc_newarray+158>
0x00007f3396e15ca9 <+41>: lea 0x7(%rdi),%eax
0x00007f3396e15cac <+44>: shr $0x3,%eax
0x00007f3396e15caf <+47>: lea 0x3b524a(%rip),%rdx # 0x7f33971caf00
0x00007f3396e15cb6 <+54>: movzbl (%rdx,%rax,1),%esi
0x00007f3396e15cba <+58>: mov %esi,%r8d
0x00007f3396e15cbd <+61>: mov 0x9fc(%rdx,%r8,4),%edx
0x00007f3396e15cc5 <+69>: movslq %edx,%r9
0x00007f3396e15cc8 <+72>: sub %r9,0xc08(%rcx)
0x00007f3396e15ccf <+79>: js 0x7f3396e15d08 <tc_newarray+136>
0x00007f3396e15cd1 <+81>: mov %r8,%rdi
0x00007f3396e15cd4 <+84>: shl $0x5,%rdi
0x00007f3396e15cd8 <+88>: add %rcx,%rdi
0x00007f3396e15cdb <+91>: mov (%rdi),%rax
0x00007f3396e15cde <+94>: test %rax,%rax
0x00007f3396e15ce1 <+97>: je 0x7f3396e15d00 <tc_newarray+128>
=> 0x00007f3396e15ce3 <+99>: mov (%rax),%rsi
0x00007f3396e15ce6 <+102>: mov %rsi,(%rdi)
0x00007f3396e15ce9 <+105>: mov 0x8(%rdi),%esi
0x00007f3396e15cec <+108>: sub $0x1,%esi
0x00007f3396e15cef <+111>: cmp 0xc(%rdi),%esi
0x00007f3396e15cf2 <+114>: mov %esi,0x8(%rdi)
0x00007f3396e15cf5 <+117>: jb 0x7f3396e15d35 <tc_newarray+181>
0x00007f3396e15cf7 <+119>: add %edx,0xc00(%rcx)
0x00007f3396e15cfd <+125>: retq
0x00007f3396e15cfe <+126>: xchg %ax,%ax
0x00007f3396e15d00 <+128>: mov %rcx,%rdi
0x00007f3396e15d03 <+131>: jmpq 0x7f3396df3ae0 _ZN8tcmalloc11ThreadCache21FetchFromCentralCacheEji@plt
0x00007f3396e15d08 <+136>: mov 0xc08(%rcx),%rax
0x00007f3396e15d0f <+143>: add %rax,%r9
0x00007f3396e15d12 <+146>: mov %r9,0xc08(%rcx)
0x00007f3396e15d19 <+153>: jmpq 0x7f3396df4380 _ZN8tcmalloc27allocate_full_cpp_throw_oomEm@plt
0x00007f3396e15d1e <+158>: cmp $0x40000,%rdi
0x00007f3396e15d25 <+165>: ja 0x7f3396e15d19 <tc_newarray+153>
0x00007f3396e15d27 <+167>: lea 0x3c7f(%rdi),%eax
0x00007f3396e15d2d <+173>: shr $0x7,%eax
0x00007f3396e15d30 <+176>: jmpq 0x7f3396e15caf <tc_newarray+47>
---Type to continue, or q to quit---
0x00007f3396e15d35 <+181>: mov %esi,0xc(%rdi)
0x00007f3396e15d38 <+184>: jmp 0x7f3396e15cf7 <tc_newarray+119>
End of assembler dump.
The text was updated successfully, but these errors were encountered: