Segfault when singleton_object_pool reallocs

[wrwilliams]

From @mwkrentel:

Thread 1, where gdb says the segfault happened.


#0  0x00007fe01a8d9272 in malloc_consolidate () from /lib64/libc.so.6
#1  0x00007fe01a8dc405 in _int_malloc () from /lib64/libc.so.6
#2  0x00007fe01a8dd626 in calloc () from /lib64/libc.so.6
#3  0x00007fe01a8d222d in open_memstream () from /lib64/libc.so.6
#4  0x00007fe01a947c8b in __vsyslog_chk () from /lib64/libc.so.6
#5  0x00007fe01a8d378e in __libc_message () from /lib64/libc.so.6
#6  0x00007fe01a8d9166 in malloc_printerr () from /lib64/libc.so.6
#7  0x00007fe01a8dcf1f in _int_malloc () from /lib64/libc.so.6
#8  0x00007fe01a8dd991 in malloc () from /lib64/libc.so.6
#9  0x00007fe01b558f88 in operator new(unsigned long) () at
../../../../libstdc++-v3/libsupc++/new_op.cc:50
#10 0x00007fe01cf938ed in
_ZNSt8_Rb_treeIPN7Dyninst8ParseAPI5BlockESt4pairIKS3_S3_ESt10_Select1stIS6_ESt4lessIS3_ESaIS6_EE22_M_emplace_hint_uniqueIJRKSt21piecewise_construct_tSt5tupleIJRS5_EESH_IJEEEEESt17_Rb_tree_iteratorIS6_ESt23_Rb_tree_const_iteratorIS6_EDpOT_.isra.225
()
     at
/opt/apps/software/Core/GCC/6.2.0/include/c++/6.2.0/ext/new_allocator.h:104
#11 0x00007fe01cf93bcc in
Dyninst::ParseAPI::LoopAnalyzer::LoopAnalyzer(Dyninst::ParseAPI::Function
const*) ()
     at
/opt/apps/software/Core/GCC/6.2.0/include/c++/6.2.0/bits/stl_map.h:483
And thread 2 inside InstructionAPI.

#0  Dyninst::InstructionAPI::Instruction::~Instruction() ()
     at
/dascratch/krentel/cilk/install/boost/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:158
#1  0x00007fe01cbd47f1 in
boost::detail::sp_counted_impl_pd<Dyninst::InstructionAPI::Instruction*,
PoolDestructor<Dyninst::InstructionAPI::Instruction> >::dispose() ()
     at
/dascratch/krentel/cilk/externals/BUILD-bill/symtabAPI/dyninst/instructionAPI/src/../../common/src/singleton_object_pool.h:120
#2  0x0000000000402eeb in std::_Rb_tree<unsigned long,
std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> >,
std::_Select1st<std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> > >,
std::less<unsigned long>, std::allocator<std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> > >
 >::_M_erase(std::_Rb_tree_node<std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> > >*) ()
     at
/dascratch/krentel/cilk/install/boost/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:146
#3  0x0000000000402ec7 in std::_Rb_tree<unsigned long,
std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> >,
std::_Select1st<std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> > >,
std::less<unsigned long>, std::allocator<std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> > >
 >::_M_erase(std::_Rb_tree_node<std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> > >*) () at
/opt/apps/software/Core/GCC/6.2.0/include/c++/6.2.0/bits/stl_tree.h:1613
#4  0x0000000000402ec7 in std::_Rb_tree<unsigned long,
std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> >,
std::_Select1st<std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> > >,
std::less<unsigned long>, std::allocator<std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> > >
 >::_M_erase(std::_Rb_tree_node<std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> > >*) () at
/opt/apps/software/Core/GCC/6.2.0/include/c++/6.2.0/bits/stl_tree.h:1613
#5  0x0000000000402ec7 in std::_Rb_tree<unsigned long,
std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> >,
std::_Select1st<std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> > >,
std::less<unsigned long>, std::allocator<std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> > >
 >::_M_erase(std::_Rb_tree_node<std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> > >*) () at
/opt/apps/software/Core/GCC/6.2.0/include/c++/6.2.0/bits/stl_tree.h:1613
#6  0x0000000000402ec7 in std::_Rb_tree<unsigned long,
std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> >,
std::_Select1st<std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> > >,
std::less<unsigned long>, std::allocator<std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> > >
 >::_M_erase(std::_Rb_tree_node<std::pair<unsigned long const,
boost::shared_ptr<Dyninst::InstructionAPI::Instruction> > >*) () at
/opt/apps/software/Core/GCC/6.2.0/include/c++/6.2.0/bits/stl_tree.h:1613
#7  0x000000000040263b in doBlock(Dyninst::ParseAPI::Block*,
std::map<Dyninst::ParseAPI::Block*, bool,
std::less<Dyninst::ParseAPI::Block*>,
std::allocator<std::pair<Dyninst::ParseAPI::Block* const, bool> > >&,
FuncInfo&) ()
     at
/opt/apps/software/Core/GCC/6.2.0/include/c++/6.2.0/bits/stl_tree.h:873
#8  0x00000000004026d8 in doLoop(Dyninst::ParseAPI::Loop*,
std::map<Dyninst::ParseAPI::Block*, bool,
std::less<Dyninst::ParseAPI::Block*>,
std::allocator<std::pair<Dyninst::ParseAPI::Block* const, bool> > >&,
FuncInfo&) () at cilk-parse.cpp:238

[wrwilliams]

We're likely running into issues with unsafe memory pools, but this could also be a libc-level race.

[mwkrentel]

I updated my cilk-parse unit test to call parseFunctionRanges() and
parseLineInformation() for each module in the serial part of the
program. AFAIK, this pre-computes all of the Symtab info that the
test uses.

This is available in the dyninst subdirectory of the hpctoolkit-tests
repository on github.

git clone https://github.com/hpctoolkit/hpctoolkit-tests

I also added command-line options to disable selected parts of the
instruction processing.

-I, -Iall do not split basic blocks into instructions
-Iinline do not compute inline callsite sequences
-Iline do not compute line map info

I've tried this with both the v9.3.x and Bill's parallel-parsing
branches. With both branches, the test falls over with a segfault,
illegal instruction, failed assert or glibc memory corruption. (This
suggests the real problem is memory corruption and that the segfault
happens later.)

It segfaults when it splits a basic block into a map of instructions,
even if it does nothing with the instructions (no line map queries, no
inline seqn queries).

    map <Offset, Instruction::Ptr> imap;
    block->getInsns(imap);

You can run in this config with the following options for cilk-parse
to analyze itself with 4 threads.

./cilk-parse -Iline -Iinline cilk-parse 4

I've even seen segfaults with getInsns() turned off (option -I), both
with v9.3.x and parallel-parsing branches. It's pretty rare and
requires a large, static binary, but it does happen. But this could
easily be an entirely different segfault, not related to threads.
I don't know yet.

Right now, the segfault in instruction parsing is a blocker to doing
anything with threads.

Also, I noticed that parseFunctionRanges() is not idempotent. It
fails an assert if you call it twice. That seems pretty fragile to
me.

[wrwilliams]

Obviated by use of pooled allocators on new_parallel_parsing.

[aalmamuncse]

Hi, I am trying to compile the example codes given in dyninstAPI.pdf (for dyninst version 10.1), on Appendix section. First example (instrumenting a function, example 1.1, page 67), I get segmentation fault error when I compile the example. I am trying to instrument a very simple hello.cc file with a hello function. Could you please point any direction to get rid of this error?

[wrwilliams]

@aalmamuncse You're probably going to want to open a new issue for this one with a stack trace from the segfault; this particular issue shouldn't be relevant to your crash (unless things have changed greatly in the year since I left the project).