Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rosbag2_transport test_record can segfault #138

Closed
pbaughman opened this issue Jun 27, 2019 · 6 comments
Closed

rosbag2_transport test_record can segfault #138

pbaughman opened this issue Jun 27, 2019 · 6 comments
Labels
bug Something isn't working

Comments

@pbaughman
Copy link

I observed this failure in CI, and again on my machine. I was able to get a stack trace from GDB by doing

terminal 1> colcon build --packages-up-to rosbag2_transport
terminal 1> source install/setup.bash
terminal 1> while build/rosbag2_transport/test_record; do :; done  # Loop until the test fails, or in this case hangs

terminal 2> sudo gdb -p 1885  # The PID of the running test. . 
(gdb) where

The stack trace is as follows:

#0  0x00007f9c6b11998d in pthread_join (threadid=140308771751680, thread_return=0x0) at pthread_join.c:90
#1  0x00007f9c6a0fcb97 in std::thread::join() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00000000004e7d23 in std::_Mem_fn_base<void (std::thread::*)(), true>::operator()<, void>(std::thread&) const (this=0x7ffeb637a6e8, __object=...) at /usr/include/c++/5/functional:583
#3  0x00000000004d4286 in std::_Mem_fn_base<void (std::thread::*)(), true>::operator()<std::thread, , void>(std::reference_wrapper<std::thread>) const (this=0x7ffeb637a6e8, __ref=...) at /usr/include/c++/5/functional:619
#4  0x00000000004bb211 in std::_Bind_simple<std::_Mem_fn<void (std::thread::*)()> (std::reference_wrapper<std::thread>)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7ffeb637a6e0) at /usr/include/c++/5/functional:1531
#5  0x000000000049dc56 in std::_Bind_simple<std::_Mem_fn<void (std::thread::*)()> (std::reference_wrapper<std::thread>)>::operator()() (this=0x7ffeb637a6e0) at /usr/include/c++/5/functional:1520
#6  0x000000000047d5aa in std::__once_call_impl<std::_Bind_simple<std::_Mem_fn<void (std::thread::*)()> (std::reference_wrapper<std::thread>)> >() () at /usr/include/c++/5/mutex:706
#7  0x00007f9c6b11fa99 in __pthread_once_slow (once_control=0x41f8578, init_routine=0x44ed70 <__once_proxy@plt>) at pthread_once.c:116
#8  0x00000000004501d0 in __gthread_once (__once=0x41f8578, __func=0x44ed70 <__once_proxy@plt>) at /usr/include/x86_64-linux-gnu/c++/5/bits/gthr-default.h:699
#9  0x0000000000467da2 in std::call_once<void (std::thread::*)(), std::reference_wrapper<std::thread> >(std::once_flag&, void (std::thread::*&&)(), std::reference_wrapper<std::thread>&&) (__once=..., __f=<unknown type in /home/pete.baughman/ws/build/rosbag2_transport/test_record, CU 0x0, DIE 0xd4d17>)
    at /usr/include/c++/5/mutex:738
#10 0x0000000000460b90 in std::__future_base::_Async_state_commonV2::_M_join (this=0x41f8550) at /usr/include/c++/5/future:1638
#11 0x0000000000460af4 in std::__future_base::_Async_state_commonV2::_M_complete_async (this=0x41f8550) at /usr/include/c++/5/future:1636
#12 0x000000000045fd11 in std::__future_base::_State_baseV2::wait (this=0x41f8550) at /usr/include/c++/5/future:319
#13 0x0000000000466ecf in std::__basic_future<void>::_M_get_result (this=0x386f240) at /usr/include/c++/5/future:681
#14 0x000000000046096f in std::future<void>::get (this=0x386f240) at /usr/include/c++/5/future:846
#15 0x0000000000461158 in rosbag2_test_common::PublisherManager::run_publishers(std::function<unsigned long (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>) (this=0x2415a08, count_function=...)
    at /home/pete.baughman/ws/install/include/rosbag2_test_common/publisher_manager.hpp:77
#16 0x0000000000463531 in RecordIntegrationTestFixture::run_publishers (this=0x2415910) at path/rosbag2_transport/test/rosbag2_transport/record_integration_fixture.hpp:68
#17 0x0000000000459308 in RecordIntegrationTestFixture_published_messages_from_multiple_topics_are_recorded_Test::TestBody (this=0x2415910) at path/rosbag2_transport/test/rosbag2_transport/test_record.cpp:46
#18 0x00000000005b4e3a in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x2415910, method=&virtual testing::Test::TestBody(), location=0x600c0b "the test body") at /home/pete.baughman/ws/install/src/gtest_vendor/src/gtest.cc:2447
#19 0x00000000005a8a3d in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x2415910, method=&virtual testing::Test::TestBody(), location=0x600c0b "the test body") at /home/pete.baughman/ws/install/src/gtest_vendor/src/gtest.cc:2483
#20 0x000000000055e6ba in testing::Test::Run (this=0x2415910) at /home/pete.baughman/ws/install/src/gtest_vendor/src/gtest.cc:2523
#21 0x000000000055fc4a in testing::TestInfo::Run (this=0x2414220) at /home/pete.baughman/ws/install/src/gtest_vendor/src/gtest.cc:2703
#22 0x0000000000560978 in testing::TestCase::Run (this=0x2415330) at /home/pete.baughman/ws/install/src/gtest_vendor/src/gtest.cc:2825
#23 0x0000000000579e88 in testing::internal::UnitTestImpl::RunAllTests (this=0x2414f70) at /home/pete.baughman/ws/install/src/gtest_vendor/src/gtest.cc:5226
#24 0x00000000005b6a8c in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x2414f70, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x57995a <testing::internal::UnitTestImpl::RunAllTests()>, 
    location=0x601610 "auxiliary test code (environments or event listeners)") at /home/pete.baughman/ws/install/src/gtest_vendor/src/gtest.cc:2447
#25 0x00000000005aaa18 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x2414f70, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x57995a <testing::internal::UnitTestImpl::RunAllTests()>, 
    location=0x601610 "auxiliary test code (environments or event listeners)") at /home/pete.baughman/ws/install/src/gtest_vendor/src/gtest.cc:2483
#26 0x00000000005770f1 in testing::UnitTest::Run (this=0x984680 <testing::UnitTest::GetInstance()::instance>) at /home/pete.baughman/ws/install/src/gtest_vendor/src/gtest.cc:4834
#27 0x000000000054cf86 in RUN_ALL_TESTS () at /home/pete.baughman/ws/install/src/gtest_vendor/include/gtest/gtest.h:2372
#28 0x000000000054ce55 in main (argc=1, argv=0x7ffeb637b2f8) at /home/pete.baughman/ws/install/src/gmock_vendor/src/gmock_main.cc:53

I've got the following threads:

(gdb) info threads                                                                                                                                         
  Id   Target Id         Frame                                                                                                                             
* 1    Thread 0x7f9c6c6caf40 (LWP 1885) "test_record" 0x00007f9c6b11998d in pthread_join (threadid=140308771751680, thread_return=0x0) at pthread_join.c:90
  2    Thread 0x7f9c64076700 (LWP 1886) "test_record" 0x00007f9c6b120827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0,            
    futex_word=0x7f9c6bf09780 <rclcpp::SignalHandler::signal_handler_sem_>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205                             
  3    Thread 0x7f9c63875700 (LWP 1887) "test_record" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185       
  4    Thread 0x7f9c63074700 (LWP 1888) "test_record" 0x00007f9c69b6ba13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84                         
  5    Thread 0x7f9c62873700 (LWP 1889) "test_record" 0x00007f9c6b12194d in recvmsg () at ../sysdeps/unix/syscall-template.S:84                            
  6    Thread 0x7f9c62072700 (LWP 1890) "test_record" 0x00007f9c6b12194d in recvmsg () at ../sysdeps/unix/syscall-template.S:84                            
  7    Thread 0x7f9c61871700 (LWP 1891) "test_record" 0x00007f9c6b12194d in recvmsg () at ../sysdeps/unix/syscall-template.S:84                            
  8    Thread 0x7f9c61070700 (LWP 1892) "test_record" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185       
  9    Thread 0x7f9c53fff700 (LWP 1893) "test_record" 0x00007f9c69b6ba13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84                         
  10   Thread 0x7f9c537fe700 (LWP 1894) "test_record" 0x00007f9c6b12194d in recvmsg () at ../sysdeps/unix/syscall-template.S:84                            
  11   Thread 0x7f9c52ffd700 (LWP 1895) "test_record" 0x00007f9c6b12194d in recvmsg () at ../sysdeps/unix/syscall-template.S:84                            
  12   Thread 0x7f9c527fc700 (LWP 1896) "test_record" 0x00007f9c6b12194d in recvmsg () at ../sysdeps/unix/syscall-template.S:84                            
  13   Thread 0x7f9c515f2700 (LWP 1898) "test_record" 0x00007f9c69b6ba13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84                         
  14   Thread 0x7f9c50df1700 (LWP 1899) "test_record" 0x00007f9c6b12194d in recvmsg () at ../sysdeps/unix/syscall-template.S:84                            
  15   Thread 0x7f9c2ffff700 (LWP 1900) "test_record" 0x00007f9c6b12194d in recvmsg () at ../sysdeps/unix/syscall-template.S:84                            
  16   Thread 0x7f9c2f7fe700 (LWP 1901) "test_record" 0x00007f9c6b12194d in recvmsg () at ../sysdeps/unix/syscall-template.S:84                            
  17   Thread 0x7f9c2e7fc700 (LWP 1903) "test_record" 0x00007f9c6b121c1d in nanosleep () at ../sysdeps/unix/syscall-template.S:84  

I've got version d630f8e right now, because we're not all the way up to date with dashing. I didn't see anything in the changelog or open issues that indicates this has been noticed before. I'm going to try to reproduce this on the nightly docker image next.

@pbaughman
Copy link
Author

When using faa22cc I wasn't able to get it to hang, but I can get it to segfault and drop a core file. The stack trace is pretty deep:

(gdb) where
#0  0x00007f8e9daccd09 in hash_map_find (hash_map=hash_map@entry=0x7f8e9e32f590 <__logger_map>, key=key@entry=0x7f8e975a3cc8, key_hash=key_hash@entry=0x7f8e975a3c88, map_index=map_index@entry=0x7f8e975a3c90,
    bucket_index=bucket_index@entry=0x7f8e975a3c98, entry=entry@entry=0x7f8e975a3ca0) at /home/jenkins-agent/workspace/packaging_linux/ws/src/ros2/rcutils/src/hash_map.c:347
#1  0x00007f8e9dacdc11 in rcutils_hash_map_key_exists (hash_map=hash_map@entry=0x7f8e9e32f590 <__logger_map>, key=key@entry=0x7f8e975a3cc8) at /home/jenkins-agent/workspace/packaging_linux/ws/src/ros2/rcutils/src/hash_map.c:460
#2  0x00007f8e9e1133a7 in rcl_logging_rosout_init_publisher_for_node (node=node@entry=0x7f8e80000f90) at /home/jenkins-agent/workspace/packaging_linux/ws/src/ros2/rcl/rcl/src/rcl/logging_rosout.c:156
#3  0x00007f8e9e1158c2 in rcl_node_init (node=node@entry=0x7f8e80000f90, name=0x7f8e975ac770 "_rosbag2", namespace_=<optimized out>, context=0x55df91682e80, options=options@entry=0x7f8e80000d70)
    at /home/jenkins-agent/workspace/packaging_linux/ws/src/ros2/rcl/rcl/src/rcl/node.c:351
#4  0x00007f8e9e3de5f2 in rclcpp::node_interfaces::NodeBase::NodeBase (this=0x7f8e80000e30, node_name="_rosbag2", namespace_="", context=..., rcl_node_options=..., use_intra_process_default=<optimized out>)
    at /home/jenkins-agent/workspace/packaging_linux/ws/src/ros2/rclcpp/rclcpp/src/rclcpp/node_interfaces/node_base.cpp:66
#5  0x00007f8e9e3d7f3e in rclcpp::Node::Node (this=0x7f8e80000b30, node_name="_rosbag2", namespace_="", options=...) at /home/jenkins-agent/workspace/packaging_linux/ws/src/ros2/rclcpp/rclcpp/src/rclcpp/node.cpp:116
#6  0x00007f8e9e3d9b21 in rclcpp::Node::Node (this=0x7f8e80000b30, node_name="_rosbag2", options=...) at /home/jenkins-agent/workspace/packaging_linux/ws/src/ros2/rclcpp/rclcpp/src/rclcpp/node.cpp:103
#7  0x00007f8e9e8a5c39 in rosbag2_transport::Rosbag2Node::Rosbag2Node (this=0x7f8e80000b30, node_name="_rosbag2")
    at /home/jenkins-agent/workspace/packaging_linux/ws/src/ros2/rosbag2/rosbag2_transport/src/rosbag2_transport/rosbag2_node.cpp:33
#8  0x00007f8e9e8aa1a7 in __gnu_cxx::new_allocator<rosbag2_transport::Rosbag2Node>::construct<rosbag2_transport::Rosbag2Node, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (
    this=<optimized out>, __p=<optimized out>) at /usr/include/c++/7/ext/new_allocator.h:136
#9  std::allocator_traits<std::allocator<rosbag2_transport::Rosbag2Node> >::construct<rosbag2_transport::Rosbag2Node, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (__a=...,
    __p=<optimized out>) at /usr/include/c++/7/bits/alloc_traits.h:475
#10 std::_Sp_counted_ptr_inplace<rosbag2_transport::Rosbag2Node, std::allocator<rosbag2_transport::Rosbag2Node>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (__a=..., this=0x7f8e80000b20) at /usr/include/c++/7/bits/shared_ptr_base.h:526
#11 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<rosbag2_transport::Rosbag2Node, std::allocator<rosbag2_transport::Rosbag2Node>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >    (__a=..., this=<optimized out>) at /usr/include/c++/7/bits/shared_ptr_base.h:637
#12 std::__shared_ptr<rosbag2_transport::Rosbag2Node, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<rosbag2_transport::Rosbag2Node>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (
    __a=..., __tag=..., this=<optimized out>) at /usr/include/c++/7/bits/shared_ptr_base.h:1295
#13 std::shared_ptr<rosbag2_transport::Rosbag2Node>::shared_ptr<std::allocator<rosbag2_transport::Rosbag2Node>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (__a=..., __tag=...,
    this=<optimized out>) at /usr/include/c++/7/bits/shared_ptr.h:344
#14 std::allocate_shared<rosbag2_transport::Rosbag2Node, std::allocator<rosbag2_transport::Rosbag2Node>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (__a=...)
    at /usr/include/c++/7/bits/shared_ptr.h:691
#15 std::make_shared<rosbag2_transport::Rosbag2Node, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > () at /usr/include/c++/7/bits/shared_ptr.h:707
#16 rosbag2_transport::Rosbag2Transport::setup_node (this=this@entry=0x7f8e975ad090, node_prefix="")
    at /home/jenkins-agent/workspace/packaging_linux/ws/src/ros2/rosbag2/rosbag2_transport/src/rosbag2_transport/rosbag2_transport.cpp:82
#17 0x00007f8e9e8aac86 in rosbag2_transport::Rosbag2Transport::record (this=0x7f8e975ad090, storage_options=..., record_options=...)
    at /home/jenkins-agent/workspace/packaging_linux/ws/src/ros2/rosbag2/rosbag2_transport/src/rosbag2_transport/rosbag2_transport.cpp:70
#18 0x000055df8f8bb427 in RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}::operator()() const ()
#19 0x000055df8f8cf69d in void std::__invoke_impl<void, RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}>(std::__invoke_other, RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}&&) ()
#20 0x000055df8f8bfd6d in std::__invoke_result<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}>::type std::__invoke<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}>(std::__invoke_result&&, (RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}&&)...) ()
#21 0x000055df8f925468 in decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker<std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#22 0x000055df8f924f8a in std::thread::_Invoker<std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}> >::operator()() ()
#23 0x000055df8f9246a8 in std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}> >, void>::operator()() const ()
#24 0x000055df8f923582 in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}> >, void> >::_M_invoke(std::_Any_data const&) ()
#25 0x000055df8f8bdcfa in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const ()
#26 0x000055df8f8b9e98 in std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) ()
#27 0x000055df8f8df5e3 in void std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) ()
#28 0x000055df8f8cbfec in std::__invoke_result<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>::type std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) ()
#29 0x000055df8f8bda22 in void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const ()
#30 0x000055df8f8bda85 in void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const ()
#31 0x000055df8f8bda96 in void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() ()
#32 0x00007f8e9def1827 in __pthread_once_slow (once_control=0x55df91658b68, init_routine=0x7f8e9d2db830 <__once_proxy>) at pthread_once.c:116
#33 0x000055df8f8b157a in __gthread_once(int*, void (*)()) ()
#34 0x000055df8f8bdb3e in void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) ()
#35 0x000055df8f8b9c3c in std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) ()
#36 0x000055df8f9205a2 in std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}> >, void>::_Async_state_impl(std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}>&&)::{lambda()#1}::operator()() const ()
#37 0x000055df8f92379d in void std::__invoke_impl<void, std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}> >, void>::_Async_state_impl(std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}>&&)::{lambda()#1}>(std::__invoke_other, std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}> >, void>::_Async_state_impl(std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}>&&)::{lambda()#1}&&) ()
#38 0x000055df8f92213d in std::__invoke_result<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}> >, void>::_Async_state_impl(std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}>&&)::{lambda()#1}>::type std::__invoke<std::__future_base::_Async_state_impl<std::thread::_Invoker<s
td::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}> >, void>::_Async_state_impl(std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptionsconst&)::{lambda()#1}>&&)::{lambda()#1}>(std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}> >, void>::_Async_state_impl(std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}>&&)::{lambda()#1}&&, (std::__invoke_result&&)...) ()
#39 0x000055df8f92ee0e in decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}> >, void>::_Async_state_impl(std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}>&&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#40 0x000055df8f92d7d4 in std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}
> >, void>::_Async_state_impl(std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}>&&)::{lambda()#1}> >::operator()() ()
#41 0x000055df8f929592 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}> >, void>::_Async_state_impl(std::tuple<RecordIntegrationTestFixture::start_recording(rosbag2_transport::RecordOptions const&)::{lambda()#1}>&&)::{lambda()#1}> > >::_M_run() ()
#42 0x00007f8e9d2dc66f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#43 0x00007f8e9dee96db in start_thread (arg=0x7f8e975ae700) at pthread_create.c:463
#44 0x00007f8e9cd3788f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Here are the threads when it segfaults

(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x7f8e975ae700 (LWP 25314) 0x00007f8e9daccd09 in hash_map_find (hash_map=hash_map@entry=0x7f8e9e32f590 <__logger_map>, key=key@entry=0x7f8e975a3cc8, key_hash=key_hash@entry=0x7f8e975a3c88,
    map_index=map_index@entry=0x7f8e975a3c90, bucket_index=bucket_index@entry=0x7f8e975a3c98, entry=entry@entry=0x7f8e975a3ca0) at /home/jenkins-agent/workspace/packaging_linux/ws/src/ros2/rcutils/src/hash_map.c:347
  2    Thread 0x7f8e96dad700 (LWP 25315) 0x00007f8e9cd37bb7 in epoll_wait (epfd=11, events=events@entry=0x7f8e96dabf80, maxevents=maxevents@entry=128, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  3    Thread 0x7f8e97daf700 (LWP 25313) 0x00007f8e9def26d6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f8e9e675900 <rclcpp::SignalHandler::signal_handler_sem_>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  4    Thread 0x7f8e9ecae600 (LWP 25312) rclcpp::Service<rcl_interfaces::srv::GetParameters>::Service (this=0x55df923c41b0, node_handle=warning: RTTI symbol not found for class 'std::_Sp_counted_deleter<rcl_node_t*, rclcpp::node_interfaces::NodeBase::NodeBase(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<c
har>, std::allocator<char> > const&, std::shared_ptr<rclcpp::Context>, rcl_node_options_t const&, bool)::{lambda(rcl_node_t*)#2}, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_deleter<rcl_node_t*, rclcpp::node_interfaces::NodeBase::NodeBase(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<rclcpp::Context>, rcl_node_options_t const&, bool)::{lambda(rcl_node_t*)#2}, std::a
llocator<void>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr<rcl_node_t> (use count 2, weak count 0) = {...},
    service_name="publisher2/get_parameters", any_callback=..., service_options=...) at /home/jenkins-agent/workspace/packaging_linux/ws/src/ros2/rclcpp/rclcpp/include/rclcpp/service.hpp:103
  5    Thread 0x7f8e955aa700 (LWP 25318) 0x00007f8e9def4567 in __libc_recvmsg (fd=fd@entry=8, msg=msg@entry=0x7f8e955a9240, flags=flags@entry=0) at ../sysdeps/unix/sysv/linux/recvmsg.c:28
  6    Thread 0x7f8e8d7ac700 (LWP 25324) 0x00007f8e9cd37bb7 in epoll_wait (epfd=31, events=events@entry=0x7f8e8d7aaf80, maxevents=maxevents@entry=128, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  7    Thread 0x7f8e8f7b0700 (LWP 25320) 0x00007f8e9cd37bb7 in epoll_wait (epfd=21, events=events@entry=0x7f8e8f7aef80, maxevents=maxevents@entry=128, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  8    Thread 0x7f8e8efaf700 (LWP 25321) 0x00007f8e9def4567 in __libc_recvmsg (fd=fd@entry=16, msg=msg@entry=0x7f8e8efae240, flags=flags@entry=0) at ../sysdeps/unix/sysv/linux/recvmsg.c:28
  9    Thread 0x7f8e8e7ae700 (LWP 25322) clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:78
  10   Thread 0x7f8e95dab700 (LWP 25317) 0x00007f8e9def4567 in __libc_recvmsg (fd=fd@entry=7, msg=msg@entry=0x7f8e95daa240, flags=flags@entry=0) at ../sysdeps/unix/sysv/linux/recvmsg.c:28
  11   Thread 0x7f8e94da9700 (LWP 25319) 0x00007f8e9def4607 in __libc_sendmsg (fd=fd@entry=9, msg=msg@entry=0x7f8e94da7c10, flags=flags@entry=16384) at ../sysdeps/unix/sysv/linux/sendmsg.c:28
  12   Thread 0x7f8e965ac700 (LWP 25316) 0x00007f8e988b7c1b in eprosima::fastrtps::rtps::GuidPrefix_t::operator== (prefix=..., this=0x7f8e7c062a40)
    at /home/jenkins-agent/workspace/packaging_linux/ws/src/eProsima/Fast-RTPS/include/fastrtps/rtps/reader/../common/Guid.h:108
  13   Thread 0x7f8e8dfad700 (LWP 25323) 0x00007f8e9def4567 in __libc_recvmsg (fd=fd@entry=18, msg=msg@entry=0x7f8e8dfac240, flags=flags@entry=0) at ../sysdeps/unix/sysv/linux/recvmsg.c:28
  14   Thread 0x7f8e8cfab700 (LWP 25325) clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:78
  15   Thread 0x7f8e6ffff700 (LWP 25326) clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:78
  16   Thread 0x7f8e6f7fe700 (LWP 25327) clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:78

@pbaughman pbaughman changed the title Observed rosbag2_transport test_record hanging Observed rosbag2_transport test_record can segfault Jun 28, 2019
@pbaughman pbaughman changed the title Observed rosbag2_transport test_record can segfault rosbag2_transport test_record can segfault Jun 28, 2019
@pbaughman
Copy link
Author

I can get it to segfault pretty easily by adding some CPU stress at the same time the test is running. Run the test in a loop in one terminal, and then in a second terminal run

stress -c 12

In another terminal. The test will segfault within a minute or two

thomas-moulard pushed a commit to thomas-moulard/rosbag2 that referenced this issue Oct 22, 2019
* ros2GH-138 Move calculation of bag size

- previously in rosbag2::Info
- now in storage plugin

* ros2GH-130 Add rosbag2_bag_v2_plugins package

-This package will contain storage and converter plugins

* ros2GH-131 don't build plugins on Windows

* ros2GH-129 Add function to be generated

- massive if/else between all message types
- will be generated similar to ros1_bridge plugin

* ros2GH-138 Write storage plugin for rosbag v2 bags

* ros2GH-138 Make sure that no attempt to create a converter is made when trying to read a rsbag v2 bag file

* ros2GH-138 Add play end-to-end test for rosbag v2 plugin

* ros2GH-138 Use cmake files to find ros1 packages

- Use files from ros1_bridge via PkgConfig

* ros2GH-138 Add generator code

* ros2GH-141 Add initial version of vendor package

* ros2GH-141 Improve vendor package to build on Mac

* ros2GH-138 Cleanup CMakeLists

* ros2GH-141 Use unmanaged Instance of class-loader

- managed instance somehow isn't available for gcc 6.3

* ros2GH-141 Reduce patch and copy new toplevle CMakeLists by hand

* ros2GH-141 Fix Shared Instance usage

* ros2GH-141 Improve maintainability of vendor package

- Document what patches do and why changes are necessary
- Load ros1 packages through cmake macro
- Do not export ros1 packages via ament
- use commit hash of current master which is more stable than using melodic-devel

* ros2GH-138 Link against rclcpp - necessary for ros1_bridge

* ros2GH-138 Avoid crash when trying to play v2 bags which contain unknown message types

* ros2GH-138 Add CLI -s <storage_id> option to ros2 bag info and use it in rosbag2::info

- this allows ros2 bag info to work also when the yaml metadata file does not exsist
- this is always the case for rosbag1 bagfiles
- it could also happen for sqlite or other storage based bagfiles

* ros2GH-138 Add end-to-end info test for rosbag v2 files

* ros2GH-138 Add unit tests to rosbag_storage

* ros2GH-138 Add method to extract filename from path to FilesystemHelpers

* ros2GH-138 Add proper logging for topics which cannot be converted

* ros2GH-138 Improve finding dependencies of ros1

* ros2GH-141 Explicitly import transitive dependencies of vendor package

* ros2GH-138 Skip tests via ament if ros1 is not available

* ros2GH-133 First split of plugins

* ros2GH-133 Write serialized rosbag message

* ros2GH-133 Improve converter plugin

- move generation templates outside of plugin folders as both
  plugins need it
- use ros::serialization routines to deserialize the ros message

* ros2GH-133 Add plugin to be found by pluginlib

* ros2GH-133 Remove empty check in converter

- With the rosbag_v2_converter_plugin, we don't need to treat
  rosbag_v2 storage any different

* ros2GH-133 Assert serialization format in unit tests for storage

* ros2GH-133 Delete superfluous include folder

- Only needed if we want to link against the library

* ros2GH-133 get_all_topics_and_types returns only valid ros2 types

- This is necessary as the information is used by rosbag2_transport
- ros2 bag info still shows all topics and types
- rosbag::View::getConnections() can return multiple connections corresponding to the same topic

* ros2GH-133 Improve end to end test

- use a bagfile with messages not known to ros2

* ros2GH-133 Reformulate info message in case of missing ros1-ros2 mapping for a topic

* ros2GH-14 Find messages first

* Explicitly print message when on Windows

Co-Authored-By: Martin-Idel-SI <external.Martin.Idel@bosch-si.com>

* ros2GH-14 Refactor rosbag_storage vendor package

- Improve toplevel CMakeLists
- Put all patches into a resource subfolder

* ros2GH-14 Reflect renames of converter interfaces

* ros2GH-156 Workaround for path problems

* ros2GH-156 Add documentation for plugin

* ros2GH-156 Fix the pluginlib version to greater 2

* ros2GH-156 Prohibit CMake from declaring paths as system paths

This switches the order of ros2 and ros1 directories
resulting in build failures

* ros2GH-156 Prohibit system include paths for rosbag plugins

This can lead to switching ros1 and ros2 include paths resulting
in missing symbols as the wrong pluginlib gets included

* ros2GH-14 Split patches

* make README more verbose

* add plugin specific readme

* more readme for bag_v2 plugin
@emersonknapp emersonknapp added the bug Something isn't working label Jan 29, 2020
@emersonknapp
Copy link
Collaborator

Putting on backlog to see if we can reproduce this now:
suggestions, run with ASAN/TSAN

@ivanpauno ivanpauno added bug Something isn't working and removed bug Something isn't working labels Feb 6, 2020
@rotu
Copy link
Contributor

rotu commented Mar 24, 2020

Seems to be the same as ros2/rmw_cyclonedds#118

@Karsten1987
Copy link
Collaborator

Looking at the strack trace, I consider this ticket being the same as #329
I think this is being addressed in #338 and thus I am going to close this ticket.

@pbaughman please feel free to re-open if you still encounter this issue.

@rotu
Copy link
Contributor

rotu commented Apr 1, 2020

I agree the segfault is fixed. It looks like the original hang might still be out there - I don't know what the cause of that was.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants