SimbodyLink::GetWorldAngularVel and others are not thread safe #918

osrf-migration · 2013-10-23T20:43:46Z

Original report (archived issue) by John Hsu (Bitbucket: hsu, GitHub: hsu).

osrf-migration · 2013-10-23T21:21:12Z

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

137: [ RUN      ] PhysicsEngines/SpeedTest.ShapesWorld/2
137: 
137: �[1;32mMsg�[0m Waiting for master�[1;32mMsg�[0m Connected to gazebo master @ http://127.0.0.1:11345
137: �[1;32mMsg�[0m Publicized address: 192.168.1.19
137: �[1;31mError [RenderEngine.cc:636]�[0m Can't open display: 
137: �[1;33mWarning [RenderEngine.cc:92]�[0m Unable to create X window. Rendering will be disabled
137: terminate called after throwing an instance of 'SimTK::Exception::StageTooLow'
137:   what():  SimTK Exception thrown at State.cpp:2069:
137:   Expected stage to be at least Position in StateImpl::getCacheEntry() but current stage was Position
27/34 Test #137: INTEGRATION_speed ......................***Exception: Other 80.01 sec

osrf-migration · 2013-11-06T00:45:39Z

Original comment by John Hsu (Bitbucket: hsu, GitHub: hsu).

Can't get it to fail on default branch (10940:564a780870d88cfc309463e28c957ed650e77b0e) after 30 iterations and counting:

./test/integration/INTEGRATION_speed --gtest_filter=*SpeedTest.ShapesWorld/2 --gtest_repeat=-1

osrf-migration · 2013-11-06T00:47:49Z

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

Sometimes I see failures more when running the executable multiple times, rather than using --gtest_repeat. Also, it didn't have a display, so I don't know if that matters as well.

osrf-migration · 2013-11-06T17:25:39Z

Original comment by John Hsu (Bitbucket: hsu, GitHub: hsu).

tried with display off

DISPLAY=:100 ../build_release/test/integration/INTEGRATION_speed --gtest_filter="*SpeedTest.ShapesWorld/2"

test fails:

Note: Google Test filter = *SpeedTest.ShapesWorld/2
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from PhysicsEngines/SpeedTest
[ RUN      ] PhysicsEngines/SpeedTest.ShapesWorld/2
Msg Waiting for master
Msg Connected to gazebo master @ http://127.0.0.1:12349
Msg Publicized address: 192.168.1.162
Error [RenderEngine.cc:636] Can't open display: :100
Warning [RenderEngine.cc:92] Unable to create X window. Rendering will be disabled
Dbg ServerFixture load in 0.03 seconds, timeout after 60 seconds
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >'
  what():  boost::lock_error
Aborted (core dumped)

got a core file:

(gdb) bt
#0  0x00007fbeaeb0a425 in __GI_raise (sig=<optimized out>)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007fbeaeb0db8b in __GI_abort () at abort.c:91
#2  0x00007fbeaf45d69d in __gnu_cxx::__verbose_terminate_handler() ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007fbeaf45b846 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007fbeaf45b873 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007fbeaf45b96e in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00000000004d2734 in boost::throw_exception<boost::lock_error> (e=...)
    at /usr/include/boost/throw_exception.hpp:61
#7  0x00000000004cd39b in boost::mutex::lock (this=0x7fbe301fa370)
    at /usr/include/boost/thread/pthread/mutex.hpp:55
#8  0x00000000004d2d38 in boost::unique_lock<boost::mutex>::lock (this=0x7fbe9488da90)
    at /usr/include/boost/thread/locks.hpp:412
#9  0x00000000004d2821 in boost::unique_lock<boost::mutex>::unique_lock (this=0x7fbe9488da90, 
    m_=...) at /usr/include/boost/thread/locks.hpp:290
#10 0x00007fbeb0d96f28 in gazebo::transport::Publisher::SendMessage (this=0x7fbe301fa330)
    at /home/hsu/projects/gazebo_default_24/gazebo/transport/Publisher.cc:163
#11 0x00007fbeb0d84d71 in gazebo::transport::Node::ProcessPublishers (this=0x7fbe3028b0f0)
    at /home/hsu/projects/gazebo_default_24/gazebo/transport/Node.cc:140
#12 0x00007fbeb0d99ff2 in gazebo::transport::TopicManager::ProcessNodes (this=0x78c3e0, 
    _onlyOut=false) at /home/hsu/projects/gazebo_default_24/gazebo/transport/TopicManager.cc:155
#13 0x00007fbeb0d74653 in gazebo::transport::ConnectionManager::RunUpdate (this=0x78c580)
    at /home/hsu/projects/gazebo_default_24/gazebo/transport/ConnectionManager.cc:279
#14 0x00007fbeb0d74807 in gazebo::transport::ConnectionManager::Run (this=0x78c580)
    at /home/hsu/projects/gazebo_default_24/gazebo/transport/ConnectionManager.cc:310
#15 0x00007fbeb0d80a3a in boost::_mfi::mf0<void, gazebo::transport::ConnectionManager>::operator() (
    this=0x7fbe8858cb48, p=0x78c580) at /usr/include/boost/bind/mem_fn_template.hpp:49
#16 0x00007fbeb0d7fdc4 in boost::_bi::list1<boost::_bi::value<gazebo::transport::ConnectionManager*> >::operator()<boost::_mfi::mf0<void, gazebo::transport::ConnectionManager>, boost::_bi::list0> (
    this=0x7fbe8858cb58, f=..., a=...) at /usr/include/boost/bind/bind.hpp:253
#17 0x00007fbeb0d7f54f in boost::_bi::bind_t<void, boost::_mfi::mf0<void, gazebo::transport::ConnectionManager>, boost::_bi::list1<boost::_bi::value<gazebo::transport::ConnectionManager*> > >::operator() (this=0x7fbe8858cb48) at /usr/include/boost/bind/bind_template.hpp:20
#18 0x00007fbeb0da7ffc in boost::detail::thread_data<boost::_bi::bind_t<void, boost::_mfi::mf0<void, gazebo::transport::ConnectionManager>, boost::_bi::list1<boost::_bi::value<gazebo::transport::ConnectionManager*> > > >::run (this=0x7fbe8858c9c0) at /usr/include/boost/thread/detail/thread.hpp:61
#19 0x00007fbeafd48ce9 in thread_proxy () from /usr/lib/libboost_thread.so.1.46.1
#20 0x00007fbeb0a3fe9a in start_thread (arg=0x7fbe9488e700) at pthread_create.c:308
#21 0x00007fbeaebc83fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#22 0x0000000000000000 in ?? ()
(gdb)

Looks like Publisher::SendMessage() might be called multiple times.

osrf-migration · 2013-11-15T20:04:53Z

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

New failure in Jenkins

141: [ RUN      ] PhysicsEngines/SpeedTest.ShapesWorld/2
141: 
141: �[1;32mMsg�[0m Waiting for master�[1;32mMsg�[0m Connected to gazebo master @ http://127.0.0.1:11345
141: �[1;32mMsg�[0m Publicized address: 192.168.1.45
141: �[1;31mError [RenderEngine.cc:636]�[0m Can't open display: 
141: �[1;33mWarning [RenderEngine.cc:92]�[0m Unable to create X window. Rendering will be disabled
141: �[1;36mDbg�[0m ServerFixture load in 0.02 seconds, timeout after 60 seconds
141: terminate called after throwing an instance of 'SimTK::Exception::CacheEntryOutOfDate'
141:   what():  SimTK Exception thrown at State.cpp:2080:
141:   State Cache entry was out of date at Stage Time. This entry depends on version 5 of Stage Position but was last updated at version 3.
27/38 Test #141: INTEGRATION_speed .......................***Exception: Other140.89 sec

osrf-migration · 2013-11-19T18:40:04Z

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

I just caught a core file in my overnight testing.

155: [ RUN      ] PhysicsEngines/SpeedTest.ShapesWorld/2
155: 
155: Msg Waiting for masterMsg Connected to gazebo master @ http://127.0.0.1:11345
155: Msg Publicized address: 192.168.1.187
155: Dbg ServerFixture load in 0.09 seconds, timeout after 60 seconds
155: terminate called after throwing an instance of 'SimTK::Exception::CacheEntryOutOfDate'
155:   what():  SimTK Exception thrown at State.cpp:2080:
155:   State Cache entry was out of date at Stage Position. This entry depends on version 11 of Stage Position but was last updated at version 9.

Failing thread:

Thread 27 (Thread 0x2b051aa86700 (LWP 21125)):
#0  nelt (this=0x2b05c499d7f0) at /tmp/buildd/simbody-3.3/SimTKcommon/BigMatrix/src/MatrixHelperRep.h:401
#1  SimTK::MatrixHelper<double>::nelt (this=0x2b05c5eba288) at /tmp/buildd/simbody-3.3/SimTKcommon/BigMatrix/src/MatrixHelper.cpp:501
#2  0x00002b050187b9f3 in nelt (this=0x2b05c5eba288) at /tmp/buildd/simbody-3.3/SimTKcommon/BigMatrix/include/SimTKcommon/internal/BigMatrix.h:299
#3  size (this=0x2b05c5eba288) at /tmp/buildd/simbody-3.3/SimTKcommon/BigMatrix/include/SimTKcommon/internal/BigMatrix.h:1414
#4  SimbodyMatterSubsystemRep::calcTreeAccelerations (this=this@entry=0x2b05c6189f90, s=..., mobilityForces=..., bodyForces=..., presUDots=..., netHingeForces=..., allZ=..., allZPlus=..., A_GB=..., udot=..., 
    qdotdot=..., tau=...) at /tmp/buildd/simbody-3.3/Simbody/src/SimbodyMatterSubsystemRep.cpp:5014
#5  0x00002b050187bef1 in SimbodyMatterSubsystemRep::calcTreeForwardDynamicsOperator (this=this@entry=0x2b05c6189f90, s=..., mobilityForces=..., particleForces=..., bodyForces=..., 
    extraMobilityForces=extraMobilityForces@entry=0x0, extraBodyForces=extraBodyForces@entry=0x0, tac=..., udot=..., qdotdot=..., udotErr=...)
    at /tmp/buildd/simbody-3.3/Simbody/src/SimbodyMatterSubsystemRep.cpp:4737
#6  0x00002b0501884929 in SimbodyMatterSubsystemRep::calcLoopForwardDynamicsOperator (this=this@entry=0x2b05c6189f90, s=..., mobilityForces=..., particleForces=..., bodyForces=..., tac=..., cac=..., udot=..., 
    qdotdot=..., multipliers=..., udotErr=...) at /tmp/buildd/simbody-3.3/Simbody/src/SimbodyMatterSubsystemRep.cpp:4810
#7  0x00002b0501884c8d in SimbodyMatterSubsystemRep::realizeLoopForwardDynamics (this=0x2b05c6189f90, s=..., mobilityForces=..., particleForces=..., bodyForces=...)
    at /tmp/buildd/simbody-3.3/Simbody/src/SimbodyMatterSubsystemRep.cpp:4896
#8  0x00002b0501884db2 in SimbodyMatterSubsystemRep::realizeSubsystemAccelerationImpl (this=0x2b05c6189f90, s=...) at /tmp/buildd/simbody-3.3/Simbody/src/SimbodyMatterSubsystemRep.cpp:1258
#9  0x00002b05026d2347 in SimTK::Subsystem::Guts::realizeSubsystemAcceleration (this=0x2b05c6189f90, s=...) at /tmp/buildd/simbody-3.3/SimTKcommon/Simulation/src/Subsystem.cpp:618
#10 0x00002b0501ad4d89 in SimTK::MultibodySystemRep::realizeAccelerationImpl (this=0x2b05c63586a0, s=...) at /tmp/buildd/simbody-3.3/Simbody/src/MultibodySystem.cpp:276
#11 0x00002b0502707d62 in SimTK::System::Guts::realizeAcceleration (this=this@entry=0x2b05c63586a0, s=...) at /tmp/buildd/simbody-3.3/SimTKcommon/Simulation/src/System.cpp:524
#12 0x00002b05027083b5 in SimTK::System::Guts::realize (this=0x2b05c63586a0, s=..., g=...) at /tmp/buildd/simbody-3.3/SimTKcommon/Simulation/src/System.cpp:767
#13 0x00002b0501ed5714 in realizeStateDerivatives (s=..., this=<optimized out>) at /tmp/buildd/simbody-3.3/SimTKmath/Integrators/src/IntegratorRep.h:758
#14 SimTK::AbstractIntegratorRep::stepTo (this=0x2b0514d45c20, reportTime=0.0030000000000000001, scheduledEventTime=0.0030000000000000001)
    at /tmp/buildd/simbody-3.3/SimTKmath/Integrators/src/AbstractIntegratorRep.cpp:393
#15 0x00002b04f9c091e8 in gazebo::physics::SimbodyPhysics::UpdatePhysics (this=0x2b05c776fc70) at /tmp/gazebo_build/source/gazebo/physics/simbody/SimbodyPhysics.cc:434
#16 0x00002b04f3c935b7 in gazebo::physics::World::Update (this=this@entry=0x2b05c62172e0) at /tmp/gazebo_build/source/gazebo/physics/World.cc:659
#17 0x00002b04f3c9beb4 in gazebo::physics::World::Step (this=this@entry=0x2b05c62172e0) at /tmp/gazebo_build/source/gazebo/physics/World.cc:551
#18 0x00002b04f3c9d2dd in gazebo::physics::World::RunLoop (this=0x2b05c62172e0) at /tmp/gazebo_build/source/gazebo/physics/World.cc:404
#19 0x00002b04f59016c9 in ?? () from /usr/lib/libboost_thread.so.1.49.0
#20 0x00002b04f4c56e9a in start_thread (arg=0x2b051aa86700) at pthread_create.c:308
#21 0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#22 0x0000000000000000 in ?? ()

Test thread:

Thread 1 (Thread 0x2b051ac87700 (LWP 21126)):
#0  0x00002b04f69ed425 in __GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00002b04f69f0b8b in __GI_abort () at abort.c:91
#2  0x00002b04f6201e2d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00002b04f61fff26 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00002b04f61fff53 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00002b04f620017e in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00002b05026f1dd2 in SimTK::StateImpl::getCacheEntry (this=<optimized out>, subsys=..., index=...) at /tmp/buildd/simbody-3.3/SimTKcommon/Simulation/src/State.cpp:2078
#7  0x00002b050188f719 in SimbodyMatterSubsystemRep::getTreeVelocityCache (this=0x2b05c6189f90, s=...) at /tmp/buildd/simbody-3.3/Simbody/src/SimbodyMatterSubsystemRep.h:978
#8  0x00002b050199def0 in getBodyVelocity (s=..., this=0x2b0515100be0) at /tmp/buildd/simbody-3.3/Simbody/src/MobilizedBodyImpl.h:285
#9  SimTK::MobilizedBody::getBodyVelocity (this=<optimized out>, s=...) at /tmp/buildd/simbody-3.3/Simbody/src/MobilizedBody.cpp:242
#10 0x00002b04f9bffee3 in getBodyAngularVelocity (state=..., this=0x2b05152aa688) at /usr/include/simbody/internal/MobilizedBody.h:354
#11 gazebo::physics::SimbodyLink::GetWorldAngularVel (this=0x2b05152aa1e0) at /tmp/gazebo_build/source/gazebo/physics/simbody/SimbodyLink.cc:360
#12 0x00002b04f3c5b596 in gazebo::physics::LinkState::Load (this=0x2b05bbe06db8, _link=..., _realTime=..., _simTime=...) at /tmp/gazebo_build/source/gazebo/physics/LinkState.cc:93
#13 0x00002b04f3c70033 in gazebo::physics::ModelState::Load (this=0x2b05c11abe88, _model=..., _realTime=..., _simTime=...) at /tmp/gazebo_build/source/gazebo/physics/ModelState.cc:109
#14 0x00002b04f3caf0fb in gazebo::physics::WorldState::Load (this=0x2b05c6217848, _world=...) at /tmp/gazebo_build/source/gazebo/physics/WorldState.cc:81
#15 0x00002b04f3c9fad1 in gazebo::physics::World::LogWorker (this=0x2b05c62172e0) at /tmp/gazebo_build/source/gazebo/physics/World.cc:1954
#16 0x00002b04f59016c9 in ?? () from /usr/lib/libboost_thread.so.1.49.0
#17 0x00002b04f4c56e9a in start_thread (arg=0x2b051ac87700) at pthread_create.c:308
#18 0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#19 0x0000000000000000 in ?? ()

Full backtrace:

(gdb) thread apply all bt

Thread 27 (Thread 0x2b051aa86700 (LWP 21125)):
#0  nelt (this=0x2b05c499d7f0) at /tmp/buildd/simbody-3.3/SimTKcommon/BigMatrix/src/MatrixHelperRep.h:401
#1  SimTK::MatrixHelper<double>::nelt (this=0x2b05c5eba288) at /tmp/buildd/simbody-3.3/SimTKcommon/BigMatrix/src/MatrixHelper.cpp:501
#2  0x00002b050187b9f3 in nelt (this=0x2b05c5eba288) at /tmp/buildd/simbody-3.3/SimTKcommon/BigMatrix/include/SimTKcommon/internal/BigMatrix.h:299
#3  size (this=0x2b05c5eba288) at /tmp/buildd/simbody-3.3/SimTKcommon/BigMatrix/include/SimTKcommon/internal/BigMatrix.h:1414
#4  SimbodyMatterSubsystemRep::calcTreeAccelerations (this=this@entry=0x2b05c6189f90, s=..., mobilityForces=..., bodyForces=..., presUDots=..., netHingeForces=..., allZ=..., allZPlus=..., A_GB=..., udot=..., 
    qdotdot=..., tau=...) at /tmp/buildd/simbody-3.3/Simbody/src/SimbodyMatterSubsystemRep.cpp:5014
#5  0x00002b050187bef1 in SimbodyMatterSubsystemRep::calcTreeForwardDynamicsOperator (this=this@entry=0x2b05c6189f90, s=..., mobilityForces=..., particleForces=..., bodyForces=..., 
    extraMobilityForces=extraMobilityForces@entry=0x0, extraBodyForces=extraBodyForces@entry=0x0, tac=..., udot=..., qdotdot=..., udotErr=...)
    at /tmp/buildd/simbody-3.3/Simbody/src/SimbodyMatterSubsystemRep.cpp:4737
#6  0x00002b0501884929 in SimbodyMatterSubsystemRep::calcLoopForwardDynamicsOperator (this=this@entry=0x2b05c6189f90, s=..., mobilityForces=..., particleForces=..., bodyForces=..., tac=..., cac=..., udot=..., 
    qdotdot=..., multipliers=..., udotErr=...) at /tmp/buildd/simbody-3.3/Simbody/src/SimbodyMatterSubsystemRep.cpp:4810
#7  0x00002b0501884c8d in SimbodyMatterSubsystemRep::realizeLoopForwardDynamics (this=0x2b05c6189f90, s=..., mobilityForces=..., particleForces=..., bodyForces=...)
    at /tmp/buildd/simbody-3.3/Simbody/src/SimbodyMatterSubsystemRep.cpp:4896
#8  0x00002b0501884db2 in SimbodyMatterSubsystemRep::realizeSubsystemAccelerationImpl (this=0x2b05c6189f90, s=...) at /tmp/buildd/simbody-3.3/Simbody/src/SimbodyMatterSubsystemRep.cpp:1258
#9  0x00002b05026d2347 in SimTK::Subsystem::Guts::realizeSubsystemAcceleration (this=0x2b05c6189f90, s=...) at /tmp/buildd/simbody-3.3/SimTKcommon/Simulation/src/Subsystem.cpp:618
#10 0x00002b0501ad4d89 in SimTK::MultibodySystemRep::realizeAccelerationImpl (this=0x2b05c63586a0, s=...) at /tmp/buildd/simbody-3.3/Simbody/src/MultibodySystem.cpp:276
#11 0x00002b0502707d62 in SimTK::System::Guts::realizeAcceleration (this=this@entry=0x2b05c63586a0, s=...) at /tmp/buildd/simbody-3.3/SimTKcommon/Simulation/src/System.cpp:524
#12 0x00002b05027083b5 in SimTK::System::Guts::realize (this=0x2b05c63586a0, s=..., g=...) at /tmp/buildd/simbody-3.3/SimTKcommon/Simulation/src/System.cpp:767
#13 0x00002b0501ed5714 in realizeStateDerivatives (s=..., this=<optimized out>) at /tmp/buildd/simbody-3.3/SimTKmath/Integrators/src/IntegratorRep.h:758
#14 SimTK::AbstractIntegratorRep::stepTo (this=0x2b0514d45c20, reportTime=0.0030000000000000001, scheduledEventTime=0.0030000000000000001)
    at /tmp/buildd/simbody-3.3/SimTKmath/Integrators/src/AbstractIntegratorRep.cpp:393
#15 0x00002b04f9c091e8 in gazebo::physics::SimbodyPhysics::UpdatePhysics (this=0x2b05c776fc70) at /tmp/gazebo_build/source/gazebo/physics/simbody/SimbodyPhysics.cc:434
#16 0x00002b04f3c935b7 in gazebo::physics::World::Update (this=this@entry=0x2b05c62172e0) at /tmp/gazebo_build/source/gazebo/physics/World.cc:659
#17 0x00002b04f3c9beb4 in gazebo::physics::World::Step (this=this@entry=0x2b05c62172e0) at /tmp/gazebo_build/source/gazebo/physics/World.cc:551
#18 0x00002b04f3c9d2dd in gazebo::physics::World::RunLoop (this=0x2b05c62172e0) at /tmp/gazebo_build/source/gazebo/physics/World.cc:404
#19 0x00002b04f59016c9 in ?? () from /usr/lib/libboost_thread.so.1.49.0
#20 0x00002b04f4c56e9a in start_thread (arg=0x2b051aa86700) at pthread_create.c:308
#21 0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#22 0x0000000000000000 in ?? ()

Thread 26 (Thread 0x2b054c801700 (LWP 21104)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00002b04fac2b3cb in boost::condition_variable_any::wait<boost::unique_lock<boost::recursive_mutex> > (this=this@entry=0x2b0598338ae0, m=...)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00002b04fac2a4df in Ogre::DefaultWorkQueue::waitForNextRequest (this=0x2b0598338690) at /build/buildd/ogre-1.7.4/OgreMain/src/Threading/OgreDefaultWorkQueueStandard.cpp:163
#3  0x00002b04fac2a12c in Ogre::DefaultWorkQueue::_threadMain (this=0x2b0598338690) at /build/buildd/ogre-1.7.4/OgreMain/src/Threading/OgreDefaultWorkQueueStandard.cpp:191
#4  0x00002b04f59016c9 in ?? () from /usr/lib/libboost_thread.so.1.49.0
#5  0x00002b04f4c56e9a in start_thread (arg=0x2b054c801700) at pthread_create.c:308
#6  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 25 (Thread 0x2b051a684700 (LWP 21123)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00002b04f36ddbeb in boost::condition_variable::wait (this=0x2b051570a0f8, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:56
#2  0x00002b04f3fb806f in gazebo::sensors::SensorManager::SensorContainer::RunLoop (this=0x2b051570a0a0) at /tmp/gazebo_build/source/gazebo/sensors/SensorManager.cc:466
#3  0x00002b04f59016c9 in ?? () from /usr/lib/libboost_thread.so.1.49.0
#4  0x00002b04f4c56e9a in start_thread (arg=0x2b051a684700) at pthread_create.c:308
#5  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 24 (Thread 0x2b054de06700 (LWP 21122)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00002b04f36ddbeb in boost::condition_variable::wait (this=this@entry=0x6c58d8, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:56
#2  0x00002b04f36d7814 in gazebo::common::ModelDatabase::UpdateModelCache (this=0x6c5840, _fetchImmediately=false) at /tmp/gazebo_build/source/gazebo/common/ModelDatabase.cc:277
#3  0x00002b04f59016c9 in ?? () from /usr/lib/libboost_thread.so.1.49.0
#4  0x00002b04f4c56e9a in start_thread (arg=0x2b054de06700) at pthread_create.c:308
#5  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 23 (Thread 0x2b05457c7700 (LWP 21116)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:39
#1  0x00002b04f5f8b293 in ?? () from /usr/lib/libtbb.so.2
#2  0x00002b04f5f8b2c9 in ?? () from /usr/lib/libtbb.so.2
#3  0x00002b04f4c56e9a in start_thread (arg=0x2b05457c7700) at pthread_create.c:308
#4  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 22 (Thread 0x2b05463ca700 (LWP 21119)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:39
---Type <return> to continue, or q <return> to quit---
#1  0x00002b04f5f8b293 in ?? () from /usr/lib/libtbb.so.2
#2  0x00002b04f5f8b2c9 in ?? () from /usr/lib/libtbb.so.2
#3  0x00002b04f4c56e9a in start_thread (arg=0x2b05463ca700) at pthread_create.c:308
#4  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 21 (Thread 0x2b051a885700 (LWP 21124)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00002b04f36ddbeb in boost::condition_variable::wait (this=0x2b05152d3e18, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:56
#2  0x00002b04f3fb806f in gazebo::sensors::SensorManager::SensorContainer::RunLoop (this=0x2b05152d3dc0) at /tmp/gazebo_build/source/gazebo/sensors/SensorManager.cc:466
#3  0x00002b04f59016c9 in ?? () from /usr/lib/libboost_thread.so.1.49.0
#4  0x00002b04f4c56e9a in start_thread (arg=0x2b051a885700) at pthread_create.c:308
#5  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 20 (Thread 0x2b05467cb700 (LWP 21120)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:39
#1  0x00002b04f5f8b293 in ?? () from /usr/lib/libtbb.so.2
#2  0x00002b04f5f8b2c9 in ?? () from /usr/lib/libtbb.so.2
#3  0x00002b04f4c56e9a in start_thread (arg=0x2b05467cb700) at pthread_create.c:308
#4  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 19 (Thread 0x2b0545bc8700 (LWP 21117)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:39
#1  0x00002b04f5f8b293 in ?? () from /usr/lib/libtbb.so.2
#2  0x00002b04f5f8b2c9 in ?? () from /usr/lib/libtbb.so.2
#3  0x00002b04f4c56e9a in start_thread (arg=0x2b0545bc8700) at pthread_create.c:308
#4  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 18 (Thread 0x2b0544fc5700 (LWP 21114)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x00002b04f49e72e5 in timed_wait (wait_until=..., m=..., this=0x6c5228) at /usr/include/boost/thread/pthread/condition_variable.hpp:74
#2  timed_wait<boost::date_time::subsecond_duration<boost::posix_time::time_duration, 1000l> > (wait_duration=..., m=..., this=0x6c5228) at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:72
#3  gazebo::transport::ConnectionManager::Run (this=0x6c5220) at /tmp/gazebo_build/source/gazebo/transport/ConnectionManager.cc:316
#4  0x00002b04f59016c9 in ?? () from /usr/lib/libboost_thread.so.1.49.0
#5  0x00002b04f4c56e9a in start_thread (arg=0x2b0544fc5700) at pthread_create.c:308
#6  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 17 (Thread 0x2b054dc05700 (LWP 21121)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:39
#1  0x00002b04f5f8b293 in ?? () from /usr/lib/libtbb.so.2
#2  0x00002b04f5f8b2c9 in ?? () from /usr/lib/libtbb.so.2
#3  0x00002b04f4c56e9a in start_thread (arg=0x2b054dc05700) at pthread_create.c:308
#4  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 16 (Thread 0x2b054c400700 (LWP 21110)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00002b04fac2b3cb in boost::condition_variable_any::wait<boost::unique_lock<boost::recursive_mutex> > (this=this@entry=0x2b0598338ae0, m=...)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00002b04fac2a4df in Ogre::DefaultWorkQueue::waitForNextRequest (this=0x2b0598338690) at /build/buildd/ogre-1.7.4/OgreMain/src/Threading/OgreDefaultWorkQueueStandard.cpp:163
#3  0x00002b04fac2a12c in Ogre::DefaultWorkQueue::_threadMain (this=0x2b0598338690) at /build/buildd/ogre-1.7.4/OgreMain/src/Threading/OgreDefaultWorkQueueStandard.cpp:191
#4  0x00002b04f59016c9 in ?? () from /usr/lib/libboost_thread.so.1.49.0
#5  0x00002b04f4c56e9a in start_thread (arg=0x2b054c400700) at pthread_create.c:308
#6  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 15 (Thread 0x2b05453c6700 (LWP 21115)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:39
#1  0x00002b04f5f8b293 in ?? () from /usr/lib/libtbb.so.2
#2  0x00002b04f5f8b2c9 in ?? () from /usr/lib/libtbb.so.2
#3  0x00002b04f4c56e9a in start_thread (arg=0x2b05453c6700) at pthread_create.c:308
#4  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 14 (Thread 0x2b0519a6d700 (LWP 21112)):
#0  0x00002b04f6a9fa43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x00002b0560058b6f in ?? () from /usr/lib/x86_64-linux-gnu/libpulse.so.0
---Type <return> to continue, or q <return> to quit---
#2  0x00002b056004a0b5 in pa_mainloop_poll () from /usr/lib/x86_64-linux-gnu/libpulse.so.0
#3  0x00002b056004a719 in pa_mainloop_iterate () from /usr/lib/x86_64-linux-gnu/libpulse.so.0
#4  0x00002b056004a7d0 in pa_mainloop_run () from /usr/lib/x86_64-linux-gnu/libpulse.so.0
#5  0x00002b0560058b1f in ?? () from /usr/lib/x86_64-linux-gnu/libpulse.so.0
#6  0x00002b05604b8543 in ?? () from /usr/lib/x86_64-linux-gnu/pulseaudio/libpulsecommon-2.1.so
#7  0x00002b04f4c56e9a in start_thread (arg=0x2b0519a6d700) at pthread_create.c:308
#8  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9  0x0000000000000000 in ?? ()

Thread 13 (Thread 0x2b0544201700 (LWP 21113)):
#0  0x00002b04f4c5e52d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
#1  0x00002b04fc1dd54a in ?? () from /usr/lib/x86_64-linux-gnu/libopenal.so.1
#2  0x00002b04fc1ea5eb in ?? () from /usr/lib/x86_64-linux-gnu/libopenal.so.1
#3  0x00002b04fc1dce6a in ?? () from /usr/lib/x86_64-linux-gnu/libopenal.so.1
#4  0x00002b04f4c56e9a in start_thread (arg=0x2b0544201700) at pthread_create.c:308
#5  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 12 (Thread 0x2b050e5ac180 (LWP 20965)):
#0  clock_nanosleep (clock_id=<optimized out>, flags=<optimized out>, req=<optimized out>, rem=<optimized out>) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:51
#1  0x00002b04f36f26c6 in gazebo::common::Time::Sleep (_time=...) at /tmp/gazebo_build/source/gazebo/common/Time.cc:195
#2  0x00002b04f36f2af3 in gazebo::common::Time::MSleep (_ms=<optimized out>) at /tmp/gazebo_build/source/gazebo/common/Time.cc:214
#3  0x000000000045130d in GetPercentRealTime (this=<optimized out>) at /tmp/gazebo_build/source/test/ServerFixture.hh:267
#4  SpeedTest::ShapesWorld (this=this@entry=0xcc1b70, _physicsEngine=...) at /tmp/gazebo_build/source/test/integration/speed.cc:72
#5  0x00000000004524c1 in SpeedTest_ShapesWorld_Test::TestBody (this=0xcc1b70) at /tmp/gazebo_build/source/test/integration/speed.cc:103
#6  0x00000000004991ea in HandleSehExceptionsInMethodIfSupported<testing::Test, void> (method=<optimized out>, object=<optimized out>, location=<optimized out>)
    at /tmp/gazebo_build/source/test/gtest/src/gtest.cc:2090
#7  testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0xcc1b70, method=&virtual testing::Test::TestBody(), location=location@entry=0x4a1d9b "the test body")
    at /tmp/gazebo_build/source/test/gtest/src/gtest.cc:2126
#8  0x000000000048e6a9 in testing::Test::Run (this=this@entry=0xcc1b70) at /tmp/gazebo_build/source/test/gtest/src/gtest.cc:2162
#9  0x000000000048e787 in testing::TestInfo::Run (this=0xcbb430) at /tmp/gazebo_build/source/test/gtest/src/gtest.cc:2338
#10 0x000000000048e8c5 in testing::TestCase::Run (this=0xcbacf0) at /tmp/gazebo_build/source/test/gtest/src/gtest.cc:2445
#11 0x000000000048ec2d in RunAllTests (this=0xcb8770) at /tmp/gazebo_build/source/test/gtest/src/gtest.cc:4237
#12 testing::internal::UnitTestImpl::RunAllTests (this=0xcb8770) at /tmp/gazebo_build/source/test/gtest/src/gtest.cc:4145
#13 0x0000000000498d6a in HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (method=<optimized out>, object=<optimized out>, location=<optimized out>)
    at /tmp/gazebo_build/source/test/gtest/src/gtest.cc:2090
#14 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0xcb8770, method=
    (bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x48e940 <testing::internal::UnitTestImpl::RunAllTests()>, 
    location=location@entry=0x4a2e80 "auxiliary test code (environments or event listeners)") at /tmp/gazebo_build/source/test/gtest/src/gtest.cc:2126
#15 0x000000000048df01 in testing::UnitTest::Run (this=<optimized out>) at /tmp/gazebo_build/source/test/gtest/src/gtest.cc:3874
#16 0x000000000044e63f in main (argc=1, argv=<optimized out>) at /tmp/gazebo_build/source/test/integration/speed.cc:111

Thread 11 (Thread 0x2b0545fc9700 (LWP 21118)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:39
#1  0x00002b04f5f8b293 in ?? () from /usr/lib/libtbb.so.2
#2  0x00002b04f5f8b2c9 in ?? () from /usr/lib/libtbb.so.2
#3  0x00002b04f4c56e9a in start_thread (arg=0x2b0545fc9700) at pthread_create.c:308
#4  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 10 (Thread 0x2b054d403700 (LWP 21109)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00002b04fac2b3cb in boost::condition_variable_any::wait<boost::unique_lock<boost::recursive_mutex> > (this=this@entry=0x2b0598338ae0, m=...)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00002b04fac2a4df in Ogre::DefaultWorkQueue::waitForNextRequest (this=0x2b0598338690) at /build/buildd/ogre-1.7.4/OgreMain/src/Threading/OgreDefaultWorkQueueStandard.cpp:163
#3  0x00002b04fac2a12c in Ogre::DefaultWorkQueue::_threadMain (this=0x2b0598338690) at /build/buildd/ogre-1.7.4/OgreMain/src/Threading/OgreDefaultWorkQueueStandard.cpp:191
#4  0x00002b04f59016c9 in ?? () from /usr/lib/libboost_thread.so.1.49.0
#5  0x00002b04f4c56e9a in start_thread (arg=0x2b054d403700) at pthread_create.c:308
#6  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 9 (Thread 0x2b0546bcc700 (LWP 21103)):
#0  clock_nanosleep (clock_id=<optimized out>, flags=<optimized out>, req=<optimized out>, rem=<optimized out>) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:51
#1  0x00002b04f36f26c6 in gazebo::common::Time::Sleep (_time=...) at /tmp/gazebo_build/source/gazebo/common/Time.cc:195
#2  0x00002b04f36f2af3 in gazebo::common::Time::MSleep (_ms=<optimized out>) at /tmp/gazebo_build/source/gazebo/common/Time.cc:214
#3  0x0000000000477d0d in gazebo::Master::Run (this=0x2b05c4b7c370) at /tmp/gazebo_build/source/gazebo/Master.cc:300
#4  0x00002b04f59016c9 in ?? () from /usr/lib/libboost_thread.so.1.49.0
#5  0x00002b04f4c56e9a in start_thread (arg=0x2b0546bcc700) at pthread_create.c:308
#6  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

---Type <return> to continue, or q <return> to quit---
Thread 8 (Thread 0x2b0580c02700 (LWP 21111)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00002b04fac2b3cb in boost::condition_variable_any::wait<boost::unique_lock<boost::recursive_mutex> > (this=this@entry=0x2b0598338ae0, m=...)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00002b04fac2a4df in Ogre::DefaultWorkQueue::waitForNextRequest (this=0x2b0598338690) at /build/buildd/ogre-1.7.4/OgreMain/src/Threading/OgreDefaultWorkQueueStandard.cpp:163
#3  0x00002b04fac2a12c in Ogre::DefaultWorkQueue::_threadMain (this=0x2b0598338690) at /build/buildd/ogre-1.7.4/OgreMain/src/Threading/OgreDefaultWorkQueueStandard.cpp:191
#4  0x00002b04f59016c9 in ?? () from /usr/lib/libboost_thread.so.1.49.0
#5  0x00002b04f4c56e9a in start_thread (arg=0x2b0580c02700) at pthread_create.c:308
#6  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x2b05473ce700 (LWP 21107)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00002b04fac2b3cb in boost::condition_variable_any::wait<boost::unique_lock<boost::recursive_mutex> > (this=this@entry=0x2b0598338ae0, m=...)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00002b04fac2a4df in Ogre::DefaultWorkQueue::waitForNextRequest (this=0x2b0598338690) at /build/buildd/ogre-1.7.4/OgreMain/src/Threading/OgreDefaultWorkQueueStandard.cpp:163
#3  0x00002b04fac2a12c in Ogre::DefaultWorkQueue::_threadMain (this=0x2b0598338690) at /build/buildd/ogre-1.7.4/OgreMain/src/Threading/OgreDefaultWorkQueueStandard.cpp:191
#4  0x00002b04f59016c9 in ?? () from /usr/lib/libboost_thread.so.1.49.0
#5  0x00002b04f4c56e9a in start_thread (arg=0x2b05473ce700) at pthread_create.c:308
#6  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 6 (Thread 0x2b054d604700 (LWP 21102)):
#0  0x00002b04f6aaba93 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1  0x00002b04f49d8b65 in boost::asio::detail::epoll_reactor::run (this=0x2b0514b024d0, block=block@entry=true, ops=...) at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:366
#2  0x00002b04f49db356 in do_run_one (ec=..., private_op_queue=..., this_thread=..., lock=..., this=<optimized out>) at /usr/include/boost/asio/detail/impl/task_io_service.ipp:353
#3  boost::asio::detail::task_io_service::run (this=0x2b05c4bcb1a0, ec=...) at /usr/include/boost/asio/detail/impl/task_io_service.ipp:146
#4  0x00002b04f49eeec5 in boost::asio::io_service::run (this=0x2b0517459690) at /usr/include/boost/asio/impl/io_service.ipp:59
#5  0x00002b04f59016c9 in ?? () from /usr/lib/libboost_thread.so.1.49.0
#6  0x00002b04f4c56e9a in start_thread (arg=0x2b054d604700) at pthread_create.c:308
#7  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#8  0x0000000000000000 in ?? ()

Thread 5 (Thread 0x2b05477cf700 (LWP 21106)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00002b04fac2b3cb in boost::condition_variable_any::wait<boost::unique_lock<boost::recursive_mutex> > (this=this@entry=0x2b0598338ae0, m=...)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00002b04fac2a4df in Ogre::DefaultWorkQueue::waitForNextRequest (this=0x2b0598338690) at /build/buildd/ogre-1.7.4/OgreMain/src/Threading/OgreDefaultWorkQueueStandard.cpp:163
#3  0x00002b04fac2a12c in Ogre::DefaultWorkQueue::_threadMain (this=0x2b0598338690) at /build/buildd/ogre-1.7.4/OgreMain/src/Threading/OgreDefaultWorkQueueStandard.cpp:191
#4  0x00002b04f59016c9 in ?? () from /usr/lib/libboost_thread.so.1.49.0
#5  0x00002b04f4c56e9a in start_thread (arg=0x2b05477cf700) at pthread_create.c:308
#6  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 4 (Thread 0x2b0547bd0700 (LWP 21108)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00002b04fac2b3cb in boost::condition_variable_any::wait<boost::unique_lock<boost::recursive_mutex> > (this=this@entry=0x2b0598338ae0, m=...)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00002b04fac2a4df in Ogre::DefaultWorkQueue::waitForNextRequest (this=0x2b0598338690) at /build/buildd/ogre-1.7.4/OgreMain/src/Threading/OgreDefaultWorkQueueStandard.cpp:163
#3  0x00002b04fac2a12c in Ogre::DefaultWorkQueue::_threadMain (this=0x2b0598338690) at /build/buildd/ogre-1.7.4/OgreMain/src/Threading/OgreDefaultWorkQueueStandard.cpp:191
#4  0x00002b04f59016c9 in ?? () from /usr/lib/libboost_thread.so.1.49.0
#5  0x00002b04f4c56e9a in start_thread (arg=0x2b0547bd0700) at pthread_create.c:308
#6  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 3 (Thread 0x2b0547fd1700 (LWP 21105)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00002b04fac2b3cb in boost::condition_variable_any::wait<boost::unique_lock<boost::recursive_mutex> > (this=this@entry=0x2b0598338ae0, m=...)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:137
#2  0x00002b04fac2a4df in Ogre::DefaultWorkQueue::waitForNextRequest (this=0x2b0598338690) at /build/buildd/ogre-1.7.4/OgreMain/src/Threading/OgreDefaultWorkQueueStandard.cpp:163
#3  0x00002b04fac2a12c in Ogre::DefaultWorkQueue::_threadMain (this=0x2b0598338690) at /build/buildd/ogre-1.7.4/OgreMain/src/Threading/OgreDefaultWorkQueueStandard.cpp:191
#4  0x00002b04f59016c9 in ?? () from /usr/lib/libboost_thread.so.1.49.0
#5  0x00002b04f4c56e9a in start_thread (arg=0x2b0547fd1700) at pthread_create.c:308
#6  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 2 (Thread 0x2b050f0e6700 (LWP 21101)):
#0  clock_nanosleep (clock_id=<optimized out>, flags=<optimized out>, req=<optimized out>, rem=<optimized out>) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:51
#1  0x00002b04f36f26c6 in gazebo::common::Time::Sleep (_time=...) at /tmp/gazebo_build/source/gazebo/common/Time.cc:195
#2  0x00002b04f36f2af3 in gazebo::common::Time::MSleep (_ms=<optimized out>) at /tmp/gazebo_build/source/gazebo/common/Time.cc:214
#3  0x0000000000469b16 in gazebo::Server::Run (this=0x2b0515a6ffa0) at /tmp/gazebo_build/source/gazebo/Server.cc:492
---Type <return> to continue, or q <return> to quit---
#4  0x000000000045968e in ServerFixture::RunServer (this=0xcc1b70, _worldFilename=..., _paused=<optimized out>, _physics=...) at /tmp/gazebo_build/source/test/ServerFixture.hh:233
#5  0x00002b04f59016c9 in ?? () from /usr/lib/libboost_thread.so.1.49.0
#6  0x00002b04f4c56e9a in start_thread (arg=0x2b050f0e6700) at pthread_create.c:308
#7  0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#8  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x2b051ac87700 (LWP 21126)):
#0  0x00002b04f69ed425 in __GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00002b04f69f0b8b in __GI_abort () at abort.c:91
#2  0x00002b04f6201e2d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00002b04f61fff26 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00002b04f61fff53 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00002b04f620017e in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00002b05026f1dd2 in SimTK::StateImpl::getCacheEntry (this=<optimized out>, subsys=..., index=...) at /tmp/buildd/simbody-3.3/SimTKcommon/Simulation/src/State.cpp:2078
#7  0x00002b050188f719 in SimbodyMatterSubsystemRep::getTreeVelocityCache (this=0x2b05c6189f90, s=...) at /tmp/buildd/simbody-3.3/Simbody/src/SimbodyMatterSubsystemRep.h:978
#8  0x00002b050199def0 in getBodyVelocity (s=..., this=0x2b0515100be0) at /tmp/buildd/simbody-3.3/Simbody/src/MobilizedBodyImpl.h:285
#9  SimTK::MobilizedBody::getBodyVelocity (this=<optimized out>, s=...) at /tmp/buildd/simbody-3.3/Simbody/src/MobilizedBody.cpp:242
#10 0x00002b04f9bffee3 in getBodyAngularVelocity (state=..., this=0x2b05152aa688) at /usr/include/simbody/internal/MobilizedBody.h:354
#11 gazebo::physics::SimbodyLink::GetWorldAngularVel (this=0x2b05152aa1e0) at /tmp/gazebo_build/source/gazebo/physics/simbody/SimbodyLink.cc:360
#12 0x00002b04f3c5b596 in gazebo::physics::LinkState::Load (this=0x2b05bbe06db8, _link=..., _realTime=..., _simTime=...) at /tmp/gazebo_build/source/gazebo/physics/LinkState.cc:93
#13 0x00002b04f3c70033 in gazebo::physics::ModelState::Load (this=0x2b05c11abe88, _model=..., _realTime=..., _simTime=...) at /tmp/gazebo_build/source/gazebo/physics/ModelState.cc:109
#14 0x00002b04f3caf0fb in gazebo::physics::WorldState::Load (this=0x2b05c6217848, _world=...) at /tmp/gazebo_build/source/gazebo/physics/WorldState.cc:81
#15 0x00002b04f3c9fad1 in gazebo::physics::World::LogWorker (this=0x2b05c62172e0) at /tmp/gazebo_build/source/gazebo/physics/World.cc:1954
#16 0x00002b04f59016c9 in ?? () from /usr/lib/libboost_thread.so.1.49.0
#17 0x00002b04f4c56e9a in start_thread (arg=0x2b051ac87700) at pthread_create.c:308
#18 0x00002b04f6aab3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#19 0x0000000000000000 in ?? ()
(gdb)

osrf-migration · 2013-11-20T23:04:53Z

Original comment by John Hsu (Bitbucket: hsu, GitHub: hsu).

@scpeters what branch are you on when you got these backtraces?

thanks.

osrf-migration · 2013-11-20T23:39:07Z

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

Two of the Jenkins failures were on default, my overnight test was on gz_tool_part1 branch (a00b5c0), which is branched from default as well. I'm not sure if it happens on gazebo_2.1 at all.

osrf-migration · 2013-12-13T17:41:55Z

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

@sherm1 @tkuchida

osrf-migration · 2013-12-13T17:47:26Z

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

try locking physics update mutex in LinkState::Load?

osrf-migration · 2013-12-13T18:51:15Z

Original comment by Michael Sherman (Bitbucket: sherm1).

Per discussion at this morning's meeting: to me this looks like a timing problem -- it appears that thread 1 (testing thread) tried to pull something out of the same state that thread 27 (simulation thread) was writing on. It is thread 1 that failed. This error message is a pretty good clue:

terminate called after throwing an instance of 'SimTK::Exception::StageTooLow'
what(): SimTK Exception thrown at State.cpp:2069:
Expected stage to be at least Position in StateImpl::getCacheEntry() but current stage was Position

Looks like the stage advanced between the time the stage check failed and the time when the error message was formatted, indicating that another thread was writing to it.

osrf-migration · 2013-12-13T22:12:58Z

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

changed title from "Simbody fails INTEGRATION_speed test once in a while" to "SimbodyLink::GetWorldAngularVel and others are not thread safe"

osrf-migration · 2013-12-13T22:16:07Z

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

I've updated the title, since I believe that it is assumed that the gazebo physics API functions are thread-safe, and that some of the SimbodyPhysics functions are not.

To illustrate this, I added a test for thread safety in 8ba87c2 (branch issue_918). The test starts up a world with a bunch of pendulums and un-throttles it so it runs as fast as it can. Then in the testing thread, The Link::GetWorldLinearVel and Link::GetWorldAngularVel functions are called repeatedly. The simbody incarnation of this test fails immediately.

osrf-migration · 2013-12-13T23:01:07Z

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

I expanded the test a bit and added a fix in 179a8cf.

osrf-migration · 2013-12-13T23:30:07Z

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

See pull request #872

osrf-migration · 2013-12-17T19:46:17Z

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

changed state from "new" to "resolved"

pull request #872

osrf-migration · 2016-09-14T19:21:47Z

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).

set version to "all"

osrf-migration · 2016-09-14T19:44:40Z

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).

changed state from "resolved" to "closed"

osrf-migration closed this as completed Dec 17, 2013

osrf-migration added blocker physics::SimbodyPhysics bug Something isn't working all labels Apr 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SimbodyLink::GetWorldAngularVel and others are not thread safe #918

SimbodyLink::GetWorldAngularVel and others are not thread safe #918

osrf-migration commented Oct 23, 2013

osrf-migration commented Oct 23, 2013

osrf-migration commented Nov 6, 2013

osrf-migration commented Nov 6, 2013

osrf-migration commented Nov 6, 2013

osrf-migration commented Nov 15, 2013

osrf-migration commented Nov 19, 2013

osrf-migration commented Nov 20, 2013

osrf-migration commented Nov 20, 2013

osrf-migration commented Dec 13, 2013

osrf-migration commented Dec 13, 2013

osrf-migration commented Dec 13, 2013

osrf-migration commented Dec 13, 2013

osrf-migration commented Dec 13, 2013

osrf-migration commented Dec 13, 2013

osrf-migration commented Dec 13, 2013

osrf-migration commented Dec 17, 2013

osrf-migration commented Sep 14, 2016

osrf-migration commented Sep 14, 2016

SimbodyLink::GetWorldAngularVel and others are not thread safe #918

SimbodyLink::GetWorldAngularVel and others are not thread safe #918

Comments

osrf-migration commented Oct 23, 2013

osrf-migration commented Oct 23, 2013

osrf-migration commented Nov 6, 2013

osrf-migration commented Nov 6, 2013

osrf-migration commented Nov 6, 2013

osrf-migration commented Nov 15, 2013

osrf-migration commented Nov 19, 2013

osrf-migration commented Nov 20, 2013

osrf-migration commented Nov 20, 2013

osrf-migration commented Dec 13, 2013

osrf-migration commented Dec 13, 2013

osrf-migration commented Dec 13, 2013

osrf-migration commented Dec 13, 2013

osrf-migration commented Dec 13, 2013

osrf-migration commented Dec 13, 2013

osrf-migration commented Dec 13, 2013

osrf-migration commented Dec 17, 2013

osrf-migration commented Sep 14, 2016

osrf-migration commented Sep 14, 2016