Segfaults in Jammy packaging while running tests on amd64

[j-rivero]

Environment

• OS Version: Ubuntu Jammy only on amd64
• Source or binary build?
  • Source, building packages

First attempts to build packages for Jammy for amd64 are failing due to segfaults in the test suite:

The following tests FAILED:
	 52 - UNIT_EntityManagement_TEST (SEGFAULT)
	 56 - UNIT_JointFeatures_TEST (SEGFAULT)
	 62 - UNIT_SDFFeatures_TEST (SEGFAULT)
	 64 - UNIT_ShapeFeatures_TEST (SEGFAULT)
Errors while running CTest

This is the first time we are using 6.12.x version coming from Ubuntu with patches from our osrf fork. Not sure if this is somehow related.

The problems seems to be somehow after the test passed, so probably related to destructors calls:

52: [----------] 3 tests from EntityManagement_TEST
52: [ RUN      ] EntityManagement_TEST.ConstructEmptyWorld
52: [       OK ] EntityManagement_TEST.ConstructEmptyWorld (59 ms)
52: [ RUN      ] EntityManagement_TEST.RemoveEntities
52: [       OK ] EntityManagement_TEST.RemoveEntities (44 ms)
52: [ RUN      ] EntityManagement_TEST.ModelByIndexWithNestedModels
52: [       OK ] EntityManagement_TEST.ModelByIndexWithNestedModels (45 ms)
52: [----------] 3 tests from EntityManagement_TEST (149 ms total)
52: 
52: [----------] Global test environment tear-down
52: [==========] 3 tests from 1 test case ran. (149 ms total)
52: [  PASSED  ] 3 tests.
 52/101 Test  #52: UNIT_EntityManagement_TEST ..................***Exception: SegFault  1.13 sec
[==========] Running 3 tests from 1 test case.

...

56: [----------] 5 tests from JointTransmittedWrenchFixture
56: [ RUN      ] JointTransmittedWrenchFixture.PendulumAtZeroAngle
56: [       OK ] JointTransmittedWrenchFixture.PendulumAtZeroAngle (153 ms)
56: [ RUN      ] JointTransmittedWrenchFixture.PendulumInMotion
56: [       OK ] JointTransmittedWrenchFixture.PendulumInMotion (173 ms)
56: [ RUN      ] JointTransmittedWrenchFixture.ValidateWrenchWithSecondaryJoint
56: [       OK ] JointTransmittedWrenchFixture.ValidateWrenchWithSecondaryJoint (174 ms)
56: [ RUN      ] JointTransmittedWrenchFixture.JointLosses
56: [       OK ] JointTransmittedWrenchFixture.JointLosses (157 ms)
56: [ RUN      ] JointTransmittedWrenchFixture.ContactForces
56: [       OK ] JointTransmittedWrenchFixture.ContactForces (221 ms)
56: [----------] 5 tests from JointTransmittedWrenchFixture (878 ms total)
56: 
56: [----------] Global test environment tear-down
56: [==========] 17 tests from 2 test cases ran. (5116 ms total)
56: [  PASSED  ] 17 tests.
56: 
56:   YOU HAVE 1 DISABLED TEST
56: 
 56/101 Test  #56: UNIT_JointFeatures_TEST .....................***Exception: SegFault  5.52 sec

...

Running the Dockerfile from Jenkins I was able to reproduce the segfaults. Quick gdb on artifacts produced by packaging points to DART:

Program received signal SIGSEGV, Segmentation fault.
0x00007fc7e165bb70 in ?? ()
(gdb) bt
#0  0x00007fc7e165bb70 in ?? ()
#1  0x00007fc7e31bd39d in ?? () from /lib/x86_64-linux-gnu/libdart.so.6.12
#2  0x00007fc7e283ba56 in __cxa_finalize (d=0x7fc7e33e6000) at ./stdlib/cxa_finalize.c:83
#3  0x00007fc7e3007dfa in ?? () from /lib/x86_64-linux-gnu/libdart.so.6.12
#4  0x00007fc7e342f040 in ?? () from /lib64/ld-linux-x86-64.so.2
#5  0x0000000000000000 in ?? ()
(gdb) 

Debian did not have problems producing 5.1.0+ds1-4 version for amd64 with dart 6.12.1+dfsg4-11.

Ubuntu had the same problems producing 5.1.0+ds1-4 version for amd64 with dart 6.12.1+dfsg4-11build1 although no notification or bug was raised to me as far as I can tell.

[scpeters]

the tests worked fine when enabling Jammy for GitHub Actions CI in #329

• https://github.com/ignitionrobotics/ign-physics/runs/5484550640?check_suite_focus=true

[j-rivero]

the tests worked fine when enabling Jammy for GitHub Actions CI in #329

Umm interesting. I've verified that packages for DART are using the same 6.12.1+dfsg4-11build1 in github actions. I run the packaging job locally, it segfaults using the exactly same environment that in building packages. If I use the same source code in that system but run manually cmake, make and make test, the segfaults are gone.

Could be something related to the compilation flags injected by the building environment. both seems to be using O2 and -g. I quick manual diff on the compilation lines shows that when building packages some hardening flags are being used not present in the manual building:

-ffile-prefix-map=/home/jenkins/workspace/ign-physics5-debbuilder/build/ignition-physics-5.1.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 

I've installed dart debugging symbols and run gdb again:

rogram received signal SIGSEGV, Segmentation fault.
0x00007f0e34cb7b70 in ?? ()
(gdb) bt
#0  0x00007f0e34cb7b70 in ?? ()
#1  0x00007f0e3770039d in std::_Function_base::~_Function_base (this=<optimized out>, this=<optimized out>) at /usr/include/c++/11/bits/std_function.h:244
#2  std::function<std::shared_ptr<dart::collision::CollisionDetector> ()>::~function() (this=<optimized out>, this=<optimized out>) at /usr/include/c++/11/bits/std_function.h:334
#3  std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::function<std::shared_ptr<dart::collision::CollisionDetector> ()> >::~pair() (this=<optimized out>, this=<optimized out>)
    at /usr/include/c++/11/bits/stl_pair.h:211
#4  __gnu_cxx::new_allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::function<std::shared_ptr<dart::collision::CollisionDetector> ()> >, true> >::destroy<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::function<std::shared_ptr<dart::collision::CollisionDetector> ()> > >(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::function<std::shared_ptr<dart::collision::CollisionDetector> ()> >*) (__p=0x55bd42d53fa8, this=<optimized out>) at /usr/include/c++/11/ext/new_allocator.h:168
#5  std::allocator_traits<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::function<std::shared_ptr<dart::collision::CollisionDetector> ()> >, true> > >::destroy<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::function<std::shared_ptr<dart::collision::CollisionDetector> ()> > >(std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::function<std::shared_ptr<dart::collision::CollisionDetector> ()> >, true> >&, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::function<std::shared_ptr<dart::collision::CollisionDetector> ()> >*) (__p=0x55bd42d53fa8, __a=...) at /usr/include/c++/11/bits/alloc_traits.h:535
#6  std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::function<std::shared_ptr<dart::collision::CollisionDetector> ()> >, true> > >::_M_deallocate_node(std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::function<std::shared_ptr<dart::collision::CollisionDetector> ()> >, true>*) (
    this=0x7f0e3792b668 <dart::common::Singleton<dart::common::Factory<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, dart::collision::CollisionDetector, std::shared_ptr<dart::collision::CollisionDetector>> >::getSingleton<>()::instance+8>, __n=0x55bd42d53fa0) at /usr/include/c++/11/bits/hashtable_policy.h:1894
#7  std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::function<std::shared_ptr<dart::collision::CollisionDetector> ()> >, true> > >::_M_deallocate_nodes(std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::function<std::shared_ptr<dart::collision::CollisionDetector> ()> >, true>*) (__n=0x55bd42d7d860, 
    this=0x7f0e3792b668 <dart::common::Singleton<dart::common::Factory<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, dart::collision::CollisionDetector, std::shared_ptr<dart::collision::CollisionDetector>> >::getSingleton<>()::instance+8>) at /usr/include/c++/11/bits/hashtable_policy.h:1916
#8  std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::function<std::shared_ptr<dart::collision::CollisionDetector> ()> >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::function<std::shared_ptr<dart::collision::CollisionDetector> ()> > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::clear() (
    this=0x7f0e3792b668 <dart::common::Singleton<dart::common::Factory<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, dart::collision::CollisionDetector, std::shared_ptr<dart::collision::CollisionDetector>> >::getSingleton<>()::instance+8>) at /usr/include/c++/11/bits/hashtable.h:2320
#9  std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::function<std::shared_ptr<dart::collision::CollisionDetector> ()> >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::function<std::shared_ptr<dart::collision::CollisionDetector> ()> > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::~_Hashtable() (this=<optimized out>, this=<optimized out>) at /usr/include/c++/11/bits/hashtable.h:1532
#10 std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::function<std::shared_ptr<dart::collision::CollisionDetector> ()>, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::function<std::shared_ptr<dart::collision::CollisionDetector> ()> > > >::~unordered_map() (this=<optimized out>, this=<optimized out>) at /usr/include/c++/11/bits/unordered_map.h:102
#11 dart::common::Factory<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, dart::collision::CollisionDetector, std::shared_ptr<dart::collision::CollisionDetector>>::~Factory() (this=<optimized out>, 
    this=<optimized out>) at ./dart/common/Factory.hpp:80
#12 0x00007f0e36e71a56 in __cxa_finalize (d=0x7f0e37929000) at ./stdlib/cxa_finalize.c:83
#13 0x00007f0e3754adfa in __do_global_dtors_aux () from /lib/x86_64-linux-gnu/libdart.so.6.12
#14 0x00007f0e37972040 in ?? () from /lib64/ld-linux-x86-64.so.2
#15 0x0000000000000000 in ?? ()

[j-rivero]

Could be something related to the compilation flags injected by the building environment. both seems to be using O2 and -g. I quick manual diff on the compilation lines shows that when building packages some hardening flags are being used not present in the manual building:

I'm able to reproduce the error now outside the packaging env by using:

cmake .. -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CXX_FLAGS="-flto=auto -ffat-lto-objects"

Somehow this linker optimizations are making tests to segfault when running dart destructors.

[j-rivero]

confirmed, filtering out linker optimizations fixes the problem.

[j-rivero]

Patch in place gazebo-release/gz-physics5-release#7 (comment), tested here . Closing.