Annoying debug message: DBG: YANKING!!! (face X)

[scpeters]

There is a console message in Simbody/src/CompliantContactSubsystem.cpp that has the potential to dominate all console output when compliant contact starts to behave strangely (copied from gazebo jenkins testing):

...
105: DBG: YANKING!!! (face 403)
105: DBG: YANKING!!! (face 404)
105: DBG: YANKING!!! (face 405)
105: DBG: YANKING!!! (face 406)
105: DBG: YANKING!!! (face 407)
105: DBG: YANKING!!! (face 408)
105: DBG: YANKING!!! (face 409)
105: DBG: YANKING!!! (face 410)
105: DBG: YANKING!!! (face 411)
...

This single message increases the console log from about 2 MB to over 20 MB.

I mentioned this to @sherm1 once before, and he said that this message only displays when the code is compiled with debug settings. That is true, but it appears that the Ubuntu debs that are hosted at packages.osrfoundation.org do not have the NDEBUG symbol defined.

@j-rivero perhaps we can discuss which compilation flags we should use for the debs that we host?

[sherm1]

Simbody performs very poorly unless NDEBUG is defined -- definitely a bug if the builds aren't setting that. I'm glad you noticed, Steve!

[j-rivero]

We are currently not setting any special flag for Simbody builds in OSRF repository and fall into the default behaviour which seems to be -O2 -g (CMAKE_BUILD_TYPE=RelWithDebInfo). Same for the debian submission.

We could change them if there is a good reason to do it.

[scpeters]

There's some lines in CMakeLists.txt that tries to set compiler flags. Perhaps this is a discussion of RelWithDebInfo vs. Release?

[sherm1]

For gcc & clang it should be using Release. However, I think that is the default for the CMake-generated makefiles on Linux. I'm not sure how it ended up with RelWithDebInfo in the OSRF/Debian builds.

(On Windows RelWithDebInfo and Release generate the same code -- NDEBUG is set either way; Release just deletes symbols from the object files.)

[j-rivero]

Ouch, let me correct my previous comment. It is not defaulting to RelWithDebInfo. If no build type is passed to cmake, no build type is assumed. What is really bringing the -O2 -g is the debian policy but we can choose for going with some other options, like -O3, if you believe it's safe.

[sherm1]

The Simbody CMake script sets the compile flags for various compilers and versions. On my Mac I'm seeing

CMAKE_BUILD_TYPE=Release
CMAKE_CXX_FLAGS_RELEASE=-DNDEBUG -O3 -funroll-loops -msse2

as default settings. I think this is similar for gcc though I have hacks in there for particular compiler versions that have optimizer bugs. I think the options above would be OK. Whether Release or RelWithDebInfo, it is very important to have -DNDEBUG defined or the performance will be terrible.

[j-rivero]

A couple of quick points:

• -O2 is used more extensively as default high optimization than -O3 in the Linux world in general. In particular, distributions from-source like Gentoo, adopt this change in 2004. It can end up in quite hard to detect bugs.
• I would only recommend to enable -funroll-loops if you know very well what you are doing. I remember even a parody web page about the gentoo max-performance which name is: http://funroll-loops.info/

updated: I missed the section for the weird bugs, they come much easier from -O3 than -funroll-loops. This later can sometimes affect the performance much more than generate new bugs.

[sherm1]

:-)
Well, originally I measured a good speedup with -funroll-loops I think because of the Simbody templatized vector implementation. But that was on some random gcc long ago and I have not repeated the test recently. Any optimizations can make debugging a nightmare -- I'm not sure this one is any worse. Still, I would be fine using just -O2 for safety. I don't know whether that would affect performance substantially, but I'm sure it won't be as noticeable as leaving off -DNDEBUG !

[j-rivero]

One proposal: leave Release mode using -O2 -funroll-loops -DNDEBUG and use RelWithDebInfo with -O2 -funroll-loops -g -DNDEBUG. I would prefer to skip -O3.

[sherm1]

Sounds good. I'm going to run some performance tests to see whether there is any noticeable difference btw your proposal and the existing -O3 stuff. If not I'll change the CMake settings to use those also. Is it currently the case that the Debian builds don't inherit build flags from CMakeLists.txt?

[j-rivero]

Debian is not using them since I'm not defining a CMAKE_BUILD_TYPE in the rules files (used to call cmake). I can do that (RelWithDebInfo), not a problem.

[sherm1]

I have never tried RelWithDebInfo on gcc or clang so I don't know whether the addition of the -g flag has a performance consequence. Do you know?

[j-rivero]

-g should not impact in the binaries performance, just they get a bigger size on disk but debug information is not even loaded into memory if you don't ask for it (i.e when running gdb). When after the process of generating the binaries with -g, you run strip (which debuild does) that information is taken from the binaries and stored into the -dbg package (they are in relationship using a 'build id').

Seems some compilers (like icc) disable optimizations if -g is used, but I've never read that this happen on gcc or clang.

[sherm1]

I ran some performance comparisons using clang on OSX 10.9:

• with -DNDEBUG -O2 I saw no performance change if I added -g or -funroll-loops
• with -DNDEBUG -O3 I saw about a 2% performance improvement, which I would call insignificant
• the -g flag makes the binaries about 30% bigger (e.g. libSimTKsimbody goes from 5MB to 7MB); that seems reasonable

I think these limited experiments suggest that the safest options would be perfectly fine:
-DNDEBUG -O2 -g

I think we should go with just those options until we have good reasons for something else.

[j-rivero]

Thanks very much for the tests Michael, it is always nice when the experiments are in line with the theory :) I'm happy with -O2 -g -DNDEBUG.

I've done the pull request #159 to modify the rules file in this repo and the same commit in debian.