Double-free on exit while running test suite

[swansontec]

Running make check gives this:

make[2]: Entering directory `/home/bitcoin/build-libbitcoin-server/libbitcoin-server'
*** Error in `/home/bitcoin/build-libbitcoin-server/libbitcoin-server/test/.libs/lt-libbitcoin_server_test': double free or corruption (!prev): 0x0000000000a311c0 ***
FAIL: libbitcoin_server_test_runner.sh

Running under valgrind gives:

==13633== Memcheck, a memory error detector
==13633== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==13633== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info
==13633== Command: ./lt-libbitcoin_server_test
==13633== 

*** No errors detected
==13633== Invalid read of size 4
==13633==    at 0x596C64D: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19)
==13633==    by 0x59CB4DE: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/x86_64-linux-g
nu/libstdc++.so.6.0.19)
==13633==    by 0x5E675E9: __cxa_finalize (cxa_finalize.c:56)
==13633==    by 0x79D07B2: ??? (in /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.54.0)
==13633==    by 0x4010739: _dl_fini (dl-fini.c:252)
==13633==    by 0x5E67258: __run_exit_handlers (exit.c:82)
==13633==    by 0x5E672A4: exit (exit.c:104)
==13633==    by 0x5E4CECB: (below main) (libc-start.c:321)
==13633==  Address 0xa983350 is 16 bytes inside a block of size 28 free'd
==13633==    at 0x4C2C2BC: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==13633==    by 0x59CB4DE: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/x86_64-linux-g
nu/libstdc++.so.6.0.19)
==13633==    by 0x5E675E9: __cxa_finalize (cxa_finalize.c:56)
==13633==    by 0x6904C42: ??? (in /usr/local/lib/libboost_program_options.so.1.55.0)
==13633==    by 0x4010739: _dl_fini (dl-fini.c:252)
==13633==    by 0x5E67258: __run_exit_handlers (exit.c:82)
==13633==    by 0x5E672A4: exit (exit.c:104)
==13633==    by 0x5E4CECB: (below main) (libc-start.c:321)
==13633== 
==13633== Invalid free() / delete / delete[] / realloc()
==13633==    at 0x4C2C2BC: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==13633==    by 0x59CB4DE: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/x86_64-linux-g
nu/libstdc++.so.6.0.19)
==13633==    by 0x5E675E9: __cxa_finalize (cxa_finalize.c:56)
==13633==    by 0x79D07B2: ??? (in /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.54.0)
==13633==    by 0x4010739: _dl_fini (dl-fini.c:252)
==13633==    by 0x5E67258: __run_exit_handlers (exit.c:82)
==13633==    by 0x5E672A4: exit (exit.c:104)
==13633==    by 0x5E4CECB: (below main) (libc-start.c:321)
==13633==  Address 0xa983340 is 0 bytes inside a block of size 28 free'd
==13633==    at 0x4C2C2BC: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==13633==    by 0x59CB4DE: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/x86_64-linux-g
nu/libstdc++.so.6.0.19)
==13633==    by 0x5E675E9: __cxa_finalize (cxa_finalize.c:56)
==13633==    by 0x6904C42: ??? (in /usr/local/lib/libboost_program_options.so.1.55.0)
==13633==    by 0x4010739: _dl_fini (dl-fini.c:252)
==13633==    by 0x5E67258: __run_exit_handlers (exit.c:82)
==13633==    by 0x5E672A4: exit (exit.c:104)
==13633==    by 0x5E4CECB: (below main) (libc-start.c:321)
==13633== 
==13633== 
==13633== HEAP SUMMARY:
==13633==     in use at exit: 28 bytes in 1 blocks
==13633==   total heap usage: 742 allocs, 742 frees, 128,988 bytes allocated
==13633== 
==13633== LEAK SUMMARY:
==13633==    definitely lost: 28 bytes in 1 blocks
==13633==    indirectly lost: 0 bytes in 0 blocks
==13633==      possibly lost: 0 bytes in 0 blocks
==13633==    still reachable: 0 bytes in 0 blocks
==13633==         suppressed: 0 bytes in 0 blocks
==13633== Rerun with --leak-check=full to see details of leaked memory
==13633== 
==13633== For counts of detected and suppressed errors, rerun with: -v
==13633== ERROR SUMMARY: 4 errors from 2 contexts (suppressed: 0 from 0)

It seems like everything runs cleanly (*** No errors detected), but something goes wrong during the shutdown procedure. This is a freshly-updated Ubuntu 14.04 box, and the libbitcoin-server is the latest version (525e3950), compiled like sudo ./install.sh --build-boost --without-consensus.

[evoskuil]

My first thought was that there was a regression from my recent refactoring. But there are two versions of boost in the valgrind report, 1.55.0 and 1.54.0. Could you resolve that and let me know if there's still an issue?

[swansontec]

Good catch! I wonder how that happened. Both versions of boost are in /usr/local, so the configure scripts are probably mixing-and-matching libraries. I did a fresh re-build on this box, and got the same problem. Let me move the old /usr/local/lib out of the way and try again.

[swansontec]

Erp, no. It looks like boost 1.54 is the system-installed version, and that 1.55 is the /usr/local version. I'll try building again without the --build-boost flag.

[evoskuil]

Makes sense. I avoid these issues in my testing by combining --prefix and --build-boost.

[swansontec]

Ok, so if I build with --build-boost, libbitcoin picks up the system version (1.54), but libbitcoin-server picks up the local one (1.55). If I build without that flag, I get errors about missing boost headers (Ubuntu packaging issue?). Let me try again with an explicit --prefix=/usr/local.

[evoskuil]

When you combine both using the install script it passes the built boost path via --with-boost to all of the builds, which is picked up by the boost m4 discovery scripts. That takes priority. Otherwise the m4 scripts are complex...

[swansontec]

Yep, combining --prefix and --build-boost seems to produce a valid build. I still need to call sudo ldconfig before make check works, but the crash is gone once that happens.

I wonder if there is anything we can to do make the install script safer. It's too bad the built-in Ubuntu boost is broken.

[evoskuil]

Making the scripts safer in a non-prefix install would probably require replacing the boost discovery scripts. libtool is also a bit problematic with its selection of libraries, even when isolated to a prefix install. When building static one needs to build boost (and icu if applicable) so that libtool won't go after the shared system libs.

I never use sudo in my prefix builds. The install.sh will build the whole thing and execute tests. sudo should only be required if you do a non-prefix install.

Yeah, it sucked to have to upgrade boost again.

[swansontec]

It might be as simple as a warning like "You are using --build-boost. Please consider specifying a --prefix for safety."

[evoskuil]

Good idea, I'll work that into the install script.