Crash in src/engine/gnump/gmp_mem.cpp

[stikonas]

Hi, I was trying to investigate a crash in LibKTorrent (https://cgit.kde.org/libktorrent.git/) and somehow it points to Botan. Although, the crash looks very strange, so maybe it is not Botan's fault... Any ideas?

Just before the crash I get the message
terminate called after throwing an instance of 'Botan::Invalid_State'
what(): Pointer released to the wrong allocator

Backtrace:

Using host libthread_db library "/lib64/libthread_db.so.1".
[KCrash Handler]
#6 0x00007fec2b50b278 in raise () from /lib64/libc.so.6
#7 0x00007fec2b50c69a in abort () from /lib64/libc.so.6
#8 0x00007fec2be3b19d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/gcc/x86_64-pc-linux-gnu/6.3.0/libstdc++.so.6
#9 0x00007fec2be38f26 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/6.3.0/libstdc++.so.6
#10 0x00007fec2be37e79 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/6.3.0/libstdc++.so.6
#11 0x00007fec2be3885d in __gxx_personality_v0 () from /usr/lib/gcc/x86_64-pc-linux-gnu/6.3.0/libstdc++.so.6
#12 0x00007fec2b885b13 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/6.3.0/libgcc_s.so.1
#13 0x00007fec2b886009 in _Unwind_Resume () from /usr/lib/gcc/x86_64-pc-linux-gnu/6.3.0/libgcc_s.so.1
#14 0x00007fec09a7716c in Botan::Pooling_Allocator::deallocate (this=0x7fec00004480, ptr=0x1c1e8f0, n=104) at src/alloc/mem_pool/mem_pool.cpp:175
#15 0x00007fec09b10431 in Botan::(anonymous namespace)::gmp_free (ptr=0x1c1e8f0, n=104) at src/engine/gnump/gmp_mem.cpp:46
#16 0x00007fec2f8436c4 in mse::BigInt::~BigInt() () from /home/andrius/repositories/kde/libktorrent/build/src/libKF5Torrent.so.6
#17 0x00007fec2b50df07 in __cxa_finalize () from /lib64/libc.so.6
#18 0x00007fec2f80ca73 in ?? () from /home/andrius/repositories/kde/libktorrent/build/src/libKF5Torrent.so.6
#19 0x00007ffe10fadba0 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

[randombit]

There was a report along these lines (I think in RedHat bugzilla) that was caused due to Botan's use of mp_set_memory_functions which conflicted with use in another application or library, this resulted in errors like this especially in threaded code. Basically an object is allocated with one allocator and then surprisingly deallocated with another, causing heap corruption or other nastiness (just an exception in this instance, because the pool allocator verifies its pointer args).

Best resolution is to disable GNU MP from the Botan build. GNU MP support has already been dropped in more recent versions (since 2014 in master). HTH.

[stikonas]

Thanks a lot. It indeed works when I recompiled botan without gmp and now the crash is gone. I added a note in libktorrents CMakeLists.txt about this.

[Lekensteyn]

Fot future reference, I ran into the same issue with an application combining GnuTLS (which uses nettle which also uses GMP) and botan on Ubuntu 16.04 with the following packages:

• libgnutls30 3.4.10-4ubuntu1.4
• libbotan-1.10-1 1.10.12-1
• libnettle6 3.2-1ubuntu0.16.04.1
• libgmp10 2:6.1.0+dfsg-2

According to the news file the issue was resolved in version 1.11.10, 2014-12-10 by dropping GNU MP.
Commit removing it: 63215db