Reference to tcmalloc in leveldb's shared object causes crashes upon late loading leveldb

[Peaker]

When dynamically loading leveldb's shared object, libtcmalloc (if enabled) is implicitly loaded too.
The loading of libtcmalloc overrides malloc, free and other libc functions.
This override is fine when it happens at the beginning of execution but causes 2 problems when it is late loaded (with dlopen after executable had started):

• Memory previously allocated with malloc can be freed with tcmalloc, causing a crash
• tcmalloc itself calls library code that may have already resolved its new/delete symbols to libc's whereas direct calls to new/delete in tcmalloc are always resolved to its own implementation.

To illustrate, compile&run this C++ program (fix the path to libleveldb)

#include <dlfcn.h>
#include <string>
int main() {
    std::string s; // This fixes libstdc++ new/delete to use malloc/free. No crash without it!
    dlopen("/usr/lib/x86_64-linux-gnu/libleveldb.so", RTLD_NOW); // directly loading tcmalloc causes the same problem
    return 0;
}

This bug had been hit many times, looks like:

[src/tcmalloc.cc:283] Attempt to free invalid pointer 0x562806a73fb0

commercialhaskell/stack#2926

https://stackoverflow.com/questions/50132801/debugging-a-failure-to-access-libleveldb-with-c-sharp-on-osx

junyanz/interactive-deep-colorization#26

lamdu/lamdu#104 (comment)

I suggest removing tcmalloc references in leveldb. Document that it is recommended for use in the user application, instead. This will allow safe late-loading of leveldb.

[yairchu]

In some more detail, the crash often happens inside MallocExtension::Initialize(), in the code inside #ifdef __GLIBC__, so reproduction with the example program above is possible under Linux, but on macOS, for example, it doesn't cause a crash.