unable to statically link tcmalloc in a program that forks

[bassam]

If a program calls fork there is one more reference to libc's malloc.o that would need to be implemented for the linker to drop malloc.o. Here's a simple repro

#include <stdlib.h>
#include <unistd.h>

int main() {
  void *i = malloc(0);
  fork();
  return 0;
}

> g++ test.c -o test -static -ltcmalloc_minimal -lpthread -Wl,-Map,test.map
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_malloc':
(.text+0x6170): multiple definition of `__libc_malloc'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../../lib/libtcmalloc_minimal.a(libtcmalloc_minimal_la-tcmalloc.o):(google_malloc+0x0): first defined here
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_malloc':
(.text+0x6170): multiple definition of `malloc'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../../lib/libtcmalloc_minimal.a(libtcmalloc_minimal_la-tcmalloc.o):(google_malloc+0x0): first defined here
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_free':
(.text+0x6510): multiple definition of `__libc_free'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../../lib/libtcmalloc_minimal.a(libtcmalloc_minimal_la-tcmalloc.o):(google_malloc+0x3e10): first defined here
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_free':
(.text+0x6510): multiple definition of `free'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../../lib/libtcmalloc_minimal.a(libtcmalloc_minimal_la-tcmalloc.o):(google_malloc+0x3e10): first defined here
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_realloc':
(.text+0x6720): multiple definition of `__libc_realloc'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../../lib/libtcmalloc_minimal.a(libtcmalloc_minimal_la-tcmalloc.o):(google_malloc+0x7e0): first defined here
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_realloc':
(.text+0x6720): multiple definition of `realloc'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../../lib/libtcmalloc_minimal.a(libtcmalloc_minimal_la-tcmalloc.o):(google_malloc+0x7e0): first defined here
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_memalign':
(.text+0x6b00): multiple definition of `__libc_memalign'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../../lib/libtcmalloc_minimal.a(libtcmalloc_minimal_la-tcmalloc.o):(google_malloc+0x3140): first defined here
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_calloc':
(.text+0x6d50): multiple definition of `__libc_calloc'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../../lib/libtcmalloc_minimal.a(libtcmalloc_minimal_la-tcmalloc.o):(google_malloc+0x2970): first defined here
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_valloc':
(.text+0x8060): multiple definition of `__libc_valloc'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../../lib/libtcmalloc_minimal.a(libtcmalloc_minimal_la-tcmalloc.o):(google_malloc+0x3250): first defined here
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_pvalloc':
(.text+0x82f0): multiple definition of `__libc_pvalloc'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../../lib/libtcmalloc_minimal.a(libtcmalloc_minimal_la-tcmalloc.o):(google_malloc+0x32e0): first defined here
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__posix_memalign':
(.text+0x8fe0): multiple definition of `__posix_memalign'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../../lib/libtcmalloc_minimal.a(libtcmalloc_minimal_la-tcmalloc.o):(google_malloc+0x31c0): first defined here
collect2: error: ld returned 1 exit status

and if you look at test.map produced by the linker:

/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(malloc.o)
                              /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(fork.o) (__malloc_fork_lock_parent)

[tuliom]

That's caused by glibc bug #20432.
It was already fixed in glibc 2.25.

[alk]

malloc hooks removal commit is not related at all. I can see this issue in earlier gperftools versions.

And linking gnu libc statically is not exactly recommended option anyways.

It is quite possible that glibc 2.25 indeed fixes the problem on it's end.

Closing as it is not seemingly gperftools problem.

[alk]

Looks like you're doing it for rookd. Not being expert, let me recommend again against trying to link libc statically. It is ok to link everything else statically (and linking tcmalloc statically is great idea), but even at Google with it's massive focus on efficiency, people don't link libc statically.

[bassam]

yes this for rookd. I found a relatively easy workaround https://github.com/rook/rook/blob/master/pkg/cephmgr/cephd/malloc.go for our scenarios, in case anyone else sees this.

I know that statically linking libc is not recommended, but when running in small containers I think the alternatives of bringing in a whole distro is not much better. I'm aware of the static nss issue, are there others that come up as gotchas?