Unable to compile with -static #442

bassam · 2016-08-03T19:31:06Z

I'm unable to compile a simple static program with jemalloc. I'm using the jemalloc-dev package on Ubuntu and I've not compiled it from source. Here's a very simple repro:

#include <stdlib.h>

int main() {
  void *i = malloc(0);
  return 0;
}

When I compile this I get the following:

> g++ test.c -o test -static -ljemalloc -lpthread
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_malloc':
(.text+0x6120): multiple definition of `malloc'
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libjemalloc.a(jemalloc.o):(.text+0x27d0): first defined here
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_free':
(.text+0x6620): multiple definition of `free'
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libjemalloc.a(jemalloc.o):(.text+0xc80): first defined here
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_realloc':
(.text+0x67e0): multiple definition of `realloc'
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libjemalloc.a(jemalloc.o):(.text+0x1510): first defined here
collect2: error: ld returned 1 exit status

FWIW, when using tcmalloc it compiles fine:

> g++ test.c -o test -static -ltcmalloc_minimal -lpthread
> ./test

Do I need to build jemalloc with different options to enable this?

The text was updated successfully, but these errors were encountered:

bassam · 2016-08-13T05:57:18Z

I get the same error:

> g++ test.c /usr/lib/x86_64-linux-gnu/libjemalloc.a -o test -static -lpthread
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_malloc':
(.text+0x6120): multiple definition of `malloc'
/usr/lib/x86_64-linux-gnu/libjemalloc.a(jemalloc.o):(.text+0x27d0): first defined here
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_free':
(.text+0x6620): multiple definition of `free'
/usr/lib/x86_64-linux-gnu/libjemalloc.a(jemalloc.o):(.text+0xc80): first defined here
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_realloc':
(.text+0x67e0): multiple definition of `realloc'
/usr/lib/x86_64-linux-gnu/libjemalloc.a(jemalloc.o):(.text+0x1510): first defined here
collect2: error: ld returned 1 exit status

bassam · 2016-08-13T06:05:53Z

same issue using default jemalloc options:

git clone https://github.com/jemalloc/jemalloc.git
cd jemalloc
./autogen.sh
./configure
make

then using libjemalloc.a that I just built:

> g++ test.c jemalloc/lib/libjemalloc.a -o test -static -lpthread
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_malloc':
(.text+0x6120): multiple definition of `malloc'
jemalloc/lib/libjemalloc.a(jemalloc.o):/home/bassam/Projects/jemalloc/src/jemalloc.c:1464: first defined here
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_free':
(.text+0x6620): multiple definition of `free'
jemalloc/lib/libjemalloc.a(jemalloc.o):/home/bassam/Projects/jemalloc/src/jemalloc.c:1835: first defined here
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_realloc':
(.text+0x67e0): multiple definition of `realloc'
jemalloc/lib/libjemalloc.a(jemalloc.o):/home/bassam/Projects/jemalloc/src/jemalloc.c:1758: first defined here
collect2: error: ld returned 1 exit status

Does this work for you?

kspinka · 2016-08-13T06:08:17Z

Drop the -static (don't worry, this is still including static versions of your libs, just not the system libs). This is my preferred approach:

g++ -o test test.c jemalloc/lib/libjemalloc.a -lpthread

bassam · 2016-08-13T06:13:45Z

Thanks the -Ofast trick works. I'm building a golang binary that includes jemalloc and would like it to be completely static (as most golang binaries are).

Can you shed some light onto as to why -Ofast get around this error? Also why is the behavior different between tcmalloc and jemalloc in this regard?

I appreciate all your help!

bassam · 2016-08-13T06:21:49Z

@kspinka also it looks like with -Ofast none of the jemalloc symbols make it into the binary:

> nm -gC test | grep je_
>

kspinka · 2016-08-13T06:25:55Z

Ok, that's not the right thing to do. Forget you even saw that dirty -Ofast trick.

I really don't think it's wise to statically link libc and libpthread, as they may vary in their implementations across distros and versions (Linux I assume?).

So all that being said, the right way to do what you want to do, if you insist on this totally static situation, is to enable the jemalloc prefix; compile with: --with-jemalloc-prefix=je_

And then either use the je_malloc() type functions directly, or use glibc's malloc hooks to intercept your malloc calls: http://www.gnu.org/software/libc/manual/html_node/Hooks-for-Malloc.html

Here's an example I found (credit to Greg Dhuse):

#include <stdlib.h>
#include <jemalloc/jemalloc.h>

/* Prototypes for __malloc_hook, __free_hook */
#include <malloc.h>

/* Prototypes for our hooks.  */
static void my_init_hook (void);
static void *my_malloc_hook (size_t, const void *);
static void my_free_hook (void*, const void *);

/* Override initializing hook from the C library. */
void (*__malloc_initialize_hook) (void) = my_init_hook;


static void
my_init_hook (void)
{
    __malloc_hook = my_malloc_hook;
    __free_hook = my_free_hook;
}

static void *
my_malloc_hook (size_t size, const void *caller)
{
    return je_malloc(size);
}

static void
my_free_hook (void *ptr, const void *caller) 
{
    je_free(ptr);
}


int main() 
{
    je_malloc_stats_print(NULL, NULL, NULL);

    void *foo = malloc(123);
    je_malloc_stats_print(NULL, NULL, NULL);
    free(foo);

    return 0;
}

bassam · 2016-08-13T06:32:57Z

thanks I'll try this out. Does tcmalloc include similar hooks? I'm curious as to why it works with tcmalloc without any changes.

kspinka · 2016-08-13T07:27:09Z

I'm pretty sure they (tcmalloc) take advantage of the aliasing system in gcc to override the "weakly" defined symbols in libc with their own "strongly" defined ones. This way the linker, when seeing both, will prefer the strong ones and ignore the weak ones.

...
#define ALIAS(tc_fn)   __attribute__ ((alias (#tc_fn), used))
...
extern "C" {
  void* malloc(size_t size) __THROW               ALIAS(tc_malloc);
  void free(void* ptr) __THROW                    ALIAS(tc_free);
  void* realloc(void* ptr, size_t size) __THROW   ALIAS(tc_realloc);
  void* calloc(size_t n, size_t size) __THROW     ALIAS(tc_calloc);
  void cfree(void* ptr) __THROW                   ALIAS(tc_cfree);
  void* memalign(size_t align, size_t s) __THROW  ALIAS(tc_memalign);
  void* valloc(size_t size) __THROW               ALIAS(tc_valloc);
  void* pvalloc(size_t size) __THROW              ALIAS(tc_pvalloc);
  int posix_memalign(void** r, size_t a, size_t s) __THROW
      ALIAS(tc_posix_memalign);
#ifndef __UCLIBC__
  void malloc_stats(void) __THROW                 ALIAS(tc_malloc_stats);
#endif
  int mallopt(int cmd, int value) __THROW         ALIAS(tc_mallopt);
#ifdef HAVE_STRUCT_MALLINFO
  struct mallinfo mallinfo(void) __THROW          ALIAS(tc_mallinfo);
#endif
  size_t malloc_size(void* p) __THROW             ALIAS(tc_malloc_size);
#if defined(__ANDROID__)
  size_t malloc_usable_size(const void* p) __THROW
         ALIAS(tc_malloc_size);
#else
  size_t malloc_usable_size(void* p) __THROW      ALIAS(tc_malloc_size);
#endif
}   // extern "C"

I'm not sure why jemalloc doesn't use this. It's a good question.

jasone · 2016-09-07T17:40:49Z

Using __attribute__((alias(...), used)) will be a bit messier in jemalloc due to the existing name mangling support, but it's workable.

problematic when statically linking cf. jemalloc/jemalloc#442

djwatson · 2016-10-28T20:59:11Z

It turns out to not be a weak symbol issue, at least since ~2006

https://sourceware.org/bugzilla/show_bug.cgi?id=2765

Implementing all the __libc * symbols, like tcmalloc, allows the linker to drop all of glibc's malloc.o, and statically link correctly. Patch attached

glibc defines its malloc implementation with several weak and strong symbols: strong_alias (__libc_calloc, __calloc) weak_alias (__libc_calloc, calloc) strong_alias (__libc_free, __cfree) weak_alias (__libc_free, cfree) strong_alias (__libc_free, __free) strong_alias (__libc_free, free) strong_alias (__libc_malloc, __malloc) strong_alias (__libc_malloc, malloc) The issue is not with the weak symbols, but that other parts of glibc depend on __libc_malloc explicitly. Defining them in terms of jemalloc API's allows the linker to drop glibc's malloc.o completely from the link, and static linking no longer results in symbol collisions. Another wrinkle: jemalloc during initialization calls sysconf to get the number of CPU's. GLIBC allocates for the first time before setting up isspace (and other related) tables, which are used by sysconf. Instead, use the pthread API to get the number of CPUs with GLIBC, which seems to work. This resolves #442.

bassam · 2016-12-12T17:18:53Z

@djwatson and @jasone thanks for the fix. I can verify it works. However, if a program calls fork there is one more reference to libc's malloc.o that would need to be implemented for the linker to drop malloc.o. Here's a simple repro

#include <stdlib.h>
#include <unistd.h>

int main() {
  void *i = malloc(0);
  fork();
  return 0;
}

> g++ test.c -o test -static -ljemalloc -lpthread -Wl,-Map,test.map
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_malloc':
(.text+0x6170): multiple definition of `malloc'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libjemalloc.a(jemalloc.o):(.text+0x27d0): first defined here
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_free':
(.text+0x6510): multiple definition of `free'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libjemalloc.a(jemalloc.o):(.text+0xc80): first defined here
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(malloc.o): In function `__libc_realloc':
(.text+0x6720): multiple definition of `realloc'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libjemalloc.a(jemalloc.o):(.text+0x1510): first defined here
collect2: error: ld returned 1 exit status

and if you look at test.map produced by the linker:

/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(malloc.o)
                              /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(fork.o) (__malloc_fork_lock_parent)

jasone · 2016-12-12T17:38:38Z

@bassam, I can see how this is probably an issue for jemalloc, but I'm confused by the repro output you provided being specific to tcmalloc rather than jemalloc. Is there any significance to this?

bassam · 2016-12-12T17:39:24Z

@jasone sorry cut&paste issue fixed. FWIW, the same issue exists with tcmalloc.

jasone · 2016-12-12T17:40:31Z

Cool, thanks for the clarification.

tuliom · 2016-12-12T19:58:22Z

I think the fork issue is caused by a glibc bug which was fixed in glibc 2.25.

bassam · 2016-12-12T20:11:50Z

It does look like its fixed in 2.25, however, thats still in development and might take a long time to be available on popular distros. I wonder if it would be useful to patch this on the jemalloc side, something like:

--- src/jemalloc.c	2016-12-12 13:08:54.337306073 -0500
+++ src/jemalloc.c	2016-12-12 13:09:08.000000000 -0500
@@ -2072,6 +2072,9 @@
 void	*__libc_valloc(size_t size) PREALIAS(je_valloc);
 int	__posix_memalign(void** r, size_t a, size_t s)
     PREALIAS(je_posix_memalign);
+void	__malloc_fork_lock_parent(void){};
+void	__malloc_fork_unlock_parent(void){};
+void	__malloc_fork_unlock_child(void){};
 #undef PREALIAS
 #undef ALIAS

bassam · 2016-12-13T14:08:23Z

FWIW, I found a simple workaround https://github.com/rook/rook/blob/master/pkg/cephmgr/cephd/malloc.go. Feel free to close this if you'd like. It really is a glibc issue.

jasone · 2016-12-13T17:05:32Z

Okay, closing. Thank you for following up!

jasone added the portability label Sep 7, 2016

jasone added this to the 5.0.0 milestone Sep 7, 2016

mlin added a commit to vgteam/vg that referenced this issue Oct 19, 2016

use jemalloc in dynamically-linked executable

296fcbc

problematic when statically linking cf. jemalloc/jemalloc#442

mlin added a commit to vgteam/vg that referenced this issue Oct 19, 2016

use jemalloc in dynamically-linked executable

10f18d7

problematic when statically linking cf. jemalloc/jemalloc#442

djwatson mentioned this issue Oct 28, 2016

Support static linking of jemalloc with glibc #481

Closed

jasone modified the milestones: 4.3.0, 5.0.0 Oct 28, 2016

jasone closed this as completed in 8309388 Oct 28, 2016

bassam mentioned this issue Oct 31, 2016

Enable jemalloc and tcmalloc support in rookd rook/rook#130

Closed

jasone reopened this Dec 12, 2016

jasone modified the milestones: 5.0.0, 4.3.0 Dec 12, 2016

jasone modified the milestones: 4.4.0, 5.0.0, 4.3.0 Dec 13, 2016

jasone closed this as completed Dec 13, 2016

Kha mentioned this issue Mar 14, 2018

[WIP] Turn on jemalloc, turn off custom allocators by default leanprover/lean3#1949

Closed

lostsnow mentioned this issue Mar 24, 2019

compiler error with libjemalloc when use -DSTATIC=ON fluffos/fluffos#454

Closed

madebr mentioned this issue Aug 9, 2020

Feature/mimalloc conan-io/conan-center-index#2463

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to compile with -static #442

Unable to compile with -static #442

bassam commented Aug 3, 2016

bassam commented Aug 13, 2016

bassam commented Aug 13, 2016 •

edited

kspinka commented Aug 13, 2016 •

edited

bassam commented Aug 13, 2016

bassam commented Aug 13, 2016

kspinka commented Aug 13, 2016 •

edited

bassam commented Aug 13, 2016

kspinka commented Aug 13, 2016 •

edited

jasone commented Sep 7, 2016

djwatson commented Oct 28, 2016 •

edited

bassam commented Dec 12, 2016 •

edited

jasone commented Dec 12, 2016

bassam commented Dec 12, 2016

jasone commented Dec 12, 2016

tuliom commented Dec 12, 2016

bassam commented Dec 12, 2016

bassam commented Dec 13, 2016

jasone commented Dec 13, 2016

Unable to compile with -static #442

Unable to compile with -static #442

Comments

bassam commented Aug 3, 2016

bassam commented Aug 13, 2016

bassam commented Aug 13, 2016 • edited

kspinka commented Aug 13, 2016 • edited

bassam commented Aug 13, 2016

bassam commented Aug 13, 2016

kspinka commented Aug 13, 2016 • edited

bassam commented Aug 13, 2016

kspinka commented Aug 13, 2016 • edited

jasone commented Sep 7, 2016

djwatson commented Oct 28, 2016 • edited

bassam commented Dec 12, 2016 • edited

jasone commented Dec 12, 2016

bassam commented Dec 12, 2016

jasone commented Dec 12, 2016

tuliom commented Dec 12, 2016

bassam commented Dec 12, 2016

bassam commented Dec 13, 2016

jasone commented Dec 13, 2016

bassam commented Aug 13, 2016 •

edited

kspinka commented Aug 13, 2016 •

edited

kspinka commented Aug 13, 2016 •

edited

kspinka commented Aug 13, 2016 •

edited

djwatson commented Oct 28, 2016 •

edited

bassam commented Dec 12, 2016 •

edited