relocation error: symbol pthread_sigmask version GLIBC_2.2.5 not defined in file libc.so.6 with link time reference

[lithdew]

Encountered this error while trying to run CI tests on Github Actions for a Zig project, using the Zig nightly bundle provided here: https://ziglang.org/builds/zig-linux-x86_64-0.7.1+dfacac916.tar.xz. Linking with musl makes the problem go away.

The latest glibc version available in Github Actions is 2.27.

root@dew:/github/workspace# ldd ./zig-cache/o/ea49ec9c878272fe2410dfdf1fc66939/test
        linux-vdso.so.1 (0x00007ffe5b38e000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa3c72c5000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa3c6f27000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa3c6d08000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa3c6b04000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fa3c68fc000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fa3c76b6000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fa3c66f9000)
root@dew:/github/workspace# nm -D /lib/x86_64-linux-gnu/libpthread.so.0 | grep pthread_sig
000000000000f290 T pthread_sigmask
000000000000f3e0 T pthread_sigqueue
root@dew:/github/workspace# nm -D /lib/x86_64-linux-gnu/libc.so.6 | grep pthread_sig
root@dew:/github/workspace# 

Reproduction requires an 18.2GB Docker image that emulates the Github Actions CI Environment, which can be found here: https://hub.docker.com/layers/nektos/act-environments-ubuntu/18.04/images/sha256-4d991875312e2ad084ff5350aa193e8a7dbdc4c144476d48df6a69e1e3404427

References to pthread symbols in built executable:

root@dew:/github/workspace# objdump -T ./zig-cache/o/ea49ec9c878272fe2410dfdf1fc66939/test | grep pthread
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.3.2 pthread_cond_destroy
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.3.2 pthread_cond_init
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.3.2 pthread_cond_signal
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.3.2 pthread_cond_wait
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_create
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_getspecific
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_join
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_key_create
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_key_delete
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.12  pthread_mutex_consistent
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_mutex_destroy
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_mutex_init
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_mutex_lock
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_mutex_unlock
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_mutexattr_destroy
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_mutexattr_init
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_mutexattr_setpshared
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.12  pthread_mutexattr_setrobust
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_self
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_setspecific
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_sigmask
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_atfork
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_mutex_trylock

[mikdusan]

hello.zig reduction (fails on issue host, works for my host)

const std = @import("std");

// hack to force symbol requirement
extern "c" var pthread_sigmask: u32;

pub fn main() !void {
    const stdout = std.io.getStdOut().writer();
    try stdout.print("Hello, {}!\n", .{"world"});
    try stdout.print("Hello, {d}!\n", .{pthread_sigmask});
}

output from readelf -a hello on archlinux w/ glibc 2.32; snipped to sections/headers that have pthread_sigmask

Dynamic section at offset 0x38f70 contains 27 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libutil.so.1]
[SNIP]

Relocation section '.rela.dyn' at offset 0x850 contains 5 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
00000023b138  000100000006 R_X86_64_GLOB_DAT 0000000000000000 __libc_start_main@GLIBC_2.2.5 + 0
00000023b140  000200000006 R_X86_64_GLOB_DAT 0000000000000000 __gmon_start__ + 0
00000023a998  000300000001 R_X86_64_64       0000000000000000 __errno_location@GLIBC_2.2.5 + 0
00000023b150  000900000006 R_X86_64_GLOB_DAT 0000000000000000 environ@GLIBC_2.2.5 + 0
00000023b148  001400000006 R_X86_64_GLOB_DAT 0000000000000000 pthread_sigmask@GLIBC_2.2.5 + 0

Symbol table '.symtab' contains 928 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
[SNIP]
   919: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_mutex_lock
   920: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_mutex_unlock
   921: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_sigmask

your quick grep on discord showed 3 matches to pthread_sigmask for the command. Can you post here?

[mikdusan]

@daurnimator found this:

https://sourceware.org/pipermail/glibc-cvs/2020q2/069412.html

TL;DR: as of glibc 2.32 symbol for pthread_sigmask moved from libpthread → libc and it might be the trigger for this issue; issue host is a glibc 2.27 system.

[lithdew]

your quick grep on discord showed 3 matches to pthread_sigmask for the command. Can you post here?

Sure:

root@dew:/github/workspace# readelf -a /root/.cache/zig/o/9ef07c7e75e1fe8f2ae6df075cfa21ed/hello | grep pthread_sigmask
00000023adb8  001400000006 R_X86_64_GLOB_DAT 0000000000000000 pthread_sigmask@GLIBC_2.2.5 + 0
    20: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_sigmask@GLIBC_2.2.5 (2)
   904: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_sigmask

[amosbird]

@daurnimator found this:

https://sourceware.org/pipermail/glibc-cvs/2020q2/069412.html

TL;DR: as of glibc 2.32 symbol for pthread_sigmask moved from libpthread → libc and it might be the trigger for this issue; issue host is a glibc 2.27 system.

Hmm, is there any way to workaround this?

[ajbouh]

I also faced this issue (and suspect the same underlying cause) but was unable to find a reasonable workaround.

[motiejus]

I am getting this too on an internal project (which requires use of glibc 2.19):

id: relocation error: /lib/x86_64-linux-gnu/libnss_uber.so.2: symbol pthread_sigmask, version GLIBC_2.2.5 not defined in file libc.so.6 with link time reference

+1 on "would be nice to find a reasonable workaround". I will try to find some time to read about linkers and hopefully find one.

[mjonaitis1]

It seems that the problem might be related more to dynamically loaded libraries and incorrect symbol mapping to library from zig side.

Minimal reproducible c code example in debian jessie:
a.c file compiled by running: zig cc --target=x86_64-linux-gnu.2.19 -o a a.c

#define _GNU_SOURCE
#include <dlfcn.h>

int main()
{
  void *lib = dlopen("./libshared.so", RTLD_LAZY);
  void (*exported_func)(void);
  exported_func = (void (*)(void))dlsym(lib, "exported_func");
  exported_func();

  return 0; 
}

sharedlib.c compiled into libshared.so by running the following commands:

zig cc --target=x86_64-linux-gnu.2.19  -c -Wall -Werror -fpic sharedlib.c -o sharedlib.o
zig cc  -shared -o libshared.so sharedlib.o -lpthread

#include <signal.h>
#include <stdio.h>

void exported_func()
{
    printf("Address of pthread_sigmask: %p\n", pthread_sigmask);
}

Output of readelf -Wa libshared.so | grep -Ei "pthread_sigmask|GLIBC.*Version" -1

0000000000002610  0000000100000006 R_X86_64_GLOB_DAT      0000000000000000 __gmon_start__ + 0
0000000000002618  0000000200000006 R_X86_64_GLOB_DAT      0000000000000000 pthread_sigmask@GLIBC_2.2.5 + 0

--
     1: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__
     2: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_sigmask@GLIBC_2.2.5 (2)
     3: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND printf@GLIBC_2.2.5 (2)
--
     6: 0000000000001430    24 FUNC    GLOBAL DEFAULT   12 exported_func
     7: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_sigmask
     8: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND printf
-- .gnu.version_r 
  000000: Version: 1  File: libc.so.6  Cnt: 1
  0x0010:   Name: GLIBC_2.2.5  Flags: none  Version: 2

It seems that zig maps pthread_sigmask to libc library in .gnu.version_r section, but this symbol is still in libpthread in glibc 2.19. I assume that this fns.txt file is used to map symbols to specific libraries, because once I change this line pthread_sigmask c to pthread_sigmask pthread then zig cc correctly maps pthread_sigmask to libpthread and relocation error is gone:

0000000000002630  0000000100000006 R_X86_64_GLOB_DAT      0000000000000000 __gmon_start__ + 0
0000000000002638  0000000200000006 R_X86_64_GLOB_DAT      0000000000000000 pthread_sigmask@GLIBC_2.2.5 + 0

--
     1: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__
     2: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_sigmask@GLIBC_2.2.5 (2)
     3: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND printf@GLIBC_2.2.5 (3)
--
     6: 0000000000001450    24 FUNC    GLOBAL DEFAULT   12 exported_func
     7: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_sigmask
     8: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND printf
--
  000000: Version: 1  File: libpthread.so.0  Cnt: 1
  0x0020:   Name: GLIBC_2.2.5  Flags: none  Version: 2
  0x0010: Version: 1  File: libc.so.6  Cnt: 1
  0x0030:   Name: GLIBC_2.2.5  Flags: none  Version: 3

I assume that this edge case where a symbol can move between libraries might be overlooked and would need to be fixed by either having a separate fns.txt file for each glibc version or by introducing some versioning where there might be 2 lines for symbols like pthread_sigmask with something like pthread_sigmask pthread@2.31 where @2.31 is the last version where symbol is in that library. So when choosing specific glibc target version, correct symbol library mapping would be used.

[andrewrk]

@mjonaitis1 thanks for looking into this. Your suggested solution seems reasonable to me.