New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: build with old glibc so rkt runs on CentOS 6? #1063

Open
zopyx opened this Issue Jun 17, 2015 · 22 comments

Comments

Projects
None yet
@zopyx

zopyx commented Jun 17, 2015

Is there any chance to run rkt 0.6.1 or higher on CentOS 6.6 64 bit?

./rkt: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ./rkt)

@jonboulle

This comment has been minimized.

Contributor

jonboulle commented Jun 17, 2015

Are you using the precompiled binary release of 0.6.1?

On Wed, Jun 17, 2015 at 10:41 AM, Andreas Jung notifications@github.com
wrote:

Is there any chance to run rkt 0.6.1 or higher on CentOS 6.6 64 bit?

./rkt: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ./rkt)


Reply to this email directly or view it on GitHub
#1063.

@zopyx

This comment has been minimized.

zopyx commented Jun 17, 2015

Yes

2015-06-17 19:47 GMT+02:00 Jonathan Boulle notifications@github.com:

Are you using the precompiled binary release of 0.6.1?

On Wed, Jun 17, 2015 at 10:41 AM, Andreas Jung notifications@github.com
wrote:

Is there any chance to run rkt 0.6.1 or higher on CentOS 6.6 64 bit?

./rkt: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by
./rkt)


Reply to this email directly or view it on GitHub
#1063.


Reply to this email directly or view it on GitHub
#1063 (comment).

@zopyx

This comment has been minimized.

zopyx commented Jun 17, 2015

Building rkt myself on CentOS 6.6 did the job. Sorry for the noise.

@zopyx zopyx closed this Jun 17, 2015

@jonboulle

This comment has been minimized.

Contributor

jonboulle commented Jun 17, 2015

Actually you shouldn't need to build it yourself, we should take a look at this.

@jonboulle jonboulle reopened this Jun 17, 2015

@jonboulle jonboulle changed the title from GLIBC_2.14' not found /CentOS 6.6 to rkt does not run on CentOS 6.6 (GLIBC_2.14 not found) Jun 17, 2015

@jonboulle jonboulle added the kind/bug label Jun 17, 2015

@bitshark

This comment has been minimized.

bitshark commented Jul 2, 2015

Maybe this has something to do with how the rkt binary is linked. . . Just FYI the rkt 0.6.1 binary is (mostly) linked against GLIBC 2.2.5 -- except for a single entry in the DST -- memcpy , which is linked against GLIBC 2.14.

Ignore this if you guys are already aware. As a side note, the binary is working fine on my machine (Void Linux).

[anon@void-live rkt-v0.6.1]$ uname -a
Linux void-live 4.0.1_1 1 SMP PREEMPT Wed Apr 29 09:48:41 UTC 2015 x86_64 GNU/Linux

[anon@void-live rkt-v0.6.1]$ ls -al
total 40004
drwxr-xr-x  2 anon anon     4096 Jun 17 14:39 .
drwxrwxr-x 26 anon anon    20480 Jul  2 03:42 ..
-rw-rw-r--  1 anon anon  3762939 Feb 27 07:03 etcd-v2.0.4-linux-amd64.aci
__ -rwxr-xr-x  1 anon anon 16789672 Jun 17 14:39 rkt __
-rw-r--r--  1 anon anon 20378603 Jun 17 14:39 stage1.aci
[anon@void-live rkt-v0.6.1]$ ldd ./rkt
    linux-vdso.so.1 (0x00007fff0d9b1000)
    libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f0acd3e3000)
    libc.so.6 => /usr/lib/libc.so.6 (0x00007f0acd040000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f0acd601000)
[anon@void-live rkt-v0.6.1]$ objdump -T ./rkt | head -n 35

./rkt:     file format elf64-x86-64

DYNAMIC SYMBOL TABLE:
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 free
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_create
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_sigmask
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 abort
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 __errno_location
0000000000000000  w   D  *UND*  0000000000000000              _ITM_deregisterTMCloneTable
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 setenv
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 gai_strerror
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 memset
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fputc
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 __libc_start_main
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_attr_init
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_attr_getstacksize
0000000000000000  w   D  *UND*  0000000000000000              __gmon_start__
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 sigfillset 

0000000000000000 DF UND 0000000000000000 GLIBC_2.14 memcpy

0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 malloc
0000000000000000  w   D  *UND*  0000000000000000              _Jv_RegisterClasses
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 vfprintf
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 unsetenv
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_attr_destroy
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fwrite
0000000000000000  w   D  *UND*  0000000000000000              _ITM_registerTMCloneTable
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 getaddrinfo
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strerror
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 freeaddrinfo
0000000000000000      DO *UND*  0000000000000000  GLIBC_2.2.5 stderr
00000000004017e0 g    DF .text  0000000000000037  Base        x_cgo_malloc
0000000000401b90 g    DF .text  0000000000000037  Base        _cgo_1fe9f74ed0d8_Cfunc_decode
0000000000401520 g    DF .text  0000000000000008  Base        _cgo_632c88804cec_Cfunc_free
0000000000401a30 g    DF .text  0000000000000158  Base        decode
[anon@void-live rkt-v0.6.1]$ 

I have no idea if this is normal, but it seems odd that memcpy() is linked to a different version of glibc than free(). Maybe that's normal idk.

Anyway, so I checked GLIBC symbols in libc.so on my system. . . You can see that GLIBC has two versions of memcpy , which I assume is normal for modern linuxes (prob with exception of CentOS). . .

[anon@void-live rkt-v0.6.1]$ objdump -T /usr/lib/libc.so.6 | grep -w memcpy
0000000000088de0 g   iD  .text  0000000000000051  GLIBC_2.14  memcpy
0000000000083b90 g   iD  .text  0000000000000057 (GLIBC_2.2.5) memcpy

So in my shared libs I've got a 2.14 version of memcpy and a 2.2.5 version of memcpy.

I assume what's causing the problem in this bug, is that CentOS doesn't have memcpy 2.14 in it's shared lib, so the CentOS dynamic linker craps out.

There's a stack overflow dot com answer to fix this, assuming this is indeed the problem . . . Basically you can force rkt to use memcpy version 2.2.5 during compile time . Like this supposedly...

http://stackoverflow.com/questions/4032373/linking-against-an-old-version-of-libc-to-provide-greater-application-coverage

[anon@void-live rkt-v0.6.1]$ cat memcpy_test_bug.c 
#include <limits.h>
#include <stdlib.h>
#include <string.h>

//__asm__(".symver memcpy,memcpy@GLIBC_2.14");  commented out

int main () {
    char *src = "blah";
    char dst[8];
    memcpy(dst, src, 5);
}

[anon@void-live rkt-v0.6.1]$ gcc -o memcpy_test_bug memcpy_test_bug.c 
[anon@void-live rkt-v0.6.1]$ ldd ./memcpy_test_bug 
    linux-vdso.so.1 (0x00007ffec6db2000)
    libc.so.6 => /usr/lib/libc.so.6 (0x00007fea7fa75000)
    /lib/ld-linux-x86-64.so.2 (0x00007fea7fe18000)
[anon@void-live rkt-v0.6.1]$ objdump -T memcpy_test_bug

memcpy_test_bug:     file format elf64-x86-64

DYNAMIC SYMBOL TABLE:
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 __libc_start_main
0000000000000000  w   D  *UND*  0000000000000000              __gmon_start__
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.14  memcpy  <-- here

TWO DIFFERENT GLIBC VERSIONS AFTER COMPILE, WITH MEMCPY BEING 2.14... VERY SIMILAR TO BUG.

OK FIX! ...

[anon@void-live rkt-v0.6.1]$ cat memcpy_test.c 
#include <limits.h>
#include <stdlib.h>
#include <string.h>

__asm__(".symver memcpy,memcpy@GLIBC_2.2.5");  //uncomment to force use of 2.2.5 memcpy

int main () {
    char *src = "blah";
    char dst[8];
    memcpy(dst, src, 5);
}

[anon@void-live rkt-v0.6.1]$ gcc -o memcpy_test memcpy_test.c
[anon@void-live rkt-v0.6.1]$ ./memcpy_test
[anon@void-live rkt-v0.6.1]$ objdump -T memcpy_test

memcpy_test:     file format elf64-x86-64

DYNAMIC SYMBOL TABLE:
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 memcpy
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 __libc_start_main
0000000000000000  w   D  *UND*  0000000000000000              __gmon_start__


--^ PEW all 2.2.5 GLIBC  
@bitshark

This comment has been minimized.

bitshark commented Jul 2, 2015

Also the docs say
rkt is statically linked and does not require any dynamic libraries to be installed.
but this can't be true b/c of output from ldd.

@iaguis

This comment has been minimized.

Member

iaguis commented Jul 2, 2015

Thanks for the detailed report!

To force GLIBC_2.2.5 I guess we'd need to either use a cgo program and inline assembler or try this. I guess it's easier to just build rkt on an old distro with an old glibc.

Also the docs say
rkt is statically linked and does not require any dynamic libraries to be installed.
but this can't be true b/c of output from ldd.

That's right and I don't think we can link it statically because we use cgo...

@bitshark

This comment has been minimized.

bitshark commented Jul 2, 2015

Oh, k no problem. I was actually just messing around with the build script, but I'm new to Go so it's gonna take some experimenting I think.

Good points -- heh... I honestly don't know what the correct answer is here, altho if you wanted to get rid of this bug , I think the stack overflow answer is pretty good... One guy from SO suggests that if you decide to just use the 2.2.5 version of memcpy (like the rest of the symbols in the rkt binary)... , you should just implement a thin wrapper to memmove 2.2.5 in Go or C.

Just link memcpy statically - pull memcpy.o out of libc.a ar x /path/to/libc.a memcpy.o 
(whatever version - memcpy is pretty much a standalone function) and include it in 
your final link. Note that static linking may complicate licensing issues if your project 
is distributed to the public and not open-source.

Alternatively, you could simply implement memcpy yourself, though the hand-tuned 
assembly version in glibc is likely to be more efficient

Note that memcpy@GLIBC_2.2.5 is mapped to memmove (old versions of memcpy 
consistently copied in a predictable direction, which led to it sometimes being misused 
when memmove should have been used), and this is the only reason for the version 
bump - you could simply replace memcpy with memmove in your code for this specific 
case.

Or you could go to static linking, or you could ensure that all systems on your network 
have the same or better version than your build machine.

http://stackoverflow.com/questions/8823267/linking-against-older-symbol-version-in-a-so-file

Good luck!

@jonboulle jonboulle referenced this issue Aug 19, 2015

Open

*: package rkt for CentOS #1305

1 of 2 tasks complete
@jaybuff

This comment has been minimized.

jaybuff commented Sep 16, 2015

#1380 would resolve this.

@jaybuff

This comment has been minimized.

jaybuff commented Sep 16, 2015

The move to autoconf which happened sometime after v0.6.1 requires autoconf 2.68 which isn't easily available for CentOS (afaict). So building it yourself is no longer an easy work around to this issue.

It's unclear to me if autoconf 2.68 is really required or we can drop that down to 2.63 which ships with centos 6.

@jonboulle

This comment has been minimized.

Contributor

jonboulle commented Oct 20, 2015

@jaybuff is this resolved now?

@jaybuff

This comment has been minimized.

jaybuff commented Oct 21, 2015

No, the released 0.9.0 has the same issue. I can work around it by building locally. I read through what @bitshark said and it looks like his suggestion to force rkt to use memcpy version 2.2.5 during compile time would work by adding a line like this:

__asm__(".symver memcpy,memcpy@GLIBC_2.2.5");

but I don't know how that C code translates into go.

@jonboulle

This comment has been minimized.

Contributor

jonboulle commented Oct 21, 2015

At this stage I think I'm more inclined to say it's a requirement to BYO if
running on such an old platform; does that seem reasonable?

On Tue, Oct 20, 2015 at 5:28 PM, Jay Buffington notifications@github.com
wrote:

No, the released 0.9.0 has the same issue. I can work around it by
building locally. I read through what @bitshark
https://github.com/bitshark said and it looks like his suggestion to
force rkt to use memcpy version 2.2.5 during compile time would work by
adding a line like this:

asm(".symver memcpy,memcpy@GLIBC_2.2.5");

but I don't know how that C code translates into go.


Reply to this email directly or view it on GitHub
#1063 (comment).

@jaybuff

This comment has been minimized.

jaybuff commented Oct 21, 2015

It's not that old. RHEL 6.5 was released November 2013.

The real problem is that you have a binary on the releases page that doesn't work on systems with glibc older than 2.14. If produce the binary for that releases page in an ACI with an old glibc (like RHEL 6) then you'll have a binary that works on older and new systems.

@jaybuff

This comment has been minimized.

jaybuff commented Oct 21, 2015

The more I think about it the more I think your build system should just be a rkt run command.

@jonboulle

This comment has been minimized.

Contributor

jonboulle commented Oct 21, 2015

You are right. I think there's an issue for this. Let me fish it out later.
On Oct 20, 2015 17:51, "Jay Buffington" notifications@github.com wrote:

The more I think about it the more I think your build system should just
be a rkt run command.


Reply to this email directly or view it on GitHub
#1063 (comment).

@adfernandes

This comment has been minimized.

Contributor

adfernandes commented Nov 8, 2015

👍 for centos6 support!

@alban alban modified the milestones: v0.12.0, v0.11.0 Nov 12, 2015

@alban alban modified the milestones: v0.13.0, v0.12.0 Nov 26, 2015

@alban alban modified the milestones: v0.15.0, v0.14.0 Dec 18, 2015

@iaguis iaguis added the help wanted label Jan 7, 2016

@iaguis iaguis modified the milestones: vfuture, v0.15.0 Jan 7, 2016

@jonboulle jonboulle changed the title from rkt does not run on CentOS 6.6 (GLIBC_2.14 not found) to *: build with old glibc so rkt runs on CentOS 6? Feb 4, 2016

@jonboulle jonboulle modified the milestones: v1.2.0, vfuture Feb 4, 2016

@sheerun

This comment has been minimized.

sheerun commented Mar 16, 2016

I have similar issue.. @zopyx @jaybuff do you have rkt build working for centos 6.x lying around?

@iaguis iaguis modified the milestones: v1.3.0, v1.2.0 Mar 18, 2016

@alban alban modified the milestones: vfuture, v1.3.0 Mar 31, 2016

@lucab

This comment has been minimized.

Member

lucab commented Apr 18, 2016

Linkage to memcpy is coming from the cgo portion of zappy package: https://github.com/cznic/zappy/blob/master/encode_cgo.go#L39.
The simple and portable way to avoid this is to pass a purego tag when building (maybe gated behind a configure switch). This will impact performance, but also reduce the surface of unsafe C code. Currently, zappy and ql understand this tag.

(The more complex way is to patch zappy with an ad-hoc .symver renaming, but this is not really portable)

@jonboulle

This comment has been minimized.

Contributor

jonboulle commented Apr 19, 2016

@lucab volunteering for this one? :-)

@ErnestG4

This comment has been minimized.

ErnestG4 commented Oct 3, 2016

Bump in ref to https://coreos.zendesk.com/agent/tickets/11548
Quote:

"Appears to be dynamically linked against GLIBC_2.14 which causes problems trying to use rkt on other systems. ...Would be nice if it were actually statically built."

"Attempting to Compile rkt on Centos6 my self

Had to quickly patch an empty if expression:

./configure: line 5668: syntax error near unexpected token `else'
./configure: line 5668: `else'`

Had to emulate realpath since it didn't exist:

$ make
/bin/bash: realpath: command not found
basename: missing operand
Try `basename --help' for more information.

echo 'readlink -f "$@"' > realpath &&
chmod +x realpath &&
export PATH=$PATH:.

The old version of git there does not support the -C option.

args=()
while [[ $1 ]]; do
if [[ $1 == -C ]]; then
shift
cd "$1"
else
args+=("$1")
fi
shift
done
exec /usr/bin/git "${args[@]}"

In the end it seems like I need a 3.x kernel:

$ sudo ./build-rkt-1.15.0/target/bin/rkt --insecure-options=image run --net=host docker://busybox
image: using image from local store for image name coreos.com/rkt/stage1-coreos:1.15.0
image: using image from local store for url docker://busybox
run: disabling overlay support: "overlay entry not present in /proc/filesystems"
stage1: couldn't mount the host cgroups: mkdir /sys/fs/cgroup: no such file or directory

#1529

fly stage 1 even requires GLIBC_2.14 because I believe it yanks files out of a prebuilt pxe image!!!

/var/lib/rkt/pods/run/97f3874a-edb2-4b5a-bcbf-ed46227cb2e8/stage1/rootfs/run: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /var/lib/rkt/pods/run/97f3874a-edb2-4b5a-bcbf-ed46227cb2e8/stage1/rootfs/run)

I was looking into using --with-stage1-flavors=src but it fails complaining that autoconf has to be greater than 2.3 iirc and centos has a 2.68 package but it's one of those that installs side by side, yet even with symlinking and $PATH manipulation it's still barfing on that...

The updated notes on how I did it are:

wget https://storage.googleapis.com/golang/go1.7.1.linux-amd64.tar.gz &&
tar -xzvf go1.7.1.linux-amd64.tar.gz &&
export PATH=$PATH:$PWD/go/bin &&
export GOROOT=$HOME/go &&
mkdir $GOROOT &&
sudo yum install -y squashfs-tools libacl-devel glibc-static trousers-devel &&
wget https://github.com/coreos/rkt/archive/v1.15.0.tar.gz &&
cd rkt-1.15.0 &&
./autogen.sh &&
./configure --disable-sdjournal --with-stage1-flavors=fly --disable-tpm &&
echo 'readlink -f "$@"' > realpath &&
chmod +x realpath &&
export PATH=$PATH:. &&
make"
@miku

This comment has been minimized.

miku commented Sep 6, 2017

This is an old issue, but I ran into the exact same problem in a completely different project.

The solution was simple: Build the project's binary distribution (or rpm in my case) on a CentOS VM with an older GLIBC (maybe keep a modified VM around, since vanilla CentOS won't work). If you are interesting in the more gory details, this is the part of the Makefile to make it happen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment