Allow running a statically-compiled Linux executable

[ERnsTL]

OSv currently can only run position-independent shared objects.

This means we usually can't run a random Linux application without recompiling it (with -fPIC -shared), unless we're lucky and the application is already mostly a .so (like is the case in the JVM).

It is, as detailed below, desireable to allow running a single statically compiled executable, as created for example by the Go programming language (Golang).

The Go programming language was created with the needs of server farms at Google, cloud computing, multicore processors and concurrency in mind and excels for use to write network servers and is positioned as such. Consequentially, its popularity has grown along with the trend in virtualization and cloud computing.

Running virtualized network servers in the cloud efficiently is also a primary use-case of OSv.

Because of this use-case overlap and bright future for the Go language in cloud computing, I created this ticket in the hope that the feature of running its statically compiled executables will be given an amount of priority and be available in one of the next releases.

This issue is possibly related to:

• Issue #190

[penberg]

👍 to this and to #190. No we just need to find someone to implement it!

[glommer]

That's actually one of the ideas for our GSoC application.

On Thu, Feb 13, 2014 at 5:48 PM, Pekka Enberg notifications@github.comwrote:

[image: 👍] to this and to #190#190.
No we just need to find someone to implement it!

Reply to this email directly or view it on GitHubhttps://github.com//issues/212#issuecomment-34978995
.

[nyh]

Hi, can you please explain how this issue is different from #190? I see you
even copied a verbatim paragraph from it?

Is the difference the word static? Do you actually mean a static executable
where the libc calls have been replaced by code from glibc? I am afraid
doing this on osv is a big diversion from the way we are heading. Why is
it even necessary? What's wrong with dynamic executables, linked to
libc.so and not libc.a?

[nyh]

Now that I'm back to a computer with a decent keyboard, I can expand on my previous answer.

First, can you please explain to me why the Go people even want static linking? Static linking has so many problems: It generates huge executables, it waste tons of memory when different processes running together cannot share text pages ("shared libraries" got their name for a reason). It creates security problems when it's not enough to update a library, you also need to update all the programs that used it. Almost 10 years ago, Solaris 10 came out with an announcement: Static libc is no longer provided. You can no longer create static executables (if they use libc). Hardly anyone cried. Why would Go want to bring it back? Is it possible to compile go programs so that its runtime does not run Linux system calls directly, but rather use glibc (and links to it dynamically)? If I understand correctly, this is possible (with gccgo), so what's the benefit of not doing this?

Second, I want to explain what it would take to get OSv to run executables which are statically linked with libc.a - or executables which directly use Linux system calls and don't use libc at all.
OSv is designed as a library OS - when a program runs the malloc() function, we run OSv's code which does malloc(), which we wrote from scratch to fit the OSv design. If we statically compile with libc.a, we get some glibc's malloc() code which we can't control, which ultimatively calls the sbrk() system call. OSv does not currently support sbrk() (or even this concept), nor does it even support system calls in the Linux sense. We'll need to change all of this to run static executables. Looks really complex, and I wonder if actually worth it.

[penberg]

@nyh I assume it's to avoid Go version dependency issues but I couldn't really find a reference for it. Not that it really matters much why Go decided to go for static linking. If we want to run Go programs on OSv, that's what we have to support.

[glommer]

On Thu, Feb 13, 2014 at 11:11 PM, nyh notifications@github.com wrote:

Now that I'm back to a computer with a decent keyboard, I can expand on my
previous answer.

First, can you please explain to me why the Go people even want static
linking? Static linking has so many problems: It generates huge
executables, it waste tons of memory when different processes running
together cannot share text pages ("shared libraries" got their name for a
reason). It creates security problems when it's not enough to update a
library, you also need to update all the programs that used it. Almost 10
years ago, Solaris 10 came out with an announcement: Static libc is no
longer provided. You can no longer create static executables (if they use
libc). Hardly anyone cried. Why would Go want to bring it back? Is it
possible to compile go programs so that its runtime does not run Linux
system calls directly, but rather use glibc (and links to it dynamically)?
If I understand correctly, this is possible (with gccgo), so what's the
benefit of not doing this?

Nadav. wrt sharing pages: what are you talking about? For us that's
actually perfect. There is no one else in the system for you to share
with, so the distinction here is moot. That's a perfect combination
between go and OSv, in that aspect.

Second, I want to explain what it would take to get OSv to run executables
which are statically linked with libc.a - or executables which directly use
Linux system calls and don't use libc at all.
OSv is designed as a library OS - when a program runs the malloc()
function, we run OSv's code which does malloc(), which we wrote from
scratch to fit the OSv design. If we statically compile with libc.a, we get
some glibc's malloc() code which we can't control, which ultimatively calls
the sbrk() system call. OSv does not currently support sbrk() (or even this
concept), nor does it even support system calls in the Linux sense. We'll
need to change all of this to run static executables. Looks really complex,
and I wonder if actually worth it.

Well, let them. So what ?
sbrk() is really bizarre in the sense that so few programs use it. But we
have made the commitment to support the Linux interface to the best of our
capabilities. What's different about sbrk besides it being rare?

[gleb-cloudius]

On Thu, Feb 13, 2014 at 12:41:25PM -0800, Glauber Costa wrote:

Second, I want to explain what it would take to get OSv to run executables
which are statically linked with libc.a - or executables which directly use
Linux system calls and don't use libc at all.
OSv is designed as a library OS - when a program runs the malloc()
function, we run OSv's code which does malloc(), which we wrote from
scratch to fit the OSv design. If we statically compile with libc.a, we get
some glibc's malloc() code which we can't control, which ultimatively calls
the sbrk() system call. OSv does not currently support sbrk() (or even this
concept), nor does it even support system calls in the Linux sense. We'll
need to change all of this to run static executables. Looks really complex,
and I wonder if actually worth it.

Well, let them. So what ?
sbrk() is really bizarre in the sense that so few programs use it. But we
have made the commitment to support the Linux interface to the best of our
capabilities. What's different about sbrk besides it being rare?

It is not rare, it is what malloc uses for small allocations. It is
probably possible to force glibc to always use mmap by setting MALLOC_MMAP_THRESHOLD_
to a very small value, but does go uses glibc memory allocator at all?

        Gleb.

[glommer]

On Fri, Feb 14, 2014 at 10:06 PM, Gleb Natapov notifications@github.comwrote:

On Thu, Feb 13, 2014 at 12:41:25PM -0800, Glauber Costa wrote:

Second, I want to explain what it would take to get OSv to run
executables
which are statically linked with libc.a - or executables which
directly use
Linux system calls and don't use libc at all.
OSv is designed as a library OS - when a program runs the malloc()
function, we run OSv's code which does malloc(), which we wrote from
scratch to fit the OSv design. If we statically compile with libc.a,
we get
some glibc's malloc() code which we can't control, which ultimatively
calls
the sbrk() system call. OSv does not currently support sbrk() (or even
this
concept), nor does it even support system calls in the Linux sense.
We'll
need to change all of this to run static executables. Looks really
complex,
and I wonder if actually worth it.

Well, let them. So what ?
sbrk() is really bizarre in the sense that so few programs use it. But we
have made the commitment to support the Linux interface to the best of
our
capabilities. What's different about sbrk besides it being rare?

It is not rare, it is what malloc uses for small allocations. It is

That's my definition of rare.
1 application uses it, for 1 use case.

[gleb-cloudius]

On Fri, Feb 14, 2014 at 10:08:19AM -0800, Glauber Costa wrote:

On Fri, Feb 14, 2014 at 10:06 PM, Gleb Natapov notifications@github.comwrote:

On Thu, Feb 13, 2014 at 12:41:25PM -0800, Glauber Costa wrote:

Second, I want to explain what it would take to get OSv to run
executables
which are statically linked with libc.a - or executables which
directly use
Linux system calls and don't use libc at all.
OSv is designed as a library OS - when a program runs the malloc()
function, we run OSv's code which does malloc(), which we wrote from
scratch to fit the OSv design. If we statically compile with libc.a,
we get
some glibc's malloc() code which we can't control, which ultimatively
calls
the sbrk() system call. OSv does not currently support sbrk() (or even
this
concept), nor does it even support system calls in the Linux sense.
We'll
need to change all of this to run static executables. Looks really
complex,
and I wonder if actually worth it.

Well, let them. So what ?
sbrk() is really bizarre in the sense that so few programs use it. But we
have made the commitment to support the Linux interface to the best of
our
capabilities. What's different about sbrk besides it being rare?

It is not rare, it is what malloc uses for small allocations. It is

That's my definition of rare.
1 application uses it, for 1 use case.

most (as in 99.9%) applications use it, for 1 use case. If this still fits your
definition of rare I do not know.

        Gleb.

[glommer]

On Fri, Feb 14, 2014 at 10:13 PM, Gleb Natapov notifications@github.comwrote:

On Fri, Feb 14, 2014 at 10:08:19AM -0800, Glauber Costa wrote:

On Fri, Feb 14, 2014 at 10:06 PM, Gleb Natapov <notifications@github.com
wrote:

On Thu, Feb 13, 2014 at 12:41:25PM -0800, Glauber Costa wrote:

Second, I want to explain what it would take to get OSv to run
executables
which are statically linked with libc.a - or executables which
directly use
Linux system calls and don't use libc at all.
OSv is designed as a library OS - when a program runs the malloc()
function, we run OSv's code which does malloc(), which we wrote
from
scratch to fit the OSv design. If we statically compile with
libc.a,
we get
some glibc's malloc() code which we can't control, which
ultimatively
calls
the sbrk() system call. OSv does not currently support sbrk() (or
even
this
concept), nor does it even support system calls in the Linux sense.
We'll
need to change all of this to run static executables. Looks really
complex,
and I wonder if actually worth it.

Well, let them. So what ?
sbrk() is really bizarre in the sense that so few programs use it.
But we
have made the commitment to support the Linux interface to the best
of
our
capabilities. What's different about sbrk besides it being rare?

It is not rare, it is what malloc uses for small allocations. It is

That's my definition of rare.
1 application uses it, for 1 use case.

most (as in 99.9%) applications use it, for 1 use case. If this still
fits your
definition of rare I do not know.

Yes, but they all do it through libc. Applications don't call it
themselves. So that's actually
1 entry point/code base - libc - that other programs call. Because we
provide our own libc,
exactly 0 of or applications end up using it.

This is what I mean by rare. The rest is a semantic discussion I have no
interest in carrying on.

Gleb.

Reply to this email directly or view it on GitHubhttps://github.com//issues/212#issuecomment-35109330
.

[gleb-cloudius]

On Fri, Feb 14, 2014 at 10:16:56AM -0800, Glauber Costa wrote:

On Fri, Feb 14, 2014 at 10:13 PM, Gleb Natapov notifications@github.comwrote:

On Fri, Feb 14, 2014 at 10:08:19AM -0800, Glauber Costa wrote:

On Fri, Feb 14, 2014 at 10:06 PM, Gleb Natapov <notifications@github.com
wrote:

On Thu, Feb 13, 2014 at 12:41:25PM -0800, Glauber Costa wrote:

Second, I want to explain what it would take to get OSv to run
executables
which are statically linked with libc.a - or executables which
directly use
Linux system calls and don't use libc at all.
OSv is designed as a library OS - when a program runs the malloc()
function, we run OSv's code which does malloc(), which we wrote
from
scratch to fit the OSv design. If we statically compile with
libc.a,
we get
some glibc's malloc() code which we can't control, which
ultimatively
calls
the sbrk() system call. OSv does not currently support sbrk() (or
even
this
concept), nor does it even support system calls in the Linux sense.
We'll
need to change all of this to run static executables. Looks really
complex,
and I wonder if actually worth it.

Well, let them. So what ?
sbrk() is really bizarre in the sense that so few programs use it.
But we
have made the commitment to support the Linux interface to the best
of
our
capabilities. What's different about sbrk besides it being rare?

It is not rare, it is what malloc uses for small allocations. It is

That's my definition of rare.
1 application uses it, for 1 use case.

most (as in 99.9%) applications use it, for 1 use case. If this still
fits your
definition of rare I do not know.

Yes, but they all do it through libc. Applications don't call it
themselves. So that's actually
1 entry point/code base - libc - that other programs call. Because we
provide our own libc,
exactly 0 of or applications end up using it.

That's exactly what Nadav was explaining is not the case for statically
linked executables, brk() was just an example and in reality should be
easy to implement. More complicated example may be clone() system call
which is used to implement pthread_create().

        Gleb.

[glommer]

On Fri, Feb 14, 2014 at 10:43 PM, Gleb Natapov notifications@github.comwrote:

On Fri, Feb 14, 2014 at 10:16:56AM -0800, Glauber Costa wrote:

On Fri, Feb 14, 2014 at 10:13 PM, Gleb Natapov <notifications@github.com
wrote:

On Fri, Feb 14, 2014 at 10:08:19AM -0800, Glauber Costa wrote:

On Fri, Feb 14, 2014 at 10:06 PM, Gleb Natapov <
notifications@github.com
wrote:

On Thu, Feb 13, 2014 at 12:41:25PM -0800, Glauber Costa wrote:

Second, I want to explain what it would take to get OSv to run
executables
which are statically linked with libc.a - or executables which
directly use
Linux system calls and don't use libc at all.
OSv is designed as a library OS - when a program runs the
malloc()
function, we run OSv's code which does malloc(), which we wrote
from
scratch to fit the OSv design. If we statically compile with
libc.a,
we get
some glibc's malloc() code which we can't control, which
ultimatively
calls
the sbrk() system call. OSv does not currently support sbrk()
(or
even
this
concept), nor does it even support system calls in the Linux
sense.
We'll
need to change all of this to run static executables. Looks
really
complex,
and I wonder if actually worth it.

Well, let them. So what ?
sbrk() is really bizarre in the sense that so few programs use
it.
But we
have made the commitment to support the Linux interface to the
best
of
our
capabilities. What's different about sbrk besides it being rare?

It is not rare, it is what malloc uses for small allocations. It is

That's my definition of rare.
1 application uses it, for 1 use case.

most (as in 99.9%) applications use it, for 1 use case. If this still
fits your
definition of rare I do not know.

Yes, but they all do it through libc. Applications don't call it
themselves. So that's actually
1 entry point/code base - libc - that other programs call. Because we
provide our own libc,
exactly 0 of or applications end up using it.

That's exactly what Nadav was explaining is not the case for statically
linked executables, brk() was just an example and in reality should be
easy to implement. More complicated example may be clone() system call
which is used to implement pthread_create().

ok, that's a bitch, indeed.

Gleb.

Reply to this email directly or view it on GitHubhttps://github.com//issues/212#issuecomment-35112070
.

[gebi]

JFTR... ( @nyh ) golang executables, as produced by the native go toolchain, are not only statically compiled but do NOT use glibc at all. Everything is implemented without any libc call, yes syscalls too. If you use an up to date go toolchain you can also compile a go executable without using the libc resolver at all. Instead an internal one will be used, producing static executables even when network libraries are used.

IMHO the most elegant way to integrate golang with OSv would be to add another architecture to the runtime which than can use native OSv methods to access the environment (network, ...).
One architecture which may help could be the nacl architecture which got merged recently (compiling go executables to run inside the chrome native client sandbox).

[elazarl]

@nyh Reasoning behind Go's decision to use static executable is, having a very stable execution among different servers. If the kernel is identical (and mostly it is), there could be little differences between running it in one server or another. One of the JVM guys who now works on Go mentioned in a talk his frustration of finding subtle bugs when moving a JVM program between different JVMs among different servers.

You can see Rob Pike's objection to dynamic linking (back in plan9, and obviously now as one of Go's authors) here

[nyh]

On Thu, Jul 3, 2014 at 10:04 AM, Elazar Leibovich notifications@github.com
wrote:

You can see Rob Pike's objection to dynamic linking (back in plan9, and
obviously now as one of Go's authors) here
http://harmful.cat-v.org/software/dynamic-linking/

With all due respect to Rob Pike (and I have a lot of it!), I remember
using his Unix which didn't support shared libraries, and later moving to
Unix variants that did (SunOS, System Vr4, and finally Linux), and I really
welcomed this addition. Especially when it was first introduced (when every
megabyte counted), but I do think they are still valuable today - e.g., for
allowing you to fix a library without recompiling all the software on your
system.
But I guess this isn't the question now... If Go uses static executables,
and people want to run them on OSv, we need to do this :(

Nadav Har'El
nyh@cloudius-systems.com

[penberg]

@nyh OTOH, I would pay good money to see you have that discussion with Mr. Pike. 😄

[glommer]

But I guess this isn't the question now... If Go uses static executables,

and people want to run them on OSv, we need to do this :(

Precisely. Although the discussion around it is interesting, the importance
for us at this moment
is really 0. Go decided to go this way, that's all that matters.

[gebi]

JFTR... It's not a big deal if osv does not support running go executables compiled for the linux platform. IMHO there is no need to wast time on this!

Go executables are distributed in source form and installed with a simple go get (everything that requires more than a go get will also require special attention to the environment anyway) and the go toolchain is able to cross-compile on every platform to every platform.
Thus osv can just as simple support go executables through the idea mentioned above (dedicated architecture/os) and require a separate compiled go executable instead of emulating the linux API to make a linux go exe run.

[elazarl]

@gebi I think it won't be easy to patch Go into producing PIC'd ELF, since they use their own C compiler and linker. As of using gccgo, I paste my thought from the mailing list:

The problem I see with relying on gccgo, is that it's not as well supported and as well tested as the plan9-style go.

For example:

1. It tends to lag behind the mainline Go, hence new libraries might not compile with it. https://groups.google.com/forum/#!topic/golang-nuts/4-mJfTTqULk
2. It's performance characteristics are different. For example, unless something changed, it used thread per goroutine. For some applications this is a deal breaker. And I'm pretty sure nobody stress-tested it.

So the end result of relying on gccgo, is that some use cases (especially a server with many clients, each uses a goroutine) will not be as well supported as native Go programs.

But at any rate, I think supporting standalone ELF is a worthy Go, even if I'm mistaken about gccgo's usefulness.

The main benefit of the ability to run native Linux ELF files, is that it allows someone not familiar with OSv to play with it, and try his favorite programs on OSv with minimal friction.

[pyrossh]

There is also a plan for golang 1.5 to enable packaging the go library as a shared lib. I think they may also allow you to compile a go app to .so, So that it doesn't need to be an executable won't this allow golang to work out the box in osv?

[nyh]

Modern golang already supports compiling the executable as a shared library or PIE, so Go is no longer a reason to support statically-linked executables.

[wkozaczuk]

@nyh, should it be closed because of the reasons stated in last comment?

[nyh]

Well, the original request - wanting to run statically-linked executables on OSv - still stands. It's just that now that it's not needed for Go, it is even lower priority than it was. And 90% of the arguments above, about why Go uses static executables, become irrelevant. Maybe we should open a new, clean, issue and close this one.

[wkozaczuk]

Closing this one in lieu of more focused new issues:

• Implement subset of arch_prctl syscall to support local-exec TLS of statically linked executables #1137
• Implement brk syscall to support statically linked executables #1138
• Implement subset of the clone syscall to support statically linked executables #1139
• Support loading and launching statically linked executables #1140