How to compile rust in a musl libc based linux system

[uuhan]

Hi, recently I need to compile rust in my musl libc based linux system. I download the rust's source code, configure it and make.

But it failed due to the rust/x86_64-unknown-linux-gnu/stage0/bin/rustc run failed.
It's interpreter is
/lib64/ld-linux-x86-64.so.2
but my system has just the
/lib/ld-musl-x86_64.so.1

I have already the llvm-3.5 installed, It seems to be it just lack of a runable rust stage0 binary for my system.

So, is there a STATICALLY LINKED RUST STAGE0 binary to feed my needs?
Or, how can I compile the proper stage0 rustc binary from stretch ?

[MagaTailor]

Is the musl ld compatible enough with gnu ld (and the two libc's themselves)? If so, symlinking should be sufficient.

[alexcrichton]

Yes unfortunately we only produce glibc snapshots that are dynamically linked. There are currently no statically linked snapshots of the compiler.

[nodakai]

Can you try /lib/ld-musl-x86_64.so.1 /path/to/stage0/rustc ? If successful, the rest is just shell scripting.

My understanding is it's similar to overriding a Shebang line in a shell script by explicitly calling a shell, like /bin/dash /path/to/script.sh

• http://man7.org/linux/man-pages/man8/ld.so.8.html

The .interp section in an ELF exe should serve as a "Shebang line."

[uuhan]

@petevine @nodakai Did not work. Some Symbol not found
__rawmemchr mallinfo __register_atfork

@alexcrichton Get it. : )

[nodakai]

How about referencing a pre-built glibc (eg. https://www.archlinux.org/packages/core/x86_64/glibc/ ) ? It's shipped with a dynamic loader as well.

$ strace -e file ./usr/lib/ld-linux-x86-64.so.2 --inhibit-cache --library-path ./usr/lib /bin/cat /etc/hostname 2>&1 | grep -v ENOENT
execve("./usr/lib/ld-linux-x86-64.so.2", ["./usr/lib/ld-linux-x86-64.so.2", "--inhibit-cache", "--library-path", "./usr/lib", "/bin/cat", "/etc/hostname"], [/* 52 vars */]) = 0
open("/bin/cat", O_RDONLY|O_CLOEXEC)    = 3
open("./usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
getcwd("/tmp/myglibc", 128)             = 13
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
open("/etc/hostname", O_RDONLY)         = 3
d9+++ exited with 0 +++

(Not sure if strace(8) is reliable under such a hack.)

[jirutka]

@alexcrichton in #34687:
I know I'd personally love to see rustc running on Alpine, I've tried it once or twice in docker and have been sad when it doesn't work!

Well, obviously, because Alpine doesn’t use glibc, but musl libc, that is only partially binary compatible with glibc (musl strictly follows standards, but glibc does not). Therefore you can’t run rustc snapshot compiled against glibc on Alpine and it’s not possible to build rustc without rustc.

@alexcrichton in #34687:
Do you know if there are cross compilers from something like Ubuntu to Alpine Linux?

This is not about distributions, but different libc. So you need something to cross-compile between glibc and musl. It’s very similar to cross-compiling between different architectures.
Yes, there are such a tools, for example musl-cross.

@alexcrichton in #34687:
If we have that then the rest of the implementation should be relatively easy I believe.

I believe that it may be easy for someone skilled in cross compiling, but unfortunately me and other guys from Alpine community, who would like Rust on Alpine, are not. And those few skilled C programmers doesn’t like Rust, because… you know… C programmers…

/cc @japaric

[japaric]

I got a bash script that uses docker to cross compile a rustc for the host x86_64-unknown-linux-musl from a Ubuntu image. The rustc is a fully statically linked binary and doesn't depend on glibc because it was built against musl. The issue is that the resulting rustc segfaults when I try to compile anything ... rustc -V works though. The root of the problem appears to be LLVM but I haven't had to look into it / fix it. I have made my work so far public in the hope that someone might be able to push it to the finish line. See this i.r-l.o thread for the details.

[jirutka]

@japaric Thanks for your effort. I’ve discussed your gist in our IRC channel and skarnet recommends to use richfelker/musl-cross-make instead of GregorR/musl-cross. He said that GregorR’s musl-cross is pretty much obsoleted by richfelker’s (dalias’) musl-cross-make that is more up-to-date.

[japaric]

@jirutka Thanks for tip. I've updated the gist to use musl-cross-make instead of musl-cross. Unfortunately, it didn't magically fix the llvm issue.

[jirutka]

@japaric I’m trying to compile rustc using your script and it always fail in the last phase with:

build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-musl/lib/libunwind-40c0f24bb84d756d.rlib(UnwindLevel1.c.o): unrecognized relocation (0x2a) in section `.text.assert_rtn'
toolchain/bin/../lib/gcc/x86_64-linux-musl/5.2.0/../../../../x86_64-linux-musl/bin/ld: final link failed: Bad value

I’ve tried it with gcc 5.3.0 and 5.2.0, with and without --enable-llvm-static-stdcpp, but with no difference.

Do you have some idea what’s wrong?

[japaric]

@jirutka

I’m trying to compile rustc using your script and it always fail in the last phase with:

While building libunwind-40c0f24bb84d756d.rlib? That looks like a new error to me.

Are you following the gist instructions "to the letter"? In particular, are you using a Ubuntu 15.10 docker image? Because, IIRC, there some was some bug in the Ubuntu 16.04 binutils package that also caused a linker error (don't remember the details). That's why the instructions say: use Ubuntu 15.10.

Or it could be a recent regression in the unwind crate (doesn't seem likely to me though). Perhaps try checking out an older version of rust-lang/rust? My last test of the gist was around 6 days ago

[jupp0r]

Wouldn't it make sense for the stage0 rustc to be statically linked in general?

[japaric]

@jupp0r The stage0 snapshots are getting deprecated though. The new plan is to bootstrap from a previous stable release (cf. #32942). Unless you mean that we should always bootstrap from a stable rustc with host=x86_64-unknown-linux-musl (which doesn't exist right now)?

[japaric]

(Heh, I'm not totally sure if I should this here or in one of the other closely related open issues but here it goes)

Update: I got a rustc that works an Alpine. I have thrown a tarball of said compiler in my Dropbox in case anyone is interested in trying it out. I tested the compiler with the following smoke test:

$ docker run --rm -it alpine /bin/sh
# apk update
# URL="https://www.dropbox.com/sh/t3vs8sjc3u8rzzr/AACjn6w1Dq75THcV62YGCQQ5a/2016-07-25.tar.gz?dl=1"
# apk add curl # for the command below
# curl -sL $URL | tar -xz
# apk add gcc # need libgcc_s for rustc to work
# rustc -V
# ldd $(which rustc)
        /lib/ld-musl-x86_64.so.1 (0x560e95330000)
        librustc_driver-3db96603e7de68a8.so => /bin/../lib/librustc_driver-3db96603e7de68a8.so (0x7fb4c1f44000)
        libstd-b357b9eff40a747f.so => /bin/../lib/libstd-b357b9eff40a747f.so (0x7fb4c19d9000)
        libc.so => /lib/ld-musl-x86_64.so.1 (0x560e95330000)
        (..)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7fb4ba14f000)
        (..)
# apk add musl-dev # need a bunch of libraries (rt, dl, c, m, etc) when linking a Rust binary
# echo 'fn main() { println!("Hello, world!") }' > hello.rs
# rustc hello.rs
# ./hello
Hello, world!
# echo 'fn main() { panic!("Hello, world!") }' > panic.rs
# rustc panic.rs
# RUST_BACKTRACE=1 ./panic # backtraces work
thread 'main' panicked at 'Hello, world!', panic.rs:1
stack backtrace:
   1:     0x558cae97656f - std::sys::backtrace::tracing::imp::write::h4ec66725a4c8d28d
   2:     0x558cae978c6b - std::panicking::default_hook::_{{closure}}::hcb8d8a999157470b
   3:     0x558cae978125 - std::panicking::default_hook::hc2e39a27ba5d2d45
   4:     0x558cae9786de - std::panicking::rust_panic_with_hook::h59941143cbee269d
   5:     0x558cae971c53 - std::panicking::begin_panic::hc74c3c2af73ca1a0
   6:     0x558cae971d99 - panic::main::h27a6e2b4553ca3bd
   7:     0x558cae9783a8 - std::panicking::try::call::h494c5dbd41ce3146
   8:     0x558cae97e936 - __rust_maybe_catch_panic
   9:     0x558cae977981 - std::rt::lang_start::hfb76de782d5db06c
  10:     0x558cae971dc9 - main
  11:     0x7fc9ee95772e - <unknown>

I cheated to make this work though. I hacked the x86_64-unknown-linux-musl target to make it dynamically link to libc (and libgcc_s) so it no longer produces statically linked binaries. Also when I tested a rustc compiled with jemalloc enabled it crashed with a SIGBUS. I had to build rustc with jemalloc disabled to get the working compiler I linked above.

The way forward: I think we should add a new musl target that's just like the existing x86_64-unknown-linux-musl but that produces dynamically linked binaries. Then we can start producing binary releases of rustc for that target (via cross compilation) and it a few cycles the Alpine folks should be able to bootstrap a Rust compiler themselves (they need a stable compiler to bootstrap a new).

Implementations details: The implementation seems straightforward, but the unresolved question is: what should we call this new target? @alexcrichton suggested calling the new target x86_64-alpine-linux-musl but as I stated in another comment, I don't think we should conflate the "vendor" (Alpine) with dynamic linking. I think the new target should be called x86_64-unknown-linux-musl:dynamic (the stuff after the : is open to bikeshedding, also this naming convention was originally proposed by @alexcrichton IIRC). The rationale is that everything that the original target triple conveys (ARCH, VENDOR, OS and ABI/ENV) remains the same; we are changing something (how we link) that can't be expressed by the "conventional" target triple so it makes sense (to me at least) to not modify the original target triple and just tack the extra information after original target triple.

[jirutka]

This is really great news! Excellent work, @japaric! 🎉

…add a new musl target that's just like the existing x86_64-unknown-linux-musl but that produces dynamically linked binaries.

Absolutely. Although I’m more a fan of static linking, I don’t understand Rust’s implication musl → static linking. musl can be statically linked without any problems (unlike ***** glibc…), but that doesn’t mean that it can’t be dynamically linked as well. Most of the Alpine Linux is dynamically linked.

…suggested calling the new target x86_64-alpine-linux-musl … I don't think we should conflate the "vendor" (Alpine) with dynamic linking.

Me neither, musl libc and Alpine are two different projects and Alpine is probably not the only distro that uses (dynamically linked) musl.

I think the new target should be called x86_64-unknown-linux-musl:dynamic…

I’m afraid that it’s already too late to change it, but to be consistent with others, x86_64-unknown-linux-musl should not implicate static linking. Thus it would make more sense to add :static suffix for static linking.

…the stuff after the : is open to bikeshedding…

I’m not sure about colon. It may be problematic e.g. on Windows when used in a file name. Triplet is already not a triplet, but quaternion, so maybe it would not hurt to use a quintuplet? ;)

That said, I’m probably okay with any name, I can’t wait for rustc officially supported for musl. 😸

I’m gonna try your tarball yesterday. Thanks a lot!

Could you please also update your gist with the build script?

/cc @LeoUnglaub

[japaric]

I’m afraid that it’s already too late to change it, but to be consistent with others, x86_64-unknown-linux-musl should not implicate static linking.

I agree, but it's too late. Changing it will probably break a bunch of build scripts.

Could you please also update your gist with the build script?

Sure, I'll ping you when it's updated.

[jupp0r]

I've been working on producing a static rustc the last few days, to no avail unfortunately.

[japaric]

Sure, I'll ping you when it's updated.

@jirutka The gist has been updated.

[jirutka]

@japaric I can confirm that rustc you’ve compiled works well on Alpine Linux. Could you please create a complete snapshot like those hosted on http://static.rust-lang.org? I’ve spent half day just trying to built cargo, using cargo-bootstrap, and it’s like a nightmare. :(

[jirutka]

Uff, I finally managed to build cargo on musl (with a lot of ad-hoc fixes during the process)!

cargo-0.11.0-nightly-x86_64-alpine-linux-musl.tar.gz (3.9 MiB)

[MagaTailor]

Not bad @jirutka! I wonder if panic=abort and lto works too? (I'm thinking about a smallest x86_64 fully optimized cargo binary to date)

[japaric]

Could you please create a complete snapshot like those hosted on http://static.rust-lang.org?

I think you only need to call make rustc-stage3 instead of make to get one of those. Do note that the bootstrap process is now using previous stable releases instead of the old snapshots. But, I guess you can use the snapshot with --local-rust=/path/to/rustc.

I’ve spent half day just trying to built cargo, using cargo-bootstrap, and it’s like a nightmare.

We are not building Cargo for x86_64-unknown-linux-musl yet? I thought that most of the necessary pieces were there already (cc @alexcrichton). Last time I tried, it was straightforward to cross compile Cargo to musl with these steps. Do note that you need to use this Docker image.

[jirutka]

I think you only need to call make rustc-stage3 instead of make to get one of those.

Hm, I’m still getting errors like below when compiling rustc on Alpine:

/usr/lib/gcc/x86_64-alpine-linux-musl/6.1.0/../../../../x86_64-alpine-linux-musl/bin/ld:
  /tmp/rustc.DH17TbBpNCoJ/liblibc-5aeeadd0596ef5a0.rlib(posix_spawn.lo):
  relocation R_X86_64_PC32 against protected symbol `execve' can not be used when making a shared object

I guess you can use the snapshot with --local-rust=/path/to/rustc.

Yes, that’s what I’m doing.

We are not building Cargo for x86_64-unknown-linux-musl yet?

No, https://static.rust-lang.org/cargo-dist/2016-03-21/cargo-nightly-x86_64-unknown-linux-gnu.tar.gz returns 404. BTW, it’s kinda hard to find what snapshots are available when static.rust-lang.org doesn’t provide indexes…

Last time I tried, it was straightforward to cross compile Cargo to musl with these steps.

Uh, I’m stupid… I found this issue, but wanted to try it with cargo-bootstrap and then forgot about this easier way.

[japaric]

Hm, I’m still getting errors like below when compiling rustc on Alpine:

Hmm, could that be related to #34978? If that's the case, I think a "snapshot" would still have the same problems. What version of binutils do you have installed in your Alpine box? It seems the binutils version is the root of the issue. A possible solution would be to cross compile the initial musl rustc from an older Ubuntu release (15.10 maybe?).

BTW, it’s kinda hard to find what snapshots are available when static.rust-lang.org doesn’t provide indexes…

I've been annoyed by that before. Could you open an issue in rust-lang/cargo?

[japaric]

I'd expect both rustc versions to be able to produce static as well as dynamic binaries.

I agree. Whether that feature should always be exposed as flag or as a second target on all platforms is still undecided. So far, x86 Linux and ARM Linux use the second target (the musl target) approach.

Right now we have this situation (x - supported) (? - unsupported)

Building a static binary with a dynamic rustc is also supported, via cross compilation to the musl target.

I think most people are either interested in the bottom row for reasons of producing a completely static binary, even when linking against native libraries or in the right column in order to bootstrap on any linux distro.

IMO, the use cases in order of importance/frequency are: dynamic binary (the default), static binary (mainly used to ease deployment) and static rustc (not too many people bootstrap the compiler).

[alexcrichton]

@japaric

In that case, it might be possible to add the second musl target today in a way that's forward compatible with whatever mechanism we chose for the static vs dynamic selection.

Perhaps, yes, but if we extend the concept of a target to also take into account extra configuration (like how we link libc) then we may not necessarily diverge literally in the name of the target. I definitely agree however that we're always going to be producing two artifacts! It's just a question of what to call them :(

I wouldn't feel too too bad about adding new target names, if we can especially figure out a good name pattern that would solve the MSVC case as well then that'd be great.

[jupp0r]

I agree. Whether that feature should always be exposed as flag or as a second target on all platforms is still undecided. So far, x86 Linux and ARM Linux use the second target (the musl target) approach.

Does this approach work if you link against native libraries? My specific use case is producing a static linux binary including libclang.

[emilymaier]

How are you dealing with the issue that some crates have to pick between building a static library vs a shared one depending on whether the same target produces a statically linked binary or a dynamically linked one (e.g. can't build a shared library if producing a statically linked binary)? rust-lang/rfcs#1684 (which is about windows, not musl) proposes that Cargo sets an env variable to signal the difference.

I didn't get that far. The more immediate problem I had was building liblibc, as I wanted the .rlib to include libc.a and the .so to link to libc.so, but the link args only permitted one or the other for building a single target as far as I could tell.

[japaric]

@jupp0r

Sure, as long as the crate with the Rust bindings to libclang (and actually, all your other dependencies that bind to a C library) tells Cargo that it will statically link to libclang.a. But I bet that the crate (and probably the others) is configured to link dynamically to libclang.so.

BTW, which clang crate are you using?

@emilymaier

the .rlib to include libc.a and the .so to link to libc.so

My understanding is that a libfoo.rlib will either contain a copy of libbar.a or simply contain a "must link to libbar.so" instruction in its metadata (it doesn't contain a copy of libbar.so!). It doesn't seem possible to tell the rlib to do both things because when building a dynamically linked binary, rustc will have to pass libfoo.rlib, which contains libbar.a, and -lbar to the linker and the linker will likely fail because it will end up with two sets of the same symbols: one set of defined symbols (libbar.a) and set of undefined symbols (libbar.so).

[DemiMarie]

@japaric Should there be a way for a crate to link dynamically to an external library if and only if it itself is to be linked dynamically?

[jupp0r]

@japaric @DemiMarie:

https://github.com/KyleMayes/clang-sys does an excellent job of enabling dynamic or static linking of external libraries via cargo feature flags.

[japaric]

@DemiMarie That can be done in the build.rs: one can pick between cargo:rustc-link-lib=foo or cargo:rustc-link-lib=static=foo but it would be up to the crate author to enforce the restriction you mention.

@jupp0r Oh, that's nice. That crate with the static feature enabled plus the current x86_64-musl target would let you build statically linked binaries.

[jirutka]

@japaric One month passed, so I’d like to ask, is there some progress on this issue? When will Rust support dynamic linking on Musl libc and when will Rust start providing precompiled rustc for musl?

[japaric]

@jirutka Sorry, not much progress since then. We haven't decided how we want to expose the static vs dynamic linking choice to the user though there is a RFC (rust-lang/rfcs#1721) on that topic. Without that settled, it's not clear if we want to add a second musl target (same definition, different name) that always links dynamically or if we can keep using a single musl target (and have a single musl rustc release) and simply have the user specify whether they want to use static (the default, which is unlikely to change at this point) vs dynamic linking at build time.

[jirutka]

We haven't decided how we want to expose the static vs dynamic linking choice to the user though there is a RFC (rust-lang/rfcs#1721) on that topic.

This RFC is definitely the better way, but it’s one month since the last comment on the RFC and 3/4 of the year since this issue has been opened. Until the decision is made for rust-lang/rfcs#1721, could you please add new (temporary) targets -alpine-linux-musl for dynamically linked musl, so we can move forward? As you wrote in #31322 (comment):

…it might be possible to add the second musl target today in a way that's forward compatible with whatever mechanism we chose for the static vs dynamic selection.

It’ll be probably used only in Alpine and Void. I have no problem with changing the Alpine packages to use the new mechanism after rust-lang/rfcs#1721 is implemented. I guess that @chneukirchen can take care of Void packages.

[jirutka]

BTW, is it already possible to build statically linked rustc?

[japaric]

@jirutka

I asked this to the tools team. And they are leaning to:

• Accept and implement that RFC, then
• we won't add a new musl target
• we'll have binary releases of rustc. (Speculation: this rustc release will probably be a
  dynamically linked binary as that's known to work)
• you'll be able to bootstrap a dynamically linked rustc from that snapshot using some form of
  configuration (probably a config.toml).
• you'll be able to configure whether your rustc will build statically linked binaries by default, or if they'll
  be dynamically linked by default.

I didn't explicitly ask about the temporary solution of adding a new target but, IMO, seems unlikely that we'll do that because we'll have to maintain that target "forever" even if it's stop being useful/used.

BTW, is it already possible to build statically linked rustc?

I haven't tried since more than one moth ago but last time I tried it still segfaulted. Our LLVM fork got upgraded to 3.9 in the meantime so there may have been some change on that front.

[japaric]

you'll be able to configure whether your rustc will build statically linked binaries by default, or if they'll be dynamically linked by default.

To clarify, this will probably be configurable via a .cargo/config e.g. in $HOME at "run time" rather than configurable at build time e.g. via configure.

[lluixhi]

@jirutka
Just FYI, gentoo is interested too.

[japaric]

Update: PR #37545 implements the crt-static target feature that can be used to pick between statically linking a binary to musl or linking it dynamically to musl. The associated RFC has already been approved.

[misery]

PR is merged.... is it possible to build rust 1.15.0 on AlpineLinux now? :-)

[jirutka]

@misery I’ve tried to build 1.11.0 with our current 1.10.0 some time ago and it failed. So I’m afraid that I have to cross-compile it again. Unfortunately there’s still no precompiled rustc for bootstrapping rustc on must-based distro (or statically linked rustc so it can be used anywhere) and I haven’t had time yet for another cross-compiling and hacking session. 😿

[pickfire]

Yes, I have tried to build it with https://s3.amazonaws.com/rust-lang-ci/cargo-builds/fbeea902d2c9a5be6d99cc35681565d8f7832592/cargo-nightly-x86_64-unknown-linux-gnu.tar.gz as well but no luck.

Traceback (most recent call last):
  File "/home/ivan/act/pkg/aports/testing/rust/src/rustc-1.15.1-src/src/bootstrap/bootstrap.py", line 491, in <module>
    main()
  File "/home/ivan/act/pkg/aports/testing/rust/src/rustc-1.15.1-src/src/bootstrap/bootstrap.py", line 474, in main
    rb.build_bootstrap()
  File "/home/ivan/act/pkg/aports/testing/rust/src/rustc-1.15.1-src/src/bootstrap/bootstrap.py", line 297, in build_bootstrap
    self.run(args, env)
  File "/home/ivan/act/pkg/aports/testing/rust/src/rustc-1.15.1-src/src/bootstrap/bootstrap.py", line 300, in run
    proc = subprocess.Popen(args, env=env)
  File "/usr/lib/python2.7/subprocess.py", line 390, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1024, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
make: *** [Makefile:23: all] Error 1

Might as well be python 2's fault, does it supports python 3?

[ncopa]

We are adding support for ppc64le and s390x to Alpine (with musl libc). Are there any progress in a sane way to bootstrap/port rustc?

[japaric]

@ncopa A few people are working on it right now. See #40113

[corbinu]

Now that #40113 has closed does anybody know how I actually build master for Alpine. I have tried both on Alpine and Ubuntu and keep hitting walls. If anybody would be willing to help me through it I would be happy to document it for others.

[rgdoliveira]

I tried to bootstrap rust 1.20.0 on Alpine ppc64le (./x.py build) but it fails when trying to run 'cargo' ELF (that is downloaded at build time) because it was compiled using gblic.

[stefson]

It is complicated. The guy who wrote #40113 explained (tried to explain) to me how it works, see here: gentoo/musl#44 (comment)

You need for sure a fully working rust and cargo plus llvm and gcc (for linking) installed on the system you will start the build, so before that isn't done you don't have to worry about further details :)