Turn `link-args` into `advanced-linking`, add upstream crate linking and static linking

[aidanhs]

This is not complete (the most serious omission being not explaining the 'official' way to statically link (musl)) but I want to check I'm on the right lines for an appropriate format for the rust book.

@steveklabnik @alexcrichton

[rust-highfive]

r? @alexcrichton

(rust_highfive has picked a reviewer for you, use r? to override)

[steveklabnik]

lgtm

[alexcrichton]

This seems like somewhat of an unusual section, could you elaborate a bit what's going on here? For example I would not expect us to recommend this, but rather contacting the author of the upstream crate to take care of these #[link] dependencies. A failure mode of this is that if the native library is included statically you'll likely receive link errors because the native lib is included before anything depends on it.

[aidanhs]

This is simply documenting the comment and test that you added in https://github.com/rust-lang/rust/pull/12575/files#diff-1f0a1ddb1f3c380e4fa9ac764564c306R1155

[aidanhs]

Would the specific examples you gave in the comment make it clearer?

[alexcrichton]

Haha, I see :). That use case is a little flavorful, however, and I still have yet to convince myself that it's 100% correct, so I'd personally prefer to not document it as an "official way to do things". Dealing with native libraries is always an unfortunate burden to place on your downstream consumers, and pooling as much logic as possible in one library is always best practice.

[aidanhs]

I agree, it's not something that should be done often (hence it being in advanced linking) but if 'normal' (i.e. not manually hacking around with linking) rust development could require it, surely it needs to be documented somewhere?

I could alternatively add it as a couple of sentences to the rust reference to elaborate on the example there - it was just a bit excessive to have to dig through commit logs to realise that #[link on an empty extern block is acceptable.

[alexcrichton]

I think that something like this may best appropriately live in a reference-style piece of documentation rather than tutorial-like docs like the book.

[alexcrichton]

It may be worth also mentioning that there is a -C link-args argument, as well as the default linker being gcc on most systems but link.exe is used for MSVC.

[aidanhs]

Will add, this was just a copy-and-paste from the previous file.

[alexcrichton]

Ah right, thanks though!

[alexcrichton]

This statement may not quite be true as Rust doesn't really compile against a particular glibc because we don't use their headers. All Rust code using the standard library is compatible with glibc 2.18 by default, so most Rust programs are compatible with older glibc versions automatically.

[aidanhs]

Yes, it's more targeted at external native libraries which may use exciting new features.
When I replace this section with musl I'll elaborate that it's possible (even likely) that external native libraries will need to be recompiled with musl to be linkable.

[aidanhs]

I looked over this bit again. Here's what I'm talking about:

$ cat /etc/issue | head -n 1
Ubuntu 14.04.2 LTS \n \l
$ /lib/x86_64-linux-gnu/libc.so.6 | head -n 1
GNU C Library (Ubuntu EGLIBC 2.19-0ubuntu6.6) stable release version 2.19, by Roland McGrath et al.
$ cat example.rs 
fn main() {}
$ rustc example.rs
$ nm example | grep GLIBC | grep thread_atexit
                 w __cxa_thread_atexit_impl@@GLIBC_2.18
$ docker run -it --rm -v $(pwd):/r centos:centos6 bash
[root@6dec72837400 /]# cat /etc/issue | head -n 1
CentOS release 6.6 (Final)
[root@6dec72837400 /]# /lib64/libc.so.6 | head -n 1
GNU C Library stable release version 2.12, by Roland McGrath et al.
[root@6dec72837400 /]# /r/example
/r/example: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /r/example)
/r/example: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /r/example)

[alexcrichton]

Ah yeah that's somewhat different, it's that you linked against a newer version of libc but when running it against an older version not all symbols were available. This is solved with static linking because there aren't any dependencies, but it doesn't mean that dynamic linking should be avoided (just that it should be compiled against an older libc)

[aidanhs]

Ah, if it's somewhat different to what you took away from the sentence then it clearly needs rephrasing :)
The thrust is that I'd consider static linking an easier solution than finding/compiling an older libc and then trying to compile against it. It also helps in more exotic situations, like non-glibc distros or even distros without a dynamic libc.

Reading again, I see the mistake is in my sloppy wording - it should be 'linked against' rather than 'compiled against'.

[alexcrichton]

glibc is actually pretty clever in its linkage in that you can link against a newer version but remain compatible with much older versions (so long as you avoid the "only new" apis). Overall I believe that static linking is largely only motivated by being able to use new APIs on all systems, or being able to run on all systems that may not even have the dependencies in question.

[alexcrichton]

This section seems somewhat confusing to me, as it's a pretty convoluted method of producing a static binary and I don't think we want official documentation indicating that this is the sanctioned method of doing so. Creating a static binary on Linux is currently best done through targeting MUSL (which we just need to produce nightlies for), and trying to statically link glibc will probably not end super well unfortunately (as you've probably found out)

[aidanhs]

Yes, I've changed my mind about including this. It doesn't work any more anyway since rustc changed how link arguments are constructed :P
I'll replace it all with musl.

[aidanhs]

Added extra notes on link-args as requested.
Replaced glibc static linking with musl, including instructions on how to build a musl-enabled rust.

@steveklabnik do you have any feelings on where the 'build-script' style readme for building musl should go? This is the only way of obtaining a musl-enabled rust at the moment and it's not a trivial process. Unfortunately, I'm not sure if it's the right style for the rust book.

Happy to put it in a github repo maintained by me if that's what you'd prefer.

[alexcrichton]

Yeah I feel like the book may not be the right place for things like build instructions, but our old wiki location also doesn't exist so I'm not quite sure where they'd land up otherwise.

[retep998]

No sections for advanced linking on Windows? 😞

[aidanhs]

Unfortunately I've not (yet) used rust on Windows so I've no idea what details might be useful.

[steveklabnik]

Sorry about this, I missesd the updates for some reason.

Unfortunately, I'm not sure if it's the right style for the rust book.

Yeah, I'm not sure. Given that this is all in the nightly section, I'm fine with it being a bit odd. Because when this finally lands in stable, it will Just Work, right?

[aidanhs]

As soon as nightly linux builds start including musl I'd think we can delete all the build instructions because it's something you'd probably only need to do if you're trying to do dev on rust itself (and hopefully the build scripts would be available in an open repo for people to refer to).

I'm happy to leave this PR open until that happens and then remove the build instructions, I just didn't know how long it'd be and wanted to write it down somewhere.

There's a few more things that'd need to be fleshed out for stable (e.g. "if you're linking against external C libraries, you may need to build them with musl") but yes, pure rust programs should Just Work.

I have no idea if any of that answers your question 😄

[alexcrichton]

This last sentence doesn't quite 100% agree with reality I think because the "Rust runtime" doesn't actually exist. I think this paragraph may want to be re-worded a bit to emphasize that Rust dependencies are statically linked by default but dependencies like libc/librt/libm are all dynamically linked, and this section is specifically about statically linking only the system dependencies.

[alexcrichton]

It may be a bit to get MUSL nightlies ready to go and smooth integration with distributions in general, so I'm fine landing this ahead of that in the nightly section of the book as well.

[bors]

☔️ The latest upstream changes (presumably #26696) made this pull request unmergeable. Please resolve the merge conflicts.

[alexcrichton]

Closing due to inactivity, but feel free to reopen with a rebase and my comments addressed!

[aidanhs]

@alexcrichton rebased and tweaked based on comments. I've done a minor rephrasing of your comment for the 'justification' of why dynamic linking may not be desirable and removed all mention of the rust runtime.

One note: I'm not 100% clear on the definition of 'system' libraries (e.g. libpython compiled in my home dir?), so I opted for 'native' instead.

[alexcrichton]

Hm looks like github hasn't picked up the changes, so perhaps a new PR?

[aidanhs]

Continued on #27193.