rustc: Group linked libraries where needed #49316

alexcrichton · 2018-03-23T21:40:42Z

This commit fixes a longstanding issue with the compiler with circular
dependencies between libcore and libstd. The core crate requires at least one
symbol, the ability to unwind. The std crate is the crate which actually
defines this symbol, but the std crate also depends on the core crate.

This circular dependency is in general disallowed in Rust as crates cannot have
cycles amongst them. A special exception is made just for core/std, but this is
also unfortunately incompatible with how GNU linkers work. GNU linkers will
process undefined symbols in a left-to-right fashion, only actually linking an
rlib like libstd if there are any symbols used from it. This strategy is
incompatible with circular dependencies because if we otherwise don't use
symbols from libstd we don't discover that we needed it until we're later
processing libcore's symbols!

To fix this GNU linkers support the --start-group and --end-group options
which indicate "libraries between these markers may have circular dependencies
amongst them. The linker invocation has been updated to automatically pass these
arguments when we're invoking a GNU linker and automatically calculate where the
arguments need to go (around libstd and libcore)

Closes #18807
Closes #47074

rust-highfive · 2018-03-23T21:40:53Z

r? @michaelwoerister

(rust_highfive has picked a reviewer for you, use r? to override)

bors · 2018-03-23T21:47:16Z

☔ The latest upstream changes (presumably #49308) made this pull request unmergeable. Please resolve the merge conflicts.

michaelwoerister

Thanks, @alexcrichton! r=me with the nits addressed.

michaelwoerister · 2018-03-27T10:04:06Z

src/librustc_trans/back/link.rs

+        //         Linkage::NotLinked |
+        //         Linkage::IncludedFromDylib => continue,
+        //         _ => {}
+        //     }


Did you leave this in on purpose?

er oops, no!

michaelwoerister · 2018-03-27T10:05:42Z

src/librustc_trans/back/link.rs

+    if group_end.is_some() && group_start.is_none() {
+        group_end = None;
+    }
+


Maybe add an assert!(group_start.is_some() == group_end.is_some())?

Ah unfortunately that assertion tripped when I added it, so I ended up having to implement it this way for libstd specifically

alexcrichton · 2018-03-27T18:29:22Z

@bors: r=michaelwoerister

bors · 2018-03-27T18:29:22Z

📌 Commit 8ad2f7f has been approved by michaelwoerister

bors · 2018-03-28T15:34:38Z

☔ The latest upstream changes (presumably #49019) made this pull request unmergeable. Please resolve the merge conflicts.

alexcrichton · 2018-03-28T23:04:23Z

@bors: r=michaelwoerister

bors · 2018-03-28T23:04:24Z

📌 Commit 50ad5f9 has been approved by michaelwoerister

bors · 2018-03-29T10:50:15Z

⌛ Testing commit 50ad5f9fcb3f69bbba7c82a39902f479b649503a with merge 0579f55645b45d0363079f88f880a64e67635259...

bors · 2018-03-29T11:34:13Z

💔 Test failed - status-travis

This commit fixes a longstanding issue with the compiler with circular dependencies between libcore and libstd. The `core` crate requires at least one symbol, the ability to unwind. The `std` crate is the crate which actually defines this symbol, but the `std` crate also depends on the `core` crate. This circular dependency is in general disallowed in Rust as crates cannot have cycles amongst them. A special exception is made just for core/std, but this is also unfortunately incompatible with how GNU linkers work. GNU linkers will process undefined symbols in a left-to-right fashion, only actually linking an rlib like libstd if there are any symbols used from it. This strategy is incompatible with circular dependencies because if we otherwise don't use symbols from libstd we don't discover that we needed it until we're later processing libcore's symbols! To fix this GNU linkers support the `--start-group` and `--end-group` options which indicate "libraries between these markers may have circular dependencies amongst them. The linker invocation has been updated to automatically pass these arguments when we're invoking a GNU linker and automatically calculate where the arguments need to go (around libstd and libcore) Closes rust-lang#18807 Closes rust-lang#47074

alexcrichton · 2018-03-29T22:08:06Z

@bors: r=michaelwoerister

bors · 2018-03-29T22:08:06Z

📌 Commit 88114f6 has been approved by michaelwoerister

bors · 2018-03-29T23:19:59Z

⌛ Testing commit 88114f6 with merge 491f4bf...

…oerister rustc: Group linked libraries where needed This commit fixes a longstanding issue with the compiler with circular dependencies between libcore and libstd. The `core` crate requires at least one symbol, the ability to unwind. The `std` crate is the crate which actually defines this symbol, but the `std` crate also depends on the `core` crate. This circular dependency is in general disallowed in Rust as crates cannot have cycles amongst them. A special exception is made just for core/std, but this is also unfortunately incompatible with how GNU linkers work. GNU linkers will process undefined symbols in a left-to-right fashion, only actually linking an rlib like libstd if there are any symbols used from it. This strategy is incompatible with circular dependencies because if we otherwise don't use symbols from libstd we don't discover that we needed it until we're later processing libcore's symbols! To fix this GNU linkers support the `--start-group` and `--end-group` options which indicate "libraries between these markers may have circular dependencies amongst them. The linker invocation has been updated to automatically pass these arguments when we're invoking a GNU linker and automatically calculate where the arguments need to go (around libstd and libcore) Closes #18807 Closes #47074

bors · 2018-03-30T01:45:49Z

☀️ Test successful - status-appveyor, status-travis
Approved by: michaelwoerister
Pushing 491f4bf to master...

japaric · 2018-04-03T13:32:49Z

@alexcrichton it seems that panic_fmt is not in the "missing lang items" list so this logic doesn't add {start,end}-group when only that lang item is in play.

alexcrichton · 2018-04-03T17:42:03Z

@japaric hm it's listed here but I may be misremembering what that's used for. Do you have an example I can poke around?

japaric · 2018-04-04T12:42:39Z

@alexcrichton #48661 (comment) still produces a shared library that contains an undefined reference to rust_begin_unwind. The linker invocation looks like this:

"cc"
"-Wl,--as-needed"
"-Wl,-z,noexecstack"
"-m64"
"-L"
"$SYSROOT/lib/rustlib/x86_64-unknown-linux-gnu/lib"
"$PWD/target/release/deps/bar.bar0.rcgu.o"
"-o"
"$PWD/target/release/deps/libbar.so"
"-Wl,--version-script=/tmp/rustc.NVjXDd5FIjVZ/list"
"-Wl,--gc-sections"
"-Wl,-z,relro,-z,now"
"-Wl,-O1"
"-nodefaultlibs"
"-L"
"/home/japaric/tmp/bar/target/release/deps"
"-L"
"$SYSROOT/lib/rustlib/x86_64-unknown-linux-gnu/lib"
"-Wl,-Bstatic"
# -Wl,--start-group
"$PWD/target/release/deps/libpanic-3f5354702551e825.rlib"
"$SYSROOT/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-4d12ce26aecbbf20.rlib"
# -Wl,--end-group
"-shared"
"-Wl,-Bdynamic"

I would expect the logic added in this PR to add the {start,end}-group flags in the places where I added comments, but today's nightly doesn't add those flags to the linker invocation. If you re-invoke the linker with the {start,end}-group flags you get a shared library with a defined rust_begin_unwind symbol.

alexcrichton · 2018-04-04T23:13:00Z

Ok thanks for the info! I think I've narrowed this down and am testing a fix

It turns out that the support in rust-lang#49316 wasn't enough to handle all cases notably the example in rust-lang#48661. The underlying bug was connected to panic=abort where lang items were listed in the `missing_lang_items` sets but didn't actually exist anywhere. This caused the linker backend to deduce that start-group/end-group wasn't needed because not all items were defined. Instead the missing lang items that don't actually need to have a definition are filtered out and not considered for the start-group/end-group arguments Closes rust-lang#48661

alexcrichton · 2018-04-04T23:57:55Z

I've posted #49672 with a fix for that

…haelwoerister Fix another circular deps link args issue It turns out that the support in #49316 wasn't enough to handle all cases notably the example in #48661. The underlying bug was connected to panic=abort where lang items were listed in the `missing_lang_items` sets but didn't actually exist anywhere. This caused the linker backend to deduce that start-group/end-group wasn't needed because not all items were defined. Instead the missing lang items that don't actually need to have a definition are filtered out and not considered for the start-group/end-group arguments Closes #48661

rust-highfive assigned michaelwoerister Mar 23, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 23, 2018

alexcrichton force-pushed the start-group-end-group branch from 7394e66 to 60e6480 Compare March 23, 2018 21:41

alexcrichton mentioned this pull request Mar 23, 2018

Multiple codegen units breaks linking: "undefined reference to rust_begin_unwind" #47074

Closed

alexcrichton force-pushed the start-group-end-group branch 3 times, most recently from 3b41480 to eec0d65 Compare March 25, 2018 18:57

japaric mentioned this pull request Mar 27, 2018

Fix linking issues when using multiple codegen units rust-embedded/wg#53

Closed

1 task

michaelwoerister reviewed Mar 27, 2018

View reviewed changes

alexcrichton force-pushed the start-group-end-group branch from eec0d65 to 8ad2f7f Compare March 27, 2018 18:29

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 27, 2018

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Mar 28, 2018

alexcrichton force-pushed the start-group-end-group branch from 8ad2f7f to 50ad5f9 Compare March 28, 2018 23:04

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 28, 2018

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Mar 29, 2018

japaric mentioned this pull request Mar 29, 2018

Initial commit of the second newsletter. Still some TODOs for @japaric rust-embedded/wg#73

Merged

alexcrichton force-pushed the start-group-end-group branch from 50ad5f9 to 88114f6 Compare March 29, 2018 22:07

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 29, 2018

bors merged commit 88114f6 into rust-lang:master Mar 30, 2018

Bobo1239 mentioned this pull request Mar 30, 2018

upgrade OpenSSL to 1.0.2p cross-rs/cross#180

Closed

alexcrichton deleted the start-group-end-group branch March 30, 2018 09:57

glandium mentioned this pull request Mar 31, 2018

It's impossible to write a panic crate for no_std #48661

Closed

alexcrichton mentioned this pull request Apr 4, 2018

Fix another circular deps link args issue #49672

Merged

This was referenced Jun 20, 2018

undefined reference to 'rust_begin_unwind': {start,end}-group missing when using an allocator #51647

Closed

stabilize #[panic_handler] #51366

Merged

eddyb mentioned this pull request Feb 24, 2020

Improve linking of crates with circular dependencies #69371

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustc: Group linked libraries where needed #49316

rustc: Group linked libraries where needed #49316

alexcrichton commented Mar 23, 2018

rust-highfive commented Mar 23, 2018

bors commented Mar 23, 2018

michaelwoerister left a comment

michaelwoerister Mar 27, 2018

alexcrichton Mar 27, 2018

michaelwoerister Mar 27, 2018

alexcrichton Mar 27, 2018

alexcrichton commented Mar 27, 2018

bors commented Mar 27, 2018

bors commented Mar 28, 2018

alexcrichton commented Mar 28, 2018

bors commented Mar 28, 2018

bors commented Mar 29, 2018

bors commented Mar 29, 2018

alexcrichton commented Mar 29, 2018

bors commented Mar 29, 2018

bors commented Mar 29, 2018

bors commented Mar 30, 2018

japaric commented Apr 3, 2018

alexcrichton commented Apr 3, 2018

japaric commented Apr 4, 2018

alexcrichton commented Apr 4, 2018

alexcrichton commented Apr 4, 2018

rustc: Group linked libraries where needed #49316

rustc: Group linked libraries where needed #49316

Conversation

alexcrichton commented Mar 23, 2018

rust-highfive commented Mar 23, 2018

bors commented Mar 23, 2018

michaelwoerister left a comment

Choose a reason for hiding this comment

michaelwoerister Mar 27, 2018

Choose a reason for hiding this comment

alexcrichton Mar 27, 2018

Choose a reason for hiding this comment

michaelwoerister Mar 27, 2018

Choose a reason for hiding this comment

alexcrichton Mar 27, 2018

Choose a reason for hiding this comment

alexcrichton commented Mar 27, 2018

bors commented Mar 27, 2018

bors commented Mar 28, 2018

alexcrichton commented Mar 28, 2018

bors commented Mar 28, 2018

bors commented Mar 29, 2018

bors commented Mar 29, 2018

alexcrichton commented Mar 29, 2018

bors commented Mar 29, 2018

bors commented Mar 29, 2018

bors commented Mar 30, 2018

japaric commented Apr 3, 2018

alexcrichton commented Apr 3, 2018

japaric commented Apr 4, 2018

alexcrichton commented Apr 4, 2018

alexcrichton commented Apr 4, 2018