#[used] attribute #39987

japaric · 2017-02-20T21:13:44Z

(For an explanation of what this feature does, read the commit message)

I'd like to propose landing this as an experimental feature (experimental as in:
no clear stabilization path -- like asm!, #[linkage]) as it's low
maintenance (I think) and relevant to the "Usage in resource-constrained
environments" exploration area.

The main use case I see is running code before main. This could be used, for
instance, to cheaply initialize an allocator before main where the alternative
is to use lazy_static to initialize the allocator on its first use which it's
more expensive (atomics) and doesn't work on ARM Cortex-M0 microcontrollers (no
AtomicUsize on that platform)

Here's a std example of that:

unsafe extern "C" fn before_main_1() {
    println!("Hello");
}

unsafe extern "C" fn before_main_2() {
    println!("World");
}

#[link_section = ".init_arary"]
#[used]
static INIT_ARRAY: [unsafe extern "C" fn(); 2] = [before_main_1, before_main_2];

fn main() {
    println!("Goodbye");
}

$ rustc -C lto -C opt-level=3 before_main.rs
$ ./before_main
Hello
World
Goodbye

In general, this pattern could be used to let dependencies run code before
main (which sounds like it could go very wrong in some cases). There are
probably other use cases; I hope that the people I have cc-ed can comment on
those.

Note that I'm personally unsure if the above pattern is something we want to
promote / allow and that's why I'm proposing this feature as experimental. If
this leads to more footguns than benefits then we can just axe the feature.

cc @nikomatsakis ^ I know you have some thoughts on having a process for
experimental features though I'm fine with writing an RFC before landing this.

dead_code lint will have to be updated to special case #[used] symbols.
Should we extend #[used] to work on non-generic functions?

cc rust-lang/rfcs#1002
cc rust-lang/rfcs#1459
cc @dpc @JinShil

rust-highfive · 2017-02-20T21:13:50Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @arielb1 (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

japaric · 2017-02-20T21:15:29Z

cc @frehberg who also wanted this

durka · 2017-02-20T21:54:36Z

Can you explain that code example some more? You didn't actually put #[used] in it anywhere, and those functions seem to be running even though they were never called. What dark magic are you proposing to add here?

TimNN · 2017-02-20T21:57:26Z

@japaric: I'm a bit confused after reading the PR description:

What does this PR actually do? After looking at the diff, this PR adds the #[used] attribute which marks symbols as llvm.used?
(what @durka said (Especially the dark magic bit!))
Should we extend #[used] to work on non-generic functions?

Was this meant to be ... on generic functions?? (Otherwise I'm a bit confused). (Or is the attribute applied to statics and not functions?)

durka · 2017-02-20T22:05:38Z

The answer to my question is that #[link_section = ".init_array"] (OSX equivalent #[link_section = "__DATA,__mod_init_func"]) should be applied to the static in the example code. Then the claimed output is achieved. Optimizations throw away the static, presumably #[used] (on the static) would preserve it.

durka · 2017-02-20T22:06:30Z

This seems to be an implementation of a closed RFC, which is weird. Shouldn't we go through the regular procedure for adding features?

parched · 2017-02-20T22:06:51Z

This also seems related https://internals.rust-lang.org/t/precise-semantics-of-no-mangle/4098 @cbiffle

japaric · 2017-02-20T22:14:00Z

@durka Sorry about the confusion, added the missing attributes to the example. (I opened the pull request from the command line and the interface to write the PR message strips lines that start with # -- I forgot about that detail)

@TimNN It used the mechanism described here. The commit message has another example.

nagisa · 2017-02-20T22:40:37Z

In general, this pattern could be used to let dependencies run code before
main (which sounds like it could go very wrong in some cases). There are
probably other use cases; I hope that the people I have cc-ed can comment on
those.
Note that I'm personally unsure if the above pattern is something we want to
promote / allow and that's why I'm proposing this feature as experimental. If
this leads to more footguns than benefits then we can just axe the feature.

Life before main is very consciously not supported by Rust. Being able to put arbitrary stuff into arbitrary sections for sure seems useful to me, though in general I feel like it should be more prominently unsafe.

dpc · 2017-02-20T22:46:51Z

I'm happy to someone made an effort to come up with implementation. I understand reluctance to include it, and maybe it should go through formal RFE, but I hope it will at least give a green light to purse it.

Lack of mechanism like this makes some embedded (and not only embedded) things impossible in Rust, as compiler keeps removing stuff we want to retain for various reasons. There's a reason why both GCC and LLVM have a method to retain unused symbols.

The main use case I see is running code before main

Some people need it to preserve build ids, or other artefacts, otherwise not directly referenced from the code. I needed it to implement a self-test for embedded OS, where test functions marked with a macro are gather in one section, and called sequentially on "boot" (so not before main, but in similar fashion). The same mechanism is used by Linux kernel (and other kernels) to implement init system or module initialization.

dpc · 2017-02-20T22:50:44Z

Being able to put arbitrary stuff into arbitrary sections for sure seems useful to me, though in general I feel like it should be more prominently unsafe.

It's not unsafe according to Rust definition. And just to be clear: link_section is already implemented in Rust (and quite necessary for embedded software), but severely crippled by the fact that one can't make sure what has been put there, will actually be there. :)

nagisa · 2017-02-20T22:58:07Z

It's not unsafe according to Rust definition.

Oh, of course it is unsafe – it allows putting arbitrary bytes into executable sections. This includes calling unsafe functions without explicit unsafe annotation (as in example above), dereferencing raw pointers without explicit unsafe annotation, or pretty much anything else, really.

I feel its more of an oversight that link_section is stable in its current form than anything else really. But that’s not really related to the PR at hand anyway.

dpc · 2017-02-20T23:19:42Z

Oh, of course it is unsafe – it allows putting arbitrary bytes into executable sections.

I guess technically it is unsafe to allow putting arbitrary things into initialization section (.init_arary) because they get called sequentially by the platform on start. Putting arbitrary data into other sections, like executable (.code) wouldn't be a problem. You can put bunch of random data between every function in the executable and it just won't be ever called or referenced. To touch it you would have to use some unsafe methods.

However this PR is about adding #[used] as #[linked_section(...)] is already implemented.

durka · 2017-02-21T00:41:08Z

@japaric from your commit message:

Note that the linker knows nothing about #[used] and will drop LIVE
because no other object references to it.

Now I'm confused again. What is the point if this still doesn't defeat the linker?

dpc · 2017-02-21T01:08:21Z

Interesting. I'm confused as well. From http://llvm.org/docs/LangRef.html#the-llvm-used-global-variable:

"a symbol appears in the @llvm.used list, then the compiler, assembler, and linker are required to treat the symbol as if there is a reference to the symbol that it cannot see(...)"

durka · 2017-02-21T01:25:01Z

Putting test::black_box(&INIT_ARRAY); at the beginning of main is another way to preserve the symbol, but I know that's implemented with ASM so I don't know if that covers the embedded use case.

japaric · 2017-02-21T01:40:05Z

@durka

What is the point if this still doesn't defeat the linker?

The point is not to "defeat" the linker but to collaborate with it. That's why you put the #[used] static in a section that the linker will preserve (i.e. KEEP in the linker script). Without #[used] the statics won't even make it to the linker.

Putting test::black_box(&INIT_ARRAY); at the beginning of main

The example in the description is contrived. The proper use case is that you have multiple crates and you want some symbols to make it to the final binary (you mark those as #[used]) but you want those symbols, which would be part of dependencies/libraries, to not be part of the public API of those dependencies/libraries. Thus you can't use the black_box trick in the top (binary) crate.

@dpc

"a symbol appears in the @llvm.used list, then the compiler, assembler, and linker are required to treat the symbol as if there is a reference to the symbol that it cannot see(...)"

I can't speak for LLVM's documentation but the current implementation is the same as clang's. clang's __attribute((used))__-ed items are not treated specially by the linker and will be removed if unused / not referenced. For example:

__attribute__((used))
static const int USED;

static const int UNUSED;

int main() {}

#  NOTE(--gc-sections) matches rustc default
$ clang -Wl,--gc-sections foo.c

$ nm -C a.out | grep USED

# USED does make it to the object file (which is the point of this feature)
$ clang -O3 -c foo.c

$ nm -C foo.o | grep USED
0000000000000000 r USED

codyps · 2017-02-21T01:55:39Z

@dpc are you thinking of gcc's externally_visible, perhaps? avr-llvm/clang#5 https://bugs.llvm.org//show_bug.cgi?id=16683 gcc docs .

Seems it might not be necessary due to the lack of more aggressive LTO in LLVM vs gcc? Or maybe it's just clang?

dpc · 2017-02-21T04:39:03Z

I can't speak for LLVM's documentation but the current implementation is the same as clang's. clang's attribute((used))-ed items are not treated specially by the linker and will be removed if unused / not referenced.

Oh. Last time I was using this gathering stuff in section mechanism was C compiled with GCC, and I don't remember the details now. Maybe it was working due to the fact we were not using LTO, so linker was not that aggressive, or we marked it in the linker script. With Rust I never got this to work, as I was blocked on not being able to emit symbols marked with llvm.used. I think it's OK to assume that #[used] by itself does not solve all the pieces of the puzzle, and might be only one of necessary steps.

@japaric Is KEEP in linker + #[used] enough?

cbiffle · 2017-02-21T17:11:05Z

Remember that the linker deals in sections, not objects. A single referenced symbol in a section is enough to keep that section alive. With GCC, -ffunction-sections and -fdata-sections separate each function and static object into separate sections, and the -gc-sections linker flag turns on aggressive section collection. (Those three flags are also a great way to find bugs in your C build/link configuration.) Marking something (in C) with attribute((used)) is enough to subvert GCC's notion of what is used and what is not (mostly relevant for statics). If the symbol wound up in your binary without special steps in the linker script, you likely assigned it to a section with other used objects and didn't have -fdata-sections turned on. Rust seems to generate each object in a separate section by default, and will thus be subject to more aggressive link-time garbage collection whether you like it or not. So we need #[used] to get past rustc's notion of what is used, and linker script directives to ensure that the symbol is emitted into the binary. To answer your question, yes, it would be sufficient, assuming #[used] is implemented equivalently to Clang __attribute__((used)).

…

On Feb 20, 2017 8:39 PM, "Dawid Ciężarkiewicz" ***@***.***> wrote: I can't speak for LLVM's documentation but the current implementation is the same as clang's. clang's *attribute((used))*-ed items are not treated specially by the linker and will be removed if unused / not referenced. Oh. Last time I was using this gathering stuff in section mechanism was C compiled with GCC, and I don't remember the details now. Maybe it was working due to the fact we were not using LTO, so linker was not that aggressive, or we marked it in the linker script. With Rust I never got this to work, as I was blocked on not being able to emit symbols marked with llvm.used. I think it's OK to assume that #[used] by itself does not solve all the pieces of the puzzle, and might be only one of necessary steps. @japaric <https://github.com/japaric> Is KEEP in linker + #[used] enough? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39987 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACwv-Aj1QidDhAiXX6ACiUP-Vo71vKXks5remp5gaJpZM4MGn6H> .

nikomatsakis · 2017-02-21T19:55:11Z

@japaric I have a question about your initial example. Is this .init_array section something known to the embedded linker or runtime or something? Or just a handy hook that some other bit of user code will invoke before calling main()?

Regarding unstable features, I have had thoughts about having such a process, but I never got as far as writing it down. I think the general idea was going to be that one writes an RFC defining your goals and motivation. If we agree that these cannot be achieved out of tree, we would allow you to experiment in tree, but a proper (and full) RFC is required for the final result of that experimentation. i.e., you don't go directly from "experiment" to "stable feature", but rather you go from "experimental" to "unstable, on path to stabilization" and then "stable". In a sense this is always the path, but typically this "experimental" phase takes place out of tree (e.g., in a library).

dpc · 2017-02-21T20:29:45Z

@nikomatsakis .init_array is a standard mechanism in ELF files, I believe. It will be traversed and executed on by one, by linker on dynamic linking or executable start. It is platform and executable format specific, I'd say. Note - I am not 100% sure about the details, so I reserve a right to be wrong. :)

nikomatsakis · 2017-02-21T20:39:01Z

@dpc ok =) I thought it must be something like that.

codyps · 2017-02-21T20:53:16Z

.init_array: The typical name for the section storing DT_INIT_ARRAY. Part of SysV ABI, and this piece is very widely used even in cases without dynamic linking.

Just an array of function pointers. Only as arch-specific as function pointers are.

durka · 2017-02-21T21:13:00Z

Well no, it is more arch-specific than that because this section's name has meaning to the executable loader.

…

On Tue, Feb 21, 2017 at 3:53 PM, Cody Schafer ***@***.***> wrote: .init_array: The typical name for the section storing DT_INIT_ARRAY. Part of SysV ABI, and this piece is very widely used even in cases without dynamic linking. Just an array of function pointers. Only as arch-specific as function pointers are. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39987 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC3n5DsWQ_6EHLVhk9sAGzk_SVZahN1ks5re07SgaJpZM4MGn6H> .

as it's specific to ELF and won't pass on macOS / Windows

japaric · 2017-04-06T23:34:51Z

Thanks for the info, @eddyb.

I'm just going to ignore the test because I don't have hardware to tests the other methods for macOS / Windows.

@bors r=arielb1

bors · 2017-04-06T23:34:52Z

📌 Commit f4f79c3 has been approved by arielb1

bors · 2017-04-07T01:27:27Z

⌛ Testing commit f4f79c3 with merge cc966cf...

@nikomatsakis

(For an explanation of what this feature does, read the commit message) I'd like to propose landing this as an experimental feature (experimental as in: no clear stabilization path -- like `asm!`, `#[linkage]`) as it's low maintenance (I think) and relevant to the "Usage in resource-constrained environments" exploration area. The main use case I see is running code before `main`. This could be used, for instance, to cheaply initialize an allocator before `main` where the alternative is to use `lazy_static` to initialize the allocator on its first use which it's more expensive (atomics) and doesn't work on ARM Cortex-M0 microcontrollers (no `AtomicUsize` on that platform) Here's a `std` example of that: ``` rust unsafe extern "C" fn before_main_1() { println!("Hello"); } unsafe extern "C" fn before_main_2() { println!("World"); } static INIT_ARRAY: [unsafe extern "C" fn(); 2] = [before_main_1, before_main_2]; fn main() { println!("Goodbye"); } ``` ``` $ rustc -C lto -C opt-level=3 before_main.rs $ ./before_main Hello World Goodbye ``` In general, this pattern could be used to let *dependencies* run code before `main` (which sounds like it could go very wrong in some cases). There are probably other use cases; I hope that the people I have cc-ed can comment on those. Note that I'm personally unsure if the above pattern is something we want to promote / allow and that's why I'm proposing this feature as experimental. If this leads to more footguns than benefits then we can just axe the feature. cc @nikomatsakis ^ I know you have some thoughts on having a process for experimental features though I'm fine with writing an RFC before landing this. - `dead_code` lint will have to be updated to special case `#[used]` symbols. - Should we extend `#[used]` to work on non-generic functions? cc rust-lang/rfcs#1002 cc rust-lang/rfcs#1459 cc @dpc @JinShil

frewsxcv · 2017-04-07T01:34:51Z

Prioritizing the rollup that includes these changes

@bors retry

bors · 2017-04-07T02:48:15Z

⌛ Testing commit f4f79c3 with merge ccdf891...

bors · 2017-04-07T04:28:06Z

💔 Test failed - status-travis

durka · 2017-04-07T04:34:34Z

[01:21:10] ---- [run-make] run-make/used stdout ----
[01:21:10] 	
[01:21:10] error: make failed
[01:21:10] status: exit code: 2
[01:21:10] command: "make"
[01:21:10] stdout:
[01:21:10] ------------------------------------------
[01:21:10] DYLD_LIBRARY_PATH="/Users/travis/build/rust-lang/rust/build/x86_64-apple-darwin/test/run-make/used.stage2-x86_64-apple-darwin:/Users/travis/build/rust-lang/rust/build/x86_64-apple-darwin/stage2/lib:" '/Users/travis/build/rust-lang/rust/build/x86_64-apple-darwin/stage2/bin/rustc' --out-dir /Users/travis/build/rust-lang/rust/build/x86_64-apple-darwin/test/run-make/used.stage2-x86_64-apple-darwin -L /Users/travis/build/rust-lang/rust/build/x86_64-apple-darwin/test/run-make/used.stage2-x86_64-apple-darwin  -C opt-level=3 --emit=obj used.rs
[01:21:10] nm -C /Users/travis/build/rust-lang/rust/build/x86_64-apple-darwin/test/run-make/used.stage2-x86_64-apple-darwin/used.o | grep FOO
[01:21:10] 
[01:21:10] ------------------------------------------
[01:21:10] stderr:
[01:21:10] ------------------------------------------
[01:21:10] warning: static item is never used: `FOO`
[01:21:10]   --> used.rs:15:1
[01:21:10]    |
[01:21:10] 15 | static FOO: u32 = 0;
[01:21:10]    | ^^^^^^^^^^^^^^^^^^^^
[01:21:10]    |
[01:21:10]    = note: #[warn(dead_code)] on by default
[01:21:10] 
[01:21:10] warning: static item is never used: `BAR`
[01:21:10]   --> used.rs:17:1
[01:21:10]    |
[01:21:10] 17 | static BAR: u32 = 0;
[01:21:10]    | ^^^^^^^^^^^^^^^^^^^^
[01:21:10]    |
[01:21:10]    = note: #[warn(dead_code)] on by default
[01:21:10] 
[01:21:10] nm: Unknown command line argument '-C'.  Try: '/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm -help'
[01:21:10] nm: Did you mean '-A'?
[01:21:10] make[1]: *** [all] Error 1

OSX nm doesn't seem to support demangling, apparently you need to use c++filt directly.

the nm in our macOS bots don't support that flag and it's not really required

japaric · 2017-04-07T04:55:24Z

Thanks for pointing out the exact error, @durka. I was having trouble spotting it.

@bors r=arielb1

bors · 2017-04-07T04:55:24Z

📌 Commit 98037ca has been approved by arielb1

bors · 2017-04-07T04:56:50Z

⌛ Testing commit 98037ca with merge b9c5197...

@nikomatsakis

#[used] attribute (For an explanation of what this feature does, read the commit message) I'd like to propose landing this as an experimental feature (experimental as in: no clear stabilization path -- like `asm!`, `#[linkage]`) as it's low maintenance (I think) and relevant to the "Usage in resource-constrained environments" exploration area. The main use case I see is running code before `main`. This could be used, for instance, to cheaply initialize an allocator before `main` where the alternative is to use `lazy_static` to initialize the allocator on its first use which it's more expensive (atomics) and doesn't work on ARM Cortex-M0 microcontrollers (no `AtomicUsize` on that platform) Here's a `std` example of that: ``` rust unsafe extern "C" fn before_main_1() { println!("Hello"); } unsafe extern "C" fn before_main_2() { println!("World"); } #[link_section = ".init_arary"] #[used] static INIT_ARRAY: [unsafe extern "C" fn(); 2] = [before_main_1, before_main_2]; fn main() { println!("Goodbye"); } ``` ``` $ rustc -C lto -C opt-level=3 before_main.rs $ ./before_main Hello World Goodbye ``` In general, this pattern could be used to let *dependencies* run code before `main` (which sounds like it could go very wrong in some cases). There are probably other use cases; I hope that the people I have cc-ed can comment on those. Note that I'm personally unsure if the above pattern is something we want to promote / allow and that's why I'm proposing this feature as experimental. If this leads to more footguns than benefits then we can just axe the feature. cc @nikomatsakis ^ I know you have some thoughts on having a process for experimental features though I'm fine with writing an RFC before landing this. - `dead_code` lint will have to be updated to special case `#[used]` symbols. - Should we extend `#[used]` to work on non-generic functions? cc rust-lang/rfcs#1002 cc rust-lang/rfcs#1459 cc @dpc @JinShil

bors · 2017-04-07T07:22:04Z

☀️ Test successful - status-appveyor, status-travis
Approved by: arielb1
Pushing b9c5197 to master...

Centril · 2018-02-23T11:22:15Z

Triaging rust-lang/rfcs#1002 - @japaric, what's the status of this wrt. stabilization and an RFC?

rust-highfive assigned arielb1 Feb 20, 2017

This was referenced Feb 20, 2017

Pre-RFC: Rust version of __attribute(used)__ (GCC) or llvm.used (LLVM). rust-lang/rfcs#1002

Closed

Add a used attribute rust-lang/rfcs#1459

Closed

japaric force-pushed the used branch from c1cdf3f to 7fe68d5 Compare February 20, 2017 21:22

japaric force-pushed the used branch from 7fe68d5 to 5d9594a Compare February 20, 2017 22:25

ignore the .init_array doctest

f4f79c3

as it's specific to ELF and won't pass on macOS / Windows

frewsxcv mentioned this pull request Apr 7, 2017

Rollup of 6 pull requests #41129

Closed

don't pass -C to nm

98037ca

the nm in our macOS bots don't support that flag and it's not really required

bors merged commit 98037ca into rust-lang:master Apr 7, 2017

This was referenced Apr 7, 2017

:vis matcher for macro_rules #41012

Merged

Implement RFC 1268 #40097

Closed

japaric deleted the used branch April 7, 2017 15:38

eddyb mentioned this pull request Apr 13, 2017

Hide __LOAD_NEON_MODULE by using #[used] on a private static. neon-bindings/neon#204

Closed

japaric mentioned this pull request Apr 15, 2017

Crate failed to build rust-lang/docs.rs#23

Closed

japaric mentioned this pull request Apr 3, 2018

RFC: #[used] static variables rust-lang/rfcs#2386

Merged

OlegTheCat mentioned this pull request Apr 8, 2019

Extern functions are not detected by Kaleidoscope JIT TheDan64/inkwell#74

Closed

#[used] attribute #39987

#[used] attribute #39987

Conversation

japaric commented Feb 20, 2017 • edited by pnkfelix

rust-highfive commented Feb 20, 2017

japaric commented Feb 20, 2017

durka commented Feb 20, 2017 • edited

TimNN commented Feb 20, 2017 • edited

durka commented Feb 20, 2017

durka commented Feb 20, 2017

parched commented Feb 20, 2017

japaric commented Feb 20, 2017 • edited

nagisa commented Feb 20, 2017

dpc commented Feb 20, 2017 • edited

dpc commented Feb 20, 2017 • edited

nagisa commented Feb 20, 2017 • edited

dpc commented Feb 20, 2017 • edited

durka commented Feb 21, 2017

dpc commented Feb 21, 2017

durka commented Feb 21, 2017

japaric commented Feb 21, 2017

codyps commented Feb 21, 2017

dpc commented Feb 21, 2017

cbiffle commented Feb 21, 2017 via email

nikomatsakis commented Feb 21, 2017

dpc commented Feb 21, 2017

nikomatsakis commented Feb 21, 2017

codyps commented Feb 21, 2017

durka commented Feb 21, 2017 via email

japaric commented Apr 6, 2017

bors commented Apr 6, 2017

bors commented Apr 7, 2017

frewsxcv commented Apr 7, 2017

bors commented Apr 7, 2017

bors commented Apr 7, 2017

durka commented Apr 7, 2017 • edited

japaric commented Apr 7, 2017

bors commented Apr 7, 2017

bors commented Apr 7, 2017

bors commented Apr 7, 2017

Centril commented Feb 23, 2018

japaric commented Feb 20, 2017 •

edited by pnkfelix

durka commented Feb 20, 2017 •

edited

TimNN commented Feb 20, 2017 •

edited

japaric commented Feb 20, 2017 •

edited

dpc commented Feb 20, 2017 •

edited

dpc commented Feb 20, 2017 •

edited

nagisa commented Feb 20, 2017 •

edited

dpc commented Feb 20, 2017 •

edited

durka commented Apr 7, 2017 •

edited