#[used] attribute

[japaric]

(For an explanation of what this feature does, read the commit message)

I'd like to propose landing this as an experimental feature (experimental as in:
no clear stabilization path -- like asm!, #[linkage]) as it's low
maintenance (I think) and relevant to the "Usage in resource-constrained
environments" exploration area.

The main use case I see is running code before main. This could be used, for
instance, to cheaply initialize an allocator before main where the alternative
is to use lazy_static to initialize the allocator on its first use which it's
more expensive (atomics) and doesn't work on ARM Cortex-M0 microcontrollers (no
AtomicUsize on that platform)

Here's a std example of that:

unsafe extern "C" fn before_main_1() {
    println!("Hello");
}

unsafe extern "C" fn before_main_2() {
    println!("World");
}

#[link_section = ".init_arary"]
#[used]
static INIT_ARRAY: [unsafe extern "C" fn(); 2] = [before_main_1, before_main_2];

fn main() {
    println!("Goodbye");
}

$ rustc -C lto -C opt-level=3 before_main.rs
$ ./before_main
Hello
World
Goodbye

In general, this pattern could be used to let dependencies run code before
main (which sounds like it could go very wrong in some cases). There are
probably other use cases; I hope that the people I have cc-ed can comment on
those.

Note that I'm personally unsure if the above pattern is something we want to
promote / allow and that's why I'm proposing this feature as experimental. If
this leads to more footguns than benefits then we can just axe the feature.

cc @nikomatsakis ^ I know you have some thoughts on having a process for
experimental features though I'm fine with writing an RFC before landing this.

• dead_code lint will have to be updated to special case #[used] symbols.

• Should we extend #[used] to work on non-generic functions?

cc rust-lang/rfcs#1002
cc rust-lang/rfcs#1459
cc @dpc @JinShil

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @arielb1 (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[japaric]

cc @frehberg who also wanted this

[durka]

Can you explain that code example some more? You didn't actually put #[used] in it anywhere, and those functions seem to be running even though they were never called. What dark magic are you proposing to add here?

[TimNN]

@japaric: I'm a bit confused after reading the PR description:

• What does this PR actually do? After looking at the diff, this PR adds the #[used] attribute which marks symbols as llvm.used?

• (what @durka said (Especially the dark magic bit!))

• Should we extend #[used] to work on non-generic functions?

  Was this meant to be ... on generic functions?? (Otherwise I'm a bit confused). (Or is the attribute applied to statics and not functions?)

[durka]

The answer to my question is that #[link_section = ".init_array"] (OSX equivalent #[link_section = "__DATA,__mod_init_func"]) should be applied to the static in the example code. Then the claimed output is achieved. Optimizations throw away the static, presumably #[used] (on the static) would preserve it.

[durka]

This seems to be an implementation of a closed RFC, which is weird. Shouldn't we go through the regular procedure for adding features?

[parched]

This also seems related https://internals.rust-lang.org/t/precise-semantics-of-no-mangle/4098 @cbiffle

[japaric]

@durka Sorry about the confusion, added the missing attributes to the example. (I opened the pull request from the command line and the interface to write the PR message strips lines that start with # -- I forgot about that detail)

@TimNN It used the mechanism described here. The commit message has another example.

[nagisa]

In general, this pattern could be used to let dependencies run code before
main (which sounds like it could go very wrong in some cases). There are
probably other use cases; I hope that the people I have cc-ed can comment on
those.
Note that I'm personally unsure if the above pattern is something we want to
promote / allow and that's why I'm proposing this feature as experimental. If
this leads to more footguns than benefits then we can just axe the feature.

Life before main is very consciously not supported by Rust. Being able to put arbitrary stuff into arbitrary sections for sure seems useful to me, though in general I feel like it should be more prominently unsafe.

[dpc]

I'm happy to someone made an effort to come up with implementation. I understand reluctance to include it, and maybe it should go through formal RFE, but I hope it will at least give a green light to purse it.

Lack of mechanism like this makes some embedded (and not only embedded) things impossible in Rust, as compiler keeps removing stuff we want to retain for various reasons. There's a reason why both GCC and LLVM have a method to retain unused symbols.

The main use case I see is running code before main

Some people need it to preserve build ids, or other artefacts, otherwise not directly referenced from the code. I needed it to implement a self-test for embedded OS, where test functions marked with a macro are gather in one section, and called sequentially on "boot" (so not before main, but in similar fashion). The same mechanism is used by Linux kernel (and other kernels) to implement init system or module initialization.

[dpc]

Being able to put arbitrary stuff into arbitrary sections for sure seems useful to me, though in general I feel like it should be more prominently unsafe.

It's not unsafe according to Rust definition. And just to be clear: link_section is already implemented in Rust (and quite necessary for embedded software), but severely crippled by the fact that one can't make sure what has been put there, will actually be there. :)

[nagisa]

It's not unsafe according to Rust definition.

Oh, of course it is unsafe – it allows putting arbitrary bytes into executable sections. This includes calling unsafe functions without explicit unsafe annotation (as in example above), dereferencing raw pointers without explicit unsafe annotation, or pretty much anything else, really.

I feel its more of an oversight that link_section is stable in its current form than anything else really. But that’s not really related to the PR at hand anyway.

[dpc]

Oh, of course it is unsafe – it allows putting arbitrary bytes into executable sections.

I guess technically it is unsafe to allow putting arbitrary things into initialization section (.init_arary) because they get called sequentially by the platform on start. Putting arbitrary data into other sections, like executable (.code) wouldn't be a problem. You can put bunch of random data between every function in the executable and it just won't be ever called or referenced. To touch it you would have to use some unsafe methods.

However this PR is about adding #[used] as #[linked_section(...)] is already implemented.

[durka]

@japaric from your commit message:

Note that the linker knows nothing about #[used] and will drop LIVE
because no other object references to it.

Now I'm confused again. What is the point if this still doesn't defeat the linker?

[dpc]

Interesting. I'm confused as well. From http://llvm.org/docs/LangRef.html#the-llvm-used-global-variable:

"a symbol appears in the @llvm.used list, then the compiler, assembler, and linker are required to treat the symbol as if there is a reference to the symbol that it cannot see(...)"

[durka]

Putting test::black_box(&INIT_ARRAY); at the beginning of main is another way to preserve the symbol, but I know that's implemented with ASM so I don't know if that covers the embedded use case.

[japaric]

@durka

What is the point if this still doesn't defeat the linker?

The point is not to "defeat" the linker but to collaborate with it. That's why you put the #[used] static in a section that the linker will preserve (i.e. KEEP in the linker script). Without #[used] the statics won't even make it to the linker.

Putting test::black_box(&INIT_ARRAY); at the beginning of main

The example in the description is contrived. The proper use case is that you have multiple crates and you want some symbols to make it to the final binary (you mark those as #[used]) but you want those symbols, which would be part of dependencies/libraries, to not be part of the public API of those dependencies/libraries. Thus you can't use the black_box trick in the top (binary) crate.

@dpc

"a symbol appears in the @llvm.used list, then the compiler, assembler, and linker are required to treat the symbol as if there is a reference to the symbol that it cannot see(...)"

I can't speak for LLVM's documentation but the current implementation is the same as clang's. clang's __attribute((used))__-ed items are not treated specially by the linker and will be removed if unused / not referenced. For example:

__attribute__((used))
static const int USED;

static const int UNUSED;

int main() {}

#  NOTE(--gc-sections) matches rustc default
$ clang -Wl,--gc-sections foo.c

$ nm -C a.out | grep USED

# USED does make it to the object file (which is the point of this feature)
$ clang -O3 -c foo.c

$ nm -C foo.o | grep USED
0000000000000000 r USED

[jmesmon]

@dpc are you thinking of gcc's externally_visible, perhaps? avr-llvm/clang#5 https://bugs.llvm.org//show_bug.cgi?id=16683 gcc docs .

Seems it might not be necessary due to the lack of more aggressive LTO in LLVM vs gcc? Or maybe it's just clang?

[dpc]

I can't speak for LLVM's documentation but the current implementation is the same as clang's. clang's attribute((used))-ed items are not treated specially by the linker and will be removed if unused / not referenced.

Oh. Last time I was using this gathering stuff in section mechanism was C compiled with GCC, and I don't remember the details now. Maybe it was working due to the fact we were not using LTO, so linker was not that aggressive, or we marked it in the linker script. With Rust I never got this to work, as I was blocked on not being able to emit symbols marked with llvm.used. I think it's OK to assume that #[used] by itself does not solve all the pieces of the puzzle, and might be only one of necessary steps.

@japaric Is KEEP in linker + #[used] enough?

[cbiffle]

Remember that the linker deals in sections, not objects. A single referenced symbol in a section is enough to keep that section alive. With GCC, -ffunction-sections and -fdata-sections separate each function and static object into separate sections, and the -gc-sections linker flag turns on aggressive section collection. (Those three flags are also a great way to find bugs in your C build/link configuration.) Marking something (in C) with attribute((used)) is enough to subvert GCC's notion of what is used and what is not (mostly relevant for statics). If the symbol wound up in your binary without special steps in the linker script, you likely assigned it to a section with other used objects and didn't have -fdata-sections turned on. Rust seems to generate each object in a separate section by default, and will thus be subject to more aggressive link-time garbage collection whether you like it or not. So we need #[used] to get past rustc's notion of what is used, and linker script directives to ensure that the symbol is emitted into the binary. To answer your question, yes, it would be sufficient, assuming #[used] is implemented equivalently to Clang __attribute__((used)).

…

On Feb 20, 2017 8:39 PM, "Dawid Ciężarkiewicz" ***@***.***> wrote: I can't speak for LLVM's documentation but the current implementation is the same as clang's. clang's *attribute((used))*-ed items are not treated specially by the linker and will be removed if unused / not referenced. Oh. Last time I was using this gathering stuff in section mechanism was C compiled with GCC, and I don't remember the details now. Maybe it was working due to the fact we were not using LTO, so linker was not that aggressive, or we marked it in the linker script. With Rust I never got this to work, as I was blocked on not being able to emit symbols marked with llvm.used. I think it's OK to assume that #[used] by itself does not solve all the pieces of the puzzle, and might be only one of necessary steps. @japaric <https://github.com/japaric> Is KEEP in linker + #[used] enough? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39987 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACwv-Aj1QidDhAiXX6ACiUP-Vo71vKXks5remp5gaJpZM4MGn6H> .

[nikomatsakis]

@japaric I have a question about your initial example. Is this .init_array section something known to the embedded linker or runtime or something? Or just a handy hook that some other bit of user code will invoke before calling main()?

Regarding unstable features, I have had thoughts about having such a process, but I never got as far as writing it down. I think the general idea was going to be that one writes an RFC defining your goals and motivation. If we agree that these cannot be achieved out of tree, we would allow you to experiment in tree, but a proper (and full) RFC is required for the final result of that experimentation. i.e., you don't go directly from "experiment" to "stable feature", but rather you go from "experimental" to "unstable, on path to stabilization" and then "stable". In a sense this is always the path, but typically this "experimental" phase takes place out of tree (e.g., in a library).

[dpc]

@nikomatsakis .init_array is a standard mechanism in ELF files, I believe. It will be traversed and executed on by one, by linker on dynamic linking or executable start. It is platform and executable format specific, I'd say. Note - I am not 100% sure about the details, so I reserve a right to be wrong. :)

[nikomatsakis]

@dpc ok =) I thought it must be something like that.

[jmesmon]

.init_array: The typical name for the section storing DT_INIT_ARRAY. Part of SysV ABI, and this piece is very widely used even in cases without dynamic linking.

Just an array of function pointers. Only as arch-specific as function pointers are.

[durka]

Well no, it is more arch-specific than that because this section's name has meaning to the executable loader.

…

On Tue, Feb 21, 2017 at 3:53 PM, Cody Schafer ***@***.***> wrote: .init_array: The typical name for the section storing DT_INIT_ARRAY. Part of SysV ABI, and this piece is very widely used even in cases without dynamic linking. Just an array of function pointers. Only as arch-specific as function pointers are. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39987 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC3n5DsWQ_6EHLVhk9sAGzk_SVZahN1ks5re07SgaJpZM4MGn6H> .

[eddyb]

@japaric Each platform has its own way to do this.

[japaric]

Thanks for the info, @eddyb.

I'm just going to ignore the test because I don't have hardware to tests the other methods for macOS / Windows.

@bors r=arielb1

[bors]

📌 Commit f4f79c3 has been approved by arielb1

[bors]

⌛️ Testing commit f4f79c3 with merge cc966cf...

[frewsxcv]

Prioritizing the rollup that includes these changes

@bors retry

[bors]

⌛️ Testing commit f4f79c3 with merge ccdf891...

[bors]

💔 Test failed - status-travis

[durka]

[01:21:10] ---- [run-make] run-make/used stdout ----
[01:21:10] 	
[01:21:10] error: make failed
[01:21:10] status: exit code: 2
[01:21:10] command: "make"
[01:21:10] stdout:
[01:21:10] ------------------------------------------
[01:21:10] DYLD_LIBRARY_PATH="/Users/travis/build/rust-lang/rust/build/x86_64-apple-darwin/test/run-make/used.stage2-x86_64-apple-darwin:/Users/travis/build/rust-lang/rust/build/x86_64-apple-darwin/stage2/lib:" '/Users/travis/build/rust-lang/rust/build/x86_64-apple-darwin/stage2/bin/rustc' --out-dir /Users/travis/build/rust-lang/rust/build/x86_64-apple-darwin/test/run-make/used.stage2-x86_64-apple-darwin -L /Users/travis/build/rust-lang/rust/build/x86_64-apple-darwin/test/run-make/used.stage2-x86_64-apple-darwin  -C opt-level=3 --emit=obj used.rs
[01:21:10] nm -C /Users/travis/build/rust-lang/rust/build/x86_64-apple-darwin/test/run-make/used.stage2-x86_64-apple-darwin/used.o | grep FOO
[01:21:10] 
[01:21:10] ------------------------------------------
[01:21:10] stderr:
[01:21:10] ------------------------------------------
[01:21:10] warning: static item is never used: `FOO`
[01:21:10]   --> used.rs:15:1
[01:21:10]    |
[01:21:10] 15 | static FOO: u32 = 0;
[01:21:10]    | ^^^^^^^^^^^^^^^^^^^^
[01:21:10]    |
[01:21:10]    = note: #[warn(dead_code)] on by default
[01:21:10] 
[01:21:10] warning: static item is never used: `BAR`
[01:21:10]   --> used.rs:17:1
[01:21:10]    |
[01:21:10] 17 | static BAR: u32 = 0;
[01:21:10]    | ^^^^^^^^^^^^^^^^^^^^
[01:21:10]    |
[01:21:10]    = note: #[warn(dead_code)] on by default
[01:21:10] 
[01:21:10] nm: Unknown command line argument '-C'.  Try: '/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm -help'
[01:21:10] nm: Did you mean '-A'?
[01:21:10] make[1]: *** [all] Error 1

OSX nm doesn't seem to support demangling, apparently you need to use c++filt directly.

[japaric]

Thanks for pointing out the exact error, @durka. I was having trouble spotting it.

@bors r=arielb1

[bors]

📌 Commit 98037ca has been approved by arielb1

[bors]

⌛️ Testing commit 98037ca with merge b9c5197...

[bors]

☀️ Test successful - status-appveyor, status-travis
Approved by: arielb1
Pushing b9c5197 to master...

[Centril]

Triaging rust-lang/rfcs#1002 - @japaric, what's the status of this wrt. stabilization and an RFC?