Compute `target_feature` from LLVM #31709

ranma42 · 2016-02-16T16:21:36Z

This is a work-in-progress fix for #31662.

The logic that computes the target features from the command line has been replaced with queries to the TargetMachine.

rust-highfive · 2016-02-16T16:21:50Z

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

ranma42 · 2016-02-16T16:27:56Z

I tried this and it has the desired result for the example of #31662, but I think it might need more work.
In particular, I am not confident about several issues:

is this an acceptable approach?
is a string the desired output type?
is it ok to work on a TargetMachine instance?
would it be ok to initialise LLVM earlier than in trans? (would it be ok to do it from build_configuration?)

I do not think this is ready for merge, but I would love feedback and directions.

alexcrichton · 2016-02-16T18:07:19Z

src/librustc_driver/target_features.rs

-        has_avx, "avx";
-        has_avx2, "avx2";
-        has_neon, "neon";
-        has_vfp, "vfp";


One of the purposes of a macro like this was to ensure that we had control over the names exported. We don't want to blanket export all features that LLVM itself exposes, and we also don't want to tie ourselves to exactly the names LLVM has. In other words this is here to ensure we can provide a stable interface over the possibly unstable interface of LLVM.

Filtering and renaming these features to a "rust" convention instead of the LLVM one can be done very easily with a map. The filtering makes sense to me, as rust might want to expose only a subset of the LLVM features through cfg!.
Renaming these features is just as easy, but I would be surprised if it happened only here and not in every place where a feature is named (for example when they are passed through the -C target-feature flag). That would result in an inconsistent naming, as the same feature would have a different name in the command line flags and in the code.
If you confirm that renaming is desired here and not elsewhere, I will implement it, otherwise I believe that it might belong to another PR (under the assumption that if LLVM and rust feature naming was to diverge in future, we would decide how to take care of that and possibly perform a consistent renaming everywhere).

The purpose of this macro was that for the function/name pair the cfg directive name would be defined if the function returned true. This is the only place that does that.

Basically, we need to filter what's coming out of LLVM with an explicit whitelist of what cfg directives are defined. We don't need to map names yet (as we just use LLVM names), but we may support that one day.

I added a whitelist of known LLVM features which is used to filter which ones are exposed through cfg!.

nikomatsakis · 2016-02-16T21:48:52Z

r? @alexcrichton

ranma42 · 2016-02-17T17:30:56Z

I moved the initialisation of LLVM to librustc. In order to avoid a circular dependency between crates, I had to move configure_llvm. This might not be correct, as it was a public function. Currently I made it private, but if it is meant to be part of the public interface of librustc_trans it might need to be reexported or moved to a crate which can be used/exported by librustc and librustc_trans.

alexcrichton · 2016-02-17T19:28:17Z

src/librustc/session/config.rs

@@ -720,6 +750,27 @@ pub fn build_configuration(sess: &Session) -> ast::CrateConfig {
    }
    let mut v = user_cfg.into_iter().collect::<Vec<_>>();
    v.extend_from_slice(&default_cfg[..]);
+
+    // Before we touch LLVM, make sure that multithreading is enabled.
+    unsafe {


This should be extracted into its own function instead of being copy/pasted into an otherwise irrelevant function. This should also probably be called from a more principled location such as the driver itself

You mean in librustc_driver? AFAICT it is not possible to invoke functions in librustc_driver from librustc because the crates would have a circular dependency. Should I also try to move build_configuration to another crate? (is it possible/does it make sense?)

Yeah, it'd just be an implicit contract that the driver must initialize LLVM before calling this function.

ranma42 · 2016-03-02T20:45:34Z

I tried to implement all of the suggested changes. It might be possible that PNaCl needs special care if it has features we want to expose, because it does not look like a normal LLVM target. So far this should not be an issue, as all of the features that are whitelisted are from other targets.

I added the init_llvm function in librustc_trans::back::write and it should be called before any use of LLVM. Is src/librustc/README.md the most appropriate place to warn people that init_llvm should be called?

alexcrichton · 2016-03-03T01:32:48Z

Unfortunately I think the makefiles will also need to change to account for the change here, rustbuild is only an alternate build system currently. It's ok to not worry much about pnacl for now though.

And yeah, init_llvm looks good to me. It's probably fine to leave it sprinkled as-is, it's not clear to me anyway about who's supposed to call it anyway.

ranma42 · 2016-03-03T23:26:50Z

I added the changes to Makefiles (based on the existing build templates from mk/main.mk and mk/llvm.mk) and fixed the tests that were using LLVM to invoke init_llvm as part of the session construction.

alexcrichton · 2016-03-04T00:52:27Z

mk/rustllvm.mk

@@ -43,6 +43,10 @@ $$(RT_OUTPUT_DIR_$(1))/$$(call CFG_STATIC_LIB_NAME_$(1),rustllvm): \
 	@$$(call E, link: $$@)
 	$$(Q)$$(call CFG_CREATE_ARCHIVE_$(1),$$@) $$^

+EXTRA_RUSTLLVM_CXXFLAGS_$(1) = $$(shell echo $$(filter $$(LLVM_ALL_COMPONENTS_$(1)),\
+	$$(LLVM_SUBTARGET_COMPONENTS)) | tr 'a-z' 'A-Z' |\
+	sed -e 's/^ //;s/\([^ ]*\)/\-DLLVM_SUBTARGET_\1/g')


Perhaps this could just be:

EXTRA_RUSTLLVM_CXXFLAGS_$(1) = $$(LLVM_ALL_COMPONENTS_$(1):%=-DLLVM_COMPONENT_%)

That way we could just ferry all components down to the C++ code?

Oops, you just made me realise that I should not be overwriting EXTRA_RUSTLLVM_CXXFLAGS_$(1), I meant to use the RUSTLLVM_TARGETS_$(1) variable!
Anyway, yes, that would work and looks much simpler, but it will result in mixed-case defines. Is it ok?

Hmm... it does not work, because the - character is not allowed in a define, but it is used in several components. Something like this seems to work:

RUSTLLVM_COMPONENTS_$(1) = $$(shell echo $$(LLVM_ALL_COMPONENTS_$(1)) |\ tr 'a-z-' 'A-Z_'| sed -e 's/^ //;s/$[^ ]*$/\-DLLVM_COMPONENT_\1/g')

and still avoids the distinction between target and non-target components.

ranma42 · 2016-03-04T10:09:13Z

@alexcrichton I tried to implement both your suggestions and the overall change now looks cleaner :)
I kept the new commits separate, but it might be a good idea to squash them before merging so that the changes that have been reverted (those to the tests, for example) are cleaned up.

alexcrichton · 2016-03-04T19:42:25Z

src/librustc_llvm/build.rs

+    for component in components {
+        let mut flag = String::from("-DLLVM_COMPONENT_");
+        flag.push_str(&component.to_uppercase());
+        cfg.flag(&flag);


zomg this looks so much nicer than make

alexcrichton · 2016-03-04T19:47:27Z

Awesome, looking great to me, thanks @ranma42!

Could you add some tests for this as well? I think the case in #31662 may be a good one to add?

alexcrichton · 2016-03-04T19:47:44Z

Also yeah I'd be fine squashing commits before landing as well.

alexcrichton · 2016-03-04T19:48:23Z

Also, out of curiosity, did you get a response from asking LLVM about these constants?

ranma42 · 2016-03-08T10:41:42Z

I tried to adapt the program from #31662, but it is not trivial to make a test which is correct for all possible targets. AFAICT the idea is that the test should check whether the sse feature is available on i686-unknown-linux-gnu. How can I write a test that is only run on a specific triple? I would like to avoid having the test broken on i586-unknown-linux-gnu.

Would a print-cfg test like the following be sufficient?

$(RUSTC) --target i686-unknown-linux-gnu --print cfg | grep sse2

This test would not pass on stable/beta release channels until target_feature is stabilised. Is this a problem?

This morning I sent a message to the llvm-dev mailing list pointing out this approach for computing the features and asking for guidance. The relevant thread starts here.

alexcrichton · 2016-03-08T16:43:44Z

Ah yeah a test like that should be fine. We've export the RUST_BOOTSTRAP_KEY during the CI builds of the stable/beta channels so I think that will pass even for those channels?

alexcrichton · 2016-03-08T16:44:01Z

Also thanks for starting that thread! It's probably worth waiting a little bit to see if anyone responds.

bors · 2016-03-27T02:51:16Z

☔ The latest upstream changes (presumably #32432) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2016-04-20T01:05:23Z

💔 Test failed - auto-win-msvc-32-opt

huonw · 2016-04-20T01:10:41Z

src/librustc_driver/target_features.rs

-pub fn has_avx2(sess: &Session) -> bool {
-    features_contain(sess, "+avx2")
-}
+    let x86_whitelist = [


These could be const X86_WHITELIST: &'static [&'static str] = &["..."]; instead of locals (may neaten the code slightly, since it should avoid the need for &...[..] below too).

alexcrichton · 2016-04-20T01:11:03Z

@bors: retry

On Tuesday, April 19, 2016, bors notifications@github.com wrote:

[image: 💔] Test failed - auto-win-msvc-32-opt
http://buildbot.rust-lang.org/builders/auto-win-msvc-32-opt/builds/3068

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#31709 (comment)

huonw · 2016-04-20T01:11:45Z

src/librustc_driver/target_features.rs

+
+    let tf = InternedString::new("target_feature");
+    for feat in whitelist {
+        if unsafe { LLVMRustHasFeature(target_machine, feat.as_ptr() as *const c_char) } {


It might be nice to have an assertion that the strings end in \0 here, to ensure people don't make a mistake when adding features in future.

huonw · 2016-04-20T01:14:00Z

Thanks for doing this! (Sorry for the delayed comments, been away.)

bors · 2016-04-20T04:41:31Z

⌛ Testing commit 1ad8561 with merge 0a0296e...

bors · 2016-04-20T05:53:16Z

💔 Test failed - auto-win-msvc-32-opt

This simplifies the code a bit and makes the types nicer, too.

Assert that the feature strings are NUL terminated, so that they will be well-formed as C strings. This is a safety check to ease the maintaninace and update of the feature lists.

ranma42 · 2016-04-20T07:44:36Z

Applied the improvements suggested by @huonw.

alexcrichton · 2016-04-20T16:19:22Z

@bors: r+ ce99a5e

bors · 2016-04-20T16:57:58Z

⌛ Testing commit ce99a5e with merge 92e3fb3...

Compute `target_feature` from LLVM This is a work-in-progress fix for #31662. The logic that computes the target features from the command line has been replaced with queries to the `TargetMachine`.

bors · 2016-04-20T19:28:32Z

@alexcrichton

librustc: remove outdated workaround Fixed upstream: llvm-mirror/llvm@ca07e256f62f @alexcrichton following up from #31709 (comment) cc @ranma42

The function should accept feature strings that old LLVM might not support. Simplify the code using the same approach used by LLVMRustPrintTargetFeatures. Dummify the function for non 4.0 LLVM and update the tests accordingly.

rust-highfive assigned nikomatsakis Feb 16, 2016

alexcrichton reviewed Feb 16, 2016
View reviewed changes

rust-highfive assigned alexcrichton and unassigned nikomatsakis Feb 16, 2016

alexcrichton reviewed Feb 17, 2016
View reviewed changes

ranma42 force-pushed the target_feature-from-llvm branch from c6d1db8 to 232c894 Compare March 2, 2016 20:34

alexcrichton reviewed Mar 4, 2016
View reviewed changes

ranma42 mentioned this pull request Mar 5, 2016

-C target-cpu=help won’t print anything unless compilation reaches trans #22380

Closed

huonw reviewed Apr 20, 2016
View reviewed changes

ranma42 added 2 commits April 20, 2016 09:08

Make the feature whitelists constants

deaa2fe

This simplifies the code a bit and makes the types nicer, too.

Check that the feature strings are well-formed

ce99a5e

Assert that the feature strings are NUL terminated, so that they will be well-formed as C strings. This is a safety check to ease the maintaninace and update of the feature lists.

ranma42 mentioned this pull request Apr 20, 2016

Add integer atomic types #33048

Merged

bors merged commit ce99a5e into rust-lang:master Apr 20, 2016

bors mentioned this pull request Apr 20, 2016

librustc_llvm: encapsulate "as" cast from/to a bool #32700

Closed

tamird mentioned this pull request Apr 20, 2016

librustc: remove outdated workaround #33117

Merged

huonw mentioned this pull request Apr 21, 2016

target_feature cfg only obeys -C target-feature="+feature" #31662

Closed

ranma42 deleted the target_feature-from-llvm branch April 21, 2016 07:36

bors added a commit that referenced this pull request Apr 23, 2016

Auto merge of #33117 - tamird:remove-outdated-workaround, r=sanxiyn

ebff638

librustc: remove outdated workaround Fixed upstream: llvm-mirror/llvm@ca07e256f62f @alexcrichton following up from #31709 (comment) cc @ranma42

ranma42 mentioned this pull request Jun 21, 2016

Support detecting all llc's x86 target features #34397

Closed

ranma42 mentioned this pull request Jul 16, 2016

Add accessors for MCSubtargetInfo CPU and Feature tables rust-lang/llvm#45

Merged

alexcrichton mentioned this pull request Dec 16, 2016

Tracking issue for cfg_target_feature #29717

Closed

1 task

ajtulloch mentioned this pull request Mar 31, 2019

[WIP] [CodeGen] Fix fp16 -> fp32 codegen on X86-64 apache/tvm#2925

Merged

Compute target_feature from LLVM #31709

Compute target_feature from LLVM #31709

Conversation

ranma42 commented Feb 16, 2016

rust-highfive commented Feb 16, 2016

ranma42 commented Feb 16, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikomatsakis commented Feb 16, 2016

ranma42 commented Feb 17, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ranma42 commented Mar 2, 2016

alexcrichton commented Mar 3, 2016

ranma42 commented Mar 3, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ranma42 commented Mar 4, 2016

Choose a reason for hiding this comment

alexcrichton commented Mar 4, 2016

alexcrichton commented Mar 4, 2016

alexcrichton commented Mar 4, 2016

ranma42 commented Mar 8, 2016

alexcrichton commented Mar 8, 2016

alexcrichton commented Mar 8, 2016

bors commented Mar 27, 2016

bors commented Apr 20, 2016

Choose a reason for hiding this comment

alexcrichton commented Apr 20, 2016

Choose a reason for hiding this comment

huonw commented Apr 20, 2016

bors commented Apr 20, 2016

bors commented Apr 20, 2016

ranma42 commented Apr 20, 2016 • edited Loading

alexcrichton commented Apr 20, 2016

bors commented Apr 20, 2016

bors commented Apr 20, 2016

Compute `target_feature` from LLVM #31709

Compute `target_feature` from LLVM #31709

ranma42 commented Apr 20, 2016 •

edited

Loading