Overhaul MSVC linker and Windows SDK detection code #30233

retep998 · Dec 6, 2015

What I've done here is try to make the code match what vcvars does much more closely. It now chooses which SDK to find based on the version of MSVC that it found. It also bases the decision of whether to find all the things on whether VCINSTALLDIR has been set, which is more likely to have only been set by an invocation of vcvars, unlike previously where it would do some things only if LIB wasn't set even though there was a valid use case for libraries to add themselves to LIB without having invoked vcvars.

There are still some debug println!s so people can test the PR and make sure it works correctly on various setups.

It supports VS 2015, 2013, and 2012. People who want to use versions of VS older (or newer) than that will have to manually invoke the appropriate vcvars bat file to set the proper environment variables themselves.

Do not merge yet.

Fixes #30229

retep998 · Dec 6, 2015

r? @alexcrichton

rust-highfive · Dec 6, 2015

r? @Aatch

(rust_highfive has picked a reviewer for you, use r? to override)

retep998 · Dec 6, 2015

cc @vadimcn

retep998 · Dec 6, 2015

Testing by @DanielKeep confirms that this also works with the standalone Microsoft Visual C++ Build Tools 2015 Technical Preview, providing an option for users that want to use msvc but without installing the entire VS IDE.

alexcrichton · Dec 11, 2015

alexcrichton reviewed Dec 11, 2015

View changes

alexcrichton · Dec 11, 2015

src/librustc_trans/back/msvc/mod.rs

+            let mut dirs: Vec<_> = readdir.filter_map(|dir| dir.ok())
+                .map(|dir| dir.path()).collect();
+            dirs.sort();
+            dirs.reverse();


Instead of reverse here you should also be able to do dirs.into_iter().rev().filter(...) below

alexcrichton · Dec 11, 2015

alexcrichton reviewed Dec 11, 2015

View changes

alexcrichton · Dec 11, 2015

src/librustc_trans/back/msvc/mod.rs

-        });
-        if !is_vs_14 {
-            return None
+    // VC's x86 libraries are not in a subfolder


Out of curiosity, what's VC?

Visual C. Shorthand for MSVC.

alexcrichton · Dec 11, 2015

alexcrichton reviewed Dec 11, 2015

View changes

alexcrichton · Dec 11, 2015

src/librustc_trans/back/msvc/mod.rs

-    fn windows_sdk_v10_subdir(sess: &Session) -> Option<&'static str> {
-        match &sess.target.target.arch[..] {
+    // The native x86 toolchain is not in a subfolder unlike the rest
+    #[cfg(target_arch = "x86")]


Can these matches be done with if cfg! instead of #[cfg]?

alexcrichton · Dec 11, 2015

I'm a little worried about this logic for a few reasons:

This is basically an entirely new implementation, there's essentially no connection to what Clang does. This is generally a red flag, but I also never found a reason for why Clang does what it does, so it may not be that bad.
It looks like the way this is written will force us to update the compiler each time a new SDK is released. That may not be able to ever avoid, however.
It's very difficult to test this. I'd like to at least try to poke around a bit locally, but unfortunately my VM is offline.

I think that if this passes manual testing that we should land this, however, as fixing compatibility with the most recent VS is likely more important for now. It'd be great, however, to get a definitive answer on what this logic should be (or perhaps providing a better guarantee that it'll work), but I unfortunately don't know how to do that :(.

cc @vadimcn

vadimcn · Dec 11, 2015

vadimcn reviewed Dec 11, 2015

View changes

vadimcn · Dec 11, 2015

src/librustc_trans/back/msvc/mod.rs

-            "arm" => Some("arm"),
-            _ => return None,
+    // We choose the linker toolchain that has the same host as rustc.
+    // This allows cross compiling to actually work.


Is this because of env vars, or just to avoid having to detect bitness of the host OS?

Mainly just to avoid having to detect bitness of the host OS. I'll add a FIXME comment saying this should probably detect the host OS bitness.

vadimcn · Dec 11, 2015

I don't know which approach is better off the top of my head.
@retep998, could expand a bit on why you've decided to re-write this logic from scratch? Not that there's anything wrong with the new code (as far as I can tell).

It looks like the way this is written will force us to update the compiler each time a new SDK is released

The VC's registry structure will likely stay the same for the next few releases, so we could try to extrapolate versions into the future and hope that it works.
Also, maybe it'd be worth to introduce environment variable overrides (e.g. RUST_MSVC_LINKER, RUST_MSVC_LIBS) to allow users to fix things up without having to rebuild the compiler from source?

retep998 · Dec 12, 2015

Currently you can already override the entire system by just executing the proper vcvars bat file causing this code to back off and just trust everything is setup correctly and it'll work. Since we never know what sort of changes will occur in new versions of VS (such as VS 2015 adding the UCRT and the Windows 10 SDK both of which required new logic), I think it's better to not try to guess and just add support for it once someone can actually test it out.

As for testing this, I have VS 2015 2013 and 2012, and by commenting out some of the versions in that VS version array I could test how it handles all three versions and it works perfectly. It falls back to the 8.1 SDK for 2015 when there's no 10 SDK, it uses the 8.1 SDK for 2013 and it uses the 8 SDK for 2012. It still would be nice to get people testing it on other setups though as I've done all my testing on 64-bit Windows 10 only.

This code should work fine for new releases of the Win10 SDK as I've borrowed the exact logic from vcvars to find the newest Win10 SDK. Considering Microsoft said that Windows 10 is their final version of Windows, this means we may be in the clear with regards to that. New versions of VS however are still a possibility although the user can always just execute vcvars to setup the environment themselves.

The reason I rewrote this was because I felt like it wasn't robust enough. This implementation should very closely mirror what vcvars does and I'm more willing to trust what Microsoft does than what clang does, at least with regards to Microsoft's own software.

ghost · Dec 12, 2015

We can now also download the standalone / command-line Build Tools 2015, without even installing full VS2015, which installs compilers+SDKs+other utilities to compile C/C++ code on Windows.

Although the package itself is a Technical Preview, but I have successfully compiled with it on Windows 7 (fresh install) to Windows 10 and build Windows XP+ compatible libs. The CL compiler version is same as the one shipped with VS2015 Update 1.

It seems like this CLI-base is in preparation for Server 2016 and Nano Server feature called "containers". Also Windows team is working to provide "docker containers" (to bridge with their "native containers") GUI/CLI, where installing VS won't be a viable option: https://github.com/docker/docker/pulls/jhowardmsft.

ghost · Dec 12, 2015

what clang does,

I know at least detecting Windows 10 SDK (which is now at a new location the first time in a long time compared to its predecessors) hasn't done correctly by gyp, the build system favored by Chrome and node.js native modules projects. Initially gyp was developed because cmake was considered not meeting the requirement. Chrome guys are now moving away (or already) from their own gyp to another build system by Google called gn which seems to have fixed this SDK detection issue (but it is definitely in amateur hour to be fully trusted).

retep998 · Dec 13, 2015

@vadimcn
Really the reason I decided to overhaul this code was because it broke on the Windows 10 SDK code which was last touched by me so I felt somewhat responsible and also I was rather annoyed because the last time I touched it was to fix it being broken on the Windows 10 SDK stuff. So, I thought to myself, I will do this properly this time. Thus I threw out the old code, dug deep into vcvarsqueryregistry.bat, and made this PR.

alexcrichton · Dec 14, 2015

I agree with @vadimcn that we should definitely have an escape hatch for future proofing, but I also agree with @retep998 that we should have the necessary detection in place that if you're in a VC shell then rustc won't be doing anything fancy. I also agree that mirroring what Microsoft does is likely better than Clang, and for now we claim we need VS 2013+ anyway so as long as we're compatible with those I'm happy.

Thanks again @retep998! r=me with the debugging prints and such removed

alexcrichton · Dec 14, 2015

Or better yet actually, feel free to just convert them all to logging

retep998 · Dec 15, 2015

Switched it to use debug! logging and rebased while I was at it.

alexcrichton · Dec 15, 2015

@bors: r+ 915cb37

bors · Dec 15, 2015

⌛️ Testing commit 915cb37 with merge ecae5f2...

bors · Dec 15, 2015

💔 Test failed - auto-mac-64-nopt-t

alexcrichton · Dec 15, 2015

@bors: retry

On Mon, Dec 14, 2015 at 7:53 PM, bors notifications@github.com wrote:

[image: 💔] Test failed - auto-mac-64-nopt-t
http://buildbot.rust-lang.org/builders/auto-mac-64-nopt-t/builds/7445

—
Reply to this email directly or view it on GitHub
#30233 (comment).

bors · Dec 15, 2015

⌛️ Testing commit 915cb37 with merge 98ef4cc...

bors · Dec 15, 2015

💔 Test failed - auto-mac-64-nopt-t

retep998 · Dec 15, 2015

I'm getting somewhat annoyed at that mac buildbot and its spurious failures.

Manishearth · Dec 15, 2015

@bors: retry

bors · Dec 15, 2015

⌛️ Testing commit 915cb37 with merge 9e63cec...

bors · Dec 15, 2015

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-nopt-t, auto-mac-64-opt, auto-win-gnu-32-nopt-t, auto-win-gnu-32-opt, auto-win-gnu-64-nopt-t, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-opt

brson · Jan 7, 2016

Requested for backport in #30229

alexcrichton · Jan 12, 2016

Accepted for a backport

rust-highfive assigned Aatch Dec 6, 2015

rust-highfive assigned alexcrichton and unassigned Aatch Dec 6, 2015

retep998 force-pushed the retep998:where-in-the-world-is-windows-sdk branch from 00b9542 to bf1cb1b Dec 6, 2015

Overhaul MSVC linker and Windows SDK detection code
Fixes #30229 Signed-off-by: Peter Atashian <retep998@gmail.com>

Loading status checks…

915cb37

retep998 force-pushed the retep998:where-in-the-world-is-windows-sdk branch from 612e2d6 to 915cb37 Dec 14, 2015

bors merged commit 915cb37 into rust-lang:master Dec 15, 2015
2 checks passed

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

homu Test successful
Details

micbou referenced this pull request Jan 5, 2016
Closed
Use MSVC version of the Rust compiler on AppVeyor #293

retep998 referenced this pull request Jan 7, 2016
Closed
Windows 10 SDK detection broken with new SDK #30229

brson added I-nominated beta-nominated T-tools labels Jan 7, 2016

alexcrichton removed the I-nominated label Jan 11, 2016

brson referenced this pull request Jan 12, 2016
Merged
More beta ports #30854

brson added the beta-accepted label Jan 12, 2016

alexcrichton removed the beta-nominated label Jan 12, 2016

alexcrichton referenced this pull request Jan 20, 2016
Closed
MSVC: Regression cross compiling from x86_64 to x86 #31063

liigo referenced this pull request Jun 22, 2016
Closed
msvc: rustc produces invalid .exe (Windows XP) #34407

rust-lang/rust

Join GitHub today

Overhaul MSVC linker and Windows SDK detection code #30233

Conversation

retep998 commented Dec 6, 2015

rust-highfive assigned Aatch Dec 6, 2015

This comment has been minimized.

retep998 commented Dec 6, 2015

rust-highfive assigned alexcrichton and unassigned Aatch Dec 6, 2015

This comment has been minimized.

rust-highfive commented Dec 6, 2015

This comment has been minimized.

retep998 commented Dec 6, 2015

retep998 force-pushed the retep998:where-in-the-world-is-windows-sdk branch from 00b9542 to bf1cb1b Dec 6, 2015

This comment has been minimized.

retep998 commented Dec 6, 2015

alexcrichton reviewed Dec 11, 2015 View changes

This comment has been minimized.

alexcrichton reviewed Dec 11, 2015 View changes

This comment has been minimized.

This comment has been minimized.

alexcrichton reviewed Dec 11, 2015 View changes

This comment has been minimized.

This comment has been minimized.

alexcrichton commented Dec 11, 2015

vadimcn reviewed Dec 11, 2015 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

vadimcn commented Dec 11, 2015

This comment has been minimized.

retep998 commented Dec 12, 2015

This comment has been minimized.

ghost commented Dec 12, 2015

This comment has been minimized.

ghost commented Dec 12, 2015

This comment has been minimized.

retep998 commented Dec 13, 2015

This comment has been minimized.

alexcrichton commented Dec 14, 2015

This comment has been minimized.

alexcrichton commented Dec 14, 2015

retep998 force-pushed the retep998:where-in-the-world-is-windows-sdk branch from 612e2d6 to 915cb37 Dec 14, 2015

This comment has been minimized.

retep998 commented Dec 15, 2015

This comment has been minimized.

alexcrichton commented Dec 15, 2015

This comment has been minimized.

bors commented Dec 15, 2015

bors added a commit that referenced this pull request Dec 15, 2015

This comment has been minimized.

bors commented Dec 15, 2015

This comment has been minimized.

alexcrichton commented Dec 15, 2015

This comment has been minimized.

bors commented Dec 15, 2015

bors added a commit that referenced this pull request Dec 15, 2015

This comment has been minimized.

bors commented Dec 15, 2015

This comment has been minimized.

retep998 commented Dec 15, 2015

This comment has been minimized.

Manishearth commented Dec 15, 2015

This comment has been minimized.

bors commented Dec 15, 2015

bors added a commit that referenced this pull request Dec 15, 2015

This comment has been minimized.

bors commented Dec 15, 2015

Hide details View details bors merged commit 915cb37 into rust-lang:master Dec 15, 2015 2 checks passed

2 checks passed

micbou referenced this pull request Jan 5, 2016

retep998 referenced this pull request Jan 7, 2016

brson added I-nominated beta-nominated T-tools labels Jan 7, 2016

This comment has been minimized.

brson commented Jan 7, 2016

alexcrichton removed the I-nominated label Jan 11, 2016

brson referenced this pull request Jan 12, 2016

brson added the beta-accepted label Jan 12, 2016

alexcrichton removed the beta-nominated label Jan 12, 2016

This comment has been minimized.

retep998 force-pushed the retep998:where-in-the-world-is-windows-sdk branch from `00b9542` to `bf1cb1b` Dec 6, 2015

alexcrichton reviewed Dec 11, 2015

View changes

alexcrichton reviewed Dec 11, 2015

View changes

alexcrichton reviewed Dec 11, 2015

View changes

vadimcn reviewed Dec 11, 2015

View changes

retep998 force-pushed the retep998:where-in-the-world-is-windows-sdk branch from `612e2d6` to `915cb37` Dec 14, 2015

bors merged commit `915cb37` into rust-lang:master Dec 15, 2015
2 checks passed