build.bat should locate msbuild.exe rather than vcvarsall.bat

[zooba]

BPO	30687
Nosy	@pfmoore, @tjguk, @jkloth, @ned-deily, @zware, @zooba, @Paxxi
PRs	bpo-30687: Fixes build scripts to find msbuild.exe and stop relying on vcvarsall.bat #2252 [3.6] bpo-30687: Fixes build scripts to find msbuild.exe and stop relying on vcvarsall.bat  #2280 [3.5] bpo-30687: Fixes build scripts to find msbuild.exe and stop relying on vcvarsall.bat  #2281

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2017-06-19.18:34:01.361>
created_at = <Date 2017-06-16.22:03:49.614>
labels = ['build', '3.7', 'OS-windows']
title = 'build.bat should locate msbuild.exe rather than vcvarsall.bat'
updated_at = <Date 2017-07-08.04:51:40.544>
user = 'https://github.com/zooba'

bugs.python.org fields:

activity = <Date 2017-07-08.04:51:40.544>
actor = 'ned.deily'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2017-06-19.18:34:01.361>
closer = 'steve.dower'
components = ['Build', 'Windows']
creation = <Date 2017-06-16.22:03:49.614>
creator = 'steve.dower'
dependencies = []
files = []
hgrepos = []
issue_num = 30687
keywords = []
message_count = 12.0
messages = ['296218', '296362', '296363', '296368', '296369', '296780', '296783', '296784', '296791', '296826', '296838', '297935']
nosy_count = 7.0
nosy_names = ['paul.moore', 'tim.golden', 'jkloth', 'ned.deily', 'zach.ware', 'steve.dower', 'Paxxi']
pr_nums = ['2252', '2280', '2281']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue30687'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

[zooba]

Due to how installation of Visual Studio 2017 has changed, it's very possible to end up with a valid install of the build tools without vcvarsall.bat being valid.

We only require MSBuild.exe to build, so we should find that tool and invoke it, rather than trying to configure the entire environment.

[zooba]

New changeset 40a23e8 by Steve Dower in branch 'master':
bpo-30687: Fixes build scripts to find msbuild.exe and stop relying on vcvarsall.bat (bpo-2252)
40a23e8

[zooba]

New changeset 06d6e3d by Steve Dower in branch '3.6':
bpo-30687: Fixes build scripts to find msbuild.exe and stop relying on vcvarsall.bat (bpo-2252) (bpo-2280)
06d6e3d

[zooba]

New changeset 2c899cc by Steve Dower in branch '3.5':
bpo-30687: Fixes build scripts to find msbuild.exe and stop relying on vcvarsall.bat (bpo-2252) (bpo-2281)
2c899cc

[zooba]

Ned - this has been merged into 3.6 branch and the buildbots look good.

Feel free to move the NEWS item around and/or cherrypick into 3.6.2 at your leisure.

[Paxxi]

This change causes build failures when VS2017 is installed without the C++ tooling as it finds MSBuild belonging to VS2017 but it can't build using it.

Microsoft recommends using this tool https://github.com/microsoft/vswhere to find and set up the paths, it also works with VS2015.

[zooba]

You're going to get build failures without the C++ tooling anyway, and msbuild should find previous installs that are compatible.

What is your system setup that causes this failure when you expect success?

[Paxxi]

Currently I have VS2017 installed without C++ tooling for .NET development. The C++ tooling breaks other projects I'm working on.

I have VS2015 Community update 3 installed with C++ tooling and the latest compatible Windows SDK.

Before this patch everything builds as expected using PCBuild\build.bat -e. After this patch it fails instantly with

Using "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\\MSBuild\15.0\Bin\msbuild.exe" (found in the Visual Studio 2017 registry)

C:\code\cpython>"C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\\MSBuild\15.0\Bin\msbuild.exe" "C:\code\cpython\PCbuild\pcbuild.proj" /t:Build /m /nologo /v:m /p:Configuration=Release /p:Platform=Win32 /p:IncludeExternals=true /p:IncludeSSL=true /p:IncludeTkinter=true /p:UseTestMarker= /p:GIT="C:\Program Files\Git\cmd\git.exe"
C:\code\cpython\PCbuild\pythoncore.vcxproj(42,3): error MSB4019: The imported project "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\VC\VCTargets\Microsoft.Cpp.Default.props" was not found. Confirm that the p ath in the <Import> declaration is correct, and that the file exists on disk.

Commenting out the following lines from find_msbuild.bat and everything is back to working

@Rem VS 2017 sets exactly one install as the "main" install, so we may find MSBuild in there.
@reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\SxS\VS7" /v 15.0 /reg:32 >nul 2>nul
@if NOT ERRORLEVEL 1 @for /F "tokens=1,2*" %%i in ('reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\SxS\VS7" /v 15.0 /reg:32') DO @(
@if "%%i"=="15.0" @if exist "%%k\MSBuild\15.0\Bin\msbuild.exe" @(set MSBUILD="%%k\MSBuild\15.0\Bin\msbuild.exe")
)
@if exist %MSBUILD% (set _Py_MSBuild_Source=Visual Studio 2017 registry) & goto :found

[zooba]

I suspect that should be filed as a bug against VS 2017 - I'll do that when I get to work on Monday.

Our workaround should be to check for that targets file and keep searching if it's not found.

Your best workaround is to start the build from the VS 2015 developer command prompt, which will have the right MSBuild on PATH and will bypass the search.

[Paxxi]

I don't believe that this is a bug in Visual Studio as MSBuild is used for .NET projects as well it should be available even without the C++ tooling installed.

Checking for the targets file seems like a workable solution.

[zooba]

The bug is MSBuild 15.0 should be able to locate earlier versions of VC, even if you haven't installed it for VS 2017.

[ned-deily]

New changeset 00134f6 by Ned Deily (Steve Dower) in branch '3.6':
bpo-30687: Fixes build scripts to find msbuild.exe and stop relying on vcvarsall.bat (bpo-2252) (bpo-2280)
00134f6