Searching of MSBuild tools: vswhere + VS as an initial tool by default

[3F]

How about to provide this (or similar) tool by default with VS as front-controller (into one known unified place) for simple requests about simple informations like searching of available msbuild tools and other informations about all installed VS instances ?

It's important for any small tools like this, that contains all logic via single batch file etc.
I mean just look the gnt.bat (full logic for work with nuget packages: creating new, installing, etc. via msbuild) and tell me, how we should now ? +vswhere.exe ?

For any searching of msbuild tools it looked like this msbuild-helper before (to build in any places like CI build-servers, any custom user's env etc. - easy and works well)

this:

for /f "usebackq tokens=1* delims=: " %%i in (`vswhere -latest -requires Microsoft.Component.MSBuild`) do (
  if /i "%%i"=="installationPath" set InstallDir=%%j
)
...

is much more useful when your vswhere (or something else) will be provided initially as a simple console tool and located in already known place (to aggregate any requests as front-controller as I said above).

This is possible by MS ? or how it should be now for something like above ?

Update 1

Final solutions to find msbuild.exe from VS2017:

Via batch scripts ( no powershell, no dotnet cli )

For online machines, now appeared this - https://github.com/3F/hMSBuild

A single compiled batch file ~19 Kb that can be embedded inside any scripts or other batch files:

• https://github.com/3F/hMSBuild/releases/latest

Usage is same as it would be same for msbuild.

Usage: hMSBuild [args to hMSBuild] [args to msbuild.exe or GetNuTool core]
------

Arguments:
----------
 -no-vs        - Disable searching from Visual Studio.
 -no-netfx     - Disable searching from .NET Framework.
 -no-vswhere   - Do not search via vswhere.

 -vsw-version {arg}  - Specific version of vswhere. Where {arg}:
     * 2.6.7 ...
     * Keywords:
       `latest` - To get latest remote version;
       `local`  - To use only local versions;
                  (.bat;.exe /or from +15.2.26418.1 VS-build)
...

Solution by Microsoft

here:

After further discussion internally, we're going to add vswhere.exe to a known location (such as %CommonProgramFiles%) in an upcoming servicing release, which will address the original issue as proposed by @3F at the top of the thread. This will give you the flexibility to use ...

Update 2

here by @heaths:

vswhere is now available as part of the install starting today with Visual Studio 15.2 preview 2:

Update 3

Just removed direct links to hMSBuild 1.x ( 2017, which are still actively referenced ). Please use latest versions:

🧱 Compiled text-based embeddable pure batch-scripts for searching of available MSBuild tools.

• https://github.com/3F/hMSBuild ( .bat ~19 KB, no powershell, no dotnet cli )

[heaths]

We are considering this, but even if we did in a subsequent update, VS2017 RTW already shipped. vswhere was meant to be redistributed as needed, so I the meantime and for VS2017 RTW support you could consider adding the file to the repo.

[3F]

VS2017 RTW already shipped

Yes, I noticed -_- ... and I don't know how to easily solve this problem instead of adding vswhere for a lot of public repos.

Moreover, as you can see this can be a simple small script-tool for end-users. So I don't know what to do at all for this cases (well, I know but...), because now it requires ... omg.

As I see, gnt.bat tool above should correctly work via tool from .NET Framework, thus nuget-packages are still not problem and we can distribute vswhere automatically (like for other my X tool for additional complex build-scripting that wrapped by single bat file and distributed via nuget server), however: the nuget package will not useful for offline machines :(, so the easy way, yes, to distribute 'as is' together with repo. That's of course amazing for simply finding path to the msbuild tools <_<

Need to think -_-

We are considering this

Good ! I hope to see something from next patch-releases at least.

Thanks,

[KindDragon]

or how it should be now for something like above ?

You can try to install it from Chocolatey, It will be automatically added to PATH

[3F]

It will be automatically added to PATH

the problem not for path to vswhere because we can still use gnt like:

:: to get the vswhere & vsSBE packages
gnt /p:ngpackages="vswhere;vsSBE.CI.MSBuild"

:: we're ready for vswhere
%vswhere% -latest -requires Microsoft.Component.MSBuild
...

:: build Conari project via powerful vsSBE scripts & msbuild tools from found path by vswhere
%msbuild% -notamd64 "Conari.sln" /l:"CI.MSBuild.dll" ... || goto err
...
etc.

but thanks, it's first time when I hear about chocolatey :)

[grafikrobot]

The Boost C++ Libraries project lobbied for this to be included before the release went out. But those requests failed. Most likely the VS release manager didn't consider this a high-priority issue. But now we have a situation where many different projects like Boost, Cmake, etc will end up including a patchwork of incompatible detection methods in their repos.

For Boost including the executable of this utility might be an option.. If it where not for the license. As, if you can believe, the Boost license is less restrictive than the MIT license in use for vswhere. My suggestion would be to relicense vswhere to the "Public Domain" or the "Boost License", or some other license that doesn't require license terms for binary inclusion.

[heaths]

@grafikrobot looking at the boostorg/boost repo I see a number of bootstrap scripts that pull down files (granted, Python scripts). Perhaps the MIT license is more restrictive but you merely need to pull it. You don't even have to include it in your repo. That's why the vswhere.exe executable itself (not ZIP'd or anything) is available here on GitHub. How is pulling that different from pulling other build-required scripts?

[grafikrobot]

@heaths I'm not sure where you see the bootstrap pulling anything. As it's designed to operate entirely offline. For example it's possible to download a Boost release unplug your network cable (or equivalent) and then do a complete build. Which is actually required for some platforms like some mainframes and other specialized hardware that are not internet connected.

[heaths]

@grafikrobot what is this? https://github.com/boostorg/boost/blob/master/bootstrap.bat

Even if we did install it with our installer now, what about those customers with RTW?

[grafikrobot]

@heaths This https://github.com/boostorg/boost/blob/master/bootstrap.bat is the script that checks if you've already built the build system. If not, it tries to build it. If that fails it tells the user some alternatives.

As for RTW people.. Good question. Boost could choose to not support it. Or only support it with a big disclaimer and some user manual configuration (i.e. we'd fail to bootstrap and the message would suggest that if they have RTW they do the manual config).

[grafikrobot]

@heaths PS.. Also for RTW customers Microsoft could choose to install vswhere as part of the next OS patch set so that everyone has it installed. Wouldn't help the customers that use OS version that are past end-of-life. But VS2017 probably doesn't run on those.

[OlafvdSpek]

@grafikrobot Why an OS update and not just a VS update?
And why not have users use "Developer Command Prompt for VS 2017" until MS fixes this?

@heaths That's what updates are for.

[heaths]

@OlafvdSpek users are not forced to take updates for VS. They might have to in some scenarios, but they have never been required. Many organizations want to do rigorous testing before ever taking updates. Even if we did, some users may not be connected or take an update, which means they're on RTW.

[OlafvdSpek]

@heaths True, so one might decide not to support RTW..

[grafikrobot]

Copying my "solution" from another issue here for the benefit of those in this topic..

Never mind.. After trying to use the vswhere tool to determine the cl.exe location with some modicum of sanity I've given up. My current suggestion will be to avoid this tool and wait for real solution from Microsoft. In the mean time we will tell users that if they want VS2017, aka cl.exe 14.10 (or 14.1 depending on whom you ask) they will need to run from the VS command prompt or set the VS150COMNTOOLS environment variable manually.

Thank you for my wasted time.. Goodbye.

[heaths]

@grafikrobot vswhere is the solution for multi-instance products. Environment variables won't solve the problem of which instances have all the tools you might expect. To deliver an even larger product faster, and to allow more customization based on customer feedback, the features where separated into more granular features (workloads and components). So to which directory would we even set a VS140COMNTOOLS environment variable?

Without getting into selecting the property chip (depends on what you want), a simple batch script like so would do what you need:

@echo off

for /f "usebackq tokens=*" %%i in (`vswhere -requires Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Core -property installationPath`) do (
    if exist "%%i" set dir=%%i
)

if exist "%dir%\VC\Tools\MSVC\14.10.25017\bin\HostX86\x86\cl.exe" (
    call "%dir%\VC\Tools\MSVC\14.10.25017\bin\HostX86\x86\cl.exe" %*
)

[Lastique]

@heaths I think the problem is that one has to hardcode the path to cl.exe in the script, including version, which apparently will be different depending on the toolset version, installed updates and whatnot. At this point there's little benefit from vswhere since one could just as well hardcode the full path or ask the user to start bootstrap from VS command line prompt.

PS: Whoever made this mess with toolset location needs to have his head checked. COM objects? Binary tool on GitHub? Downloading or packaging the binary tool in opensource projects, seriously? And all this just to find out a path to the compiler? Have a look at how it's done on other systems, like Linux or MSYS - there's a much simpler and convenient way.

[timsneath]

Thanks for your feedback and your thoughts. There are several big changes in this release, including the ability to install multiple instances of Visual Studio side-by-side and a much smaller default installation; while that offers many other benefits, one drawback is that it's not possible to hardcode a path with confidence that it contains the toolset you need. Having read through this thread, I'd like to summarize and identify three options:

1. @grafikrobot suggests that relicensing vswhere with 'the "Public Domain" or the "Boost License", or some other license that doesn't require license terms for binary inclusion would satisfy Boost needs.
2. We ship vswhere.exe with future servicing updates of Visual Studio. While as @heaths points out, some customers may take time to deploy the update, this provides a longer-term solution since this small binary can be placed in a known location on the system drive.
3. As also mentioned above, the VS command line is designed to set up all required environment variables. The advantage of this approach is that users can use the command line associated with the instance of their choice to build, and even alternate between them (e.g. switching from a release to a pre-release compiler).

Is there a preference from among these options? We'd love to move forward with one of these approaches.

Best wishes, Tim Sneath | Visual Studio Team

[grafikrobot]

@timsneath thanks you for the comment.. We are aware of the changes and stated rationales. You mention that it's not possible to hard code a path. Yet as pointed out by @Lastique that is precisely what we would need to do to use vswhere. As for your stated options:

1. That is not an option any longer. The Boost community is predominantly against bundling any binary in the source distribution, regardless of license. As it is counter to the trust model of open source software.
2. Although that is the best option. It is currently also not a viable one. The vswhere program is of limited value given the minimal information it provides. We would still end up hard coding version numbers. Which is no different than what we already do. Hence it would just add to the complexity we already deal with.
3. The VS command line would solve only part of the problem while adding extra documentation and user setup. By part of problem I refer to your second sentence which is not a use case for Boost except in the singular initial bootstrap building time. Our build system isn't limited by building for any one configuration or tool at any one time. It supports building multiple toolsets at once. For example we could build a project, like the many libraries of Boost, for MinGW, MSVC 6 and onward, Intel, gcc, emscripten, como, cray, and others in any variation of 64, 32, debug, release, profile at once and in parallel. In other words, it only solves 1% of the problem.

But given the option at this point, and likely the future, I would prefer option 3.

[OlafvdSpek]

@timsneath 4. Allow the user to pick one VS environment and have it be available by default for any standard cmd prompt or powershell. This would allow cl main.cpp to work as is (assuming the syntax for cl is correct).
This would be in addition to other options to automatically set up an environment from a bat or ps script.

[Lastique]

@OlafvdSpek Exactly. Also provide cl-14.1, cl-14.0, etc. at the same location or add command line switches to cl having the effect of selecting the toolset version and target architecture.

[timsneath]

OK - thanks for the feedback and the engagement on this issue, even knowing that we don't have the ideal solution for your needs because of other trade-offs.

After further discussion internally, we're going to add vswhere.exe to a known location (such as %CommonProgramFiles%) in an upcoming servicing release, which will address the original issue as proposed by @3F at the top of the thread. This will give you the flexibility to use options 2 and 3, as appropriate for your needs.

Since vswhere is not included in the RTW release of Visual Studio 2017, you might consider an approach something like this in your build configuration script for extra robustness:

1. Look for vswhere.exe in its known location. If present, use this to find the build tools toolset.
2. If absent, look in the default install location for Visual Studio 2017 (the 80% case):
   C:\Program Files (x86)\Microsoft Visual Studio\2017\<EDITION_NAME>
3. If still not found, prompt the user for path or tell them to run build from Developer Command Prompt for VS 2017 (vsdevcmd.bat), which will set up the appropriate environment variables.

The script prototype above from @heaths gives a good example of how to implement step 1 above.

Thanks again, Tim Sneath | Visual Studio Team

[OlafvdSpek]

@timsneath What's the feedback on #41 (comment) ?

The script prototype above from @heaths gives a good example of how to implement step 1 above.

Wouldn't that break as soon as the version != '14.10.25017'?

[res2k]

If you're using vcvarsall.bat you should be fine - it can be found in %dir%\VC\Auxiliary\Build (no version number, yay!).

I think build tools for VC <= 14.0 tend use the registry to look for vcvarsall.bat anyway, so doing the same, but via vswhere.exe, is probably not a terribly impractical approach.

[heaths]

As @timsneath mentioned we will be including vswhere in a well-known location in a subsequent release, so I'm resolving this issue.

[OlafvdSpek]

@heaths I'd still like to hear feedback on #41 (comment)

[heaths]

Please see my previous comment. A single variable would not capture the granularity of options or which toolset version (now that we ship multiple) and target architecture for native compilation. The suggestion by @res2k is a robust solution that will handle that.

[OlafvdSpek]

@heaths I know it won't, but that shouldn't stop you from allowing the user to pick one as default..

And please see #41 (comment)

[3F]

I updated information about solutions for this issue, and you can also try this if you have similar problem for today.

After further discussion internally, we're going to add vswhere.exe to a known location (such as %CommonProgramFiles%) in an upcoming servicing release

@timsneath, Good ! Thanks.

[timsneath]

@3F, this is awesome. Thanks for going the extra mile here - really appreciate it! When we update the documentation for this area, we'll be sure to link to your repo.

[3F]

@timsneath,

we'll be sure to link to your repo

All are welcome -_*

[heaths]

vswhere is now available as part of the install starting today with Visual Studio 15.2 preview 2: https://blogs.msdn.microsoft.com/heaths/2017/04/21/vswhere-is-now-installed-with-visual-studio-2017/

[OlafvdSpek]

What if the user uses a custom install dir?

[refack]

What if the user uses a custom install dir?

[unofficial answer] @OlafvdSpek AFAIK and from own my experience, the custom install dir (which I love and always use) is for the actual VS instance. The installer is always at %ProgramFiles(x86)%\Microsoft Visual Studio\Installer

[3F]

@OlafvdSpek,

As I understand, today there is no way to change path for installer:

vs_Enterprise -h

I see only --installPath key that defines path by default for new instance of VS, and not for installer. The installer still should be appeared in %ProgramFiles(x86)%\Microsoft Visual Studio\Installer (Win-x64) and %ProgramFiles%\Microsoft Visual Studio\Installer (Win-x32). if not, please add path via registry <_<

well I checked this. Now we can find vswhere.exe inside this directory. Yey !

@heaths, thanks for good news.

Later I will update logic of hMSBuild for new searhing of vswhere (locally and via NuGet).

[heaths]

The installer directory is fixed to "%ProgramFilers(x86)%\Microsoft Visual Studio\Installer". If an when we ever move it, we will of course need to maintain at least a symbolic link in that location for vswhere (to other files you find in that directory we make no such guarantee).

[sldr]

Consider:

@echo off
for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -legacy -version [11^,12^) -property installationPath`) do (
    if exist "%%i" (
        set VS11INSTALLDIR=%%i
        set VS11VCVARSALL=%%iVC\vcvarsall.bat
    )
)
for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -legacy -version [12^,13^) -property installationPath`) do (
    if exist "%%i" (
        set VS12INSTALLDIR=%%i
        set VS12VCVARSALL=%%iVC\vcvarsall.bat
    )
)
for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -legacy -version [13^,14^) -property installationPath`) do (
    if exist "%%i" (
        set VS13INSTALLDIR=%%i
        set VS13VCVARSALL=%%iVC\vcvarsall.bat
    )
)
for /f "usebackq tokens=*" %%i in ("%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -legacy -version [14^,15^) -property installationPath`) do (
    if exist "%%i" (
        set VS14INSTALLDIR=%%i
        set VS14VCVARSALL=%%iVC\vcvarsall.bat
    )
)
for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -legacy -version [15^,16^) -property installationPath`) do (
    if exist "%%i" (
        set VS15INSTALLDIR=%%i
        set VS15VCVARSALL=%%i\VC\Auxiliary\Build\vcvarsall.bat
    )
)
REM call "%VS11VCVARSALL%"
REM call "%VS12VCVARSALL%"
REM call "%VS13VCVARSALL%"
REM call "%VS14VCVARSALL%"
REM call "%VS15VCVARSALL%" x86
REM call "%VS15VCVARSALL%" x64

Just remove REM on one of the last calls for the version you want.

Hope this helps,
SLDR
(Stephen L. De Rudder)

I know it isn't perfect... ;-)