Move to MSBuild 14.0

[dsyme]

As reported elsewhere MSBuild 12.0 in Visual Studio Update 3 is now giving strange errors for some F# project files/ This is causing our AppVeyor builds to fail.

My suggestion is that we move FSharp.Compiler.Service.ProjectCrackerTool.exe to use MSBuild 14.0

@rneatherway Do you expect any problems with doing this? It looks from your previous analysis that MSBuild 14.0 is reliably installed on all Mono systems. But can we expect it to be installed on Windows systems, if Visual Studio is not installed? Will this lead to renewed requests to package the MSBuild 14.0 binaries with FSharp.Compiler.Service project cracker, e.g. in a "tools" directory?

[rneatherway]

@rneatherway Do you expect any problems with doing this? It looks from your previous analysis that MSBuild 14.0 is reliably installed on all Mono systems. But can we expect it to be installed on Windows systems, if Visual Studio is not installed? Will this lead to renewed requests to package the MSBuild 14.0 binaries with FSharp.Compiler.Service project cracker, e.g. in a "tools" directory?

Yes we can rely on its presence on Mono. On Windows, we have to require that people install it, either with Visual Studio or with the standalone Build Tools installer. The ProjectCrackerTool needs this to be a proper installation (not just the assemblies) so that the various variables in the project files point to the right places and find the MSBuild target/properties files.

[dsyme]

@rnetherway Thanks. I guess we have to do this for now, though it's unfortunate. Or could we use private dlls and set the install location variables ourselves?

[rneatherway]

It's lines like this:

 <FSharpTargetsPath>$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v$(VisualStudioVersion)\FSharp\Microsoft.FSharp.Targets</FSharpTargetsPath>

The variable is described on msdn as "MSBuildExtensionsPath32: The path of the MSBuild subfolder under the \Program Files or \Program Files (x86) folder." This variable is actually listed as "Well-Known", which means it can be overridden, whereas "Reserved" variables cannot be. This means that it might be possible to create a directory structure containing the relevant .targets and .props files necessary to build an F# project and distribute it with the project cracker.

[dsyme]

@rneatherway

This variable is actually listed as "Well-Known", which means it can be overridden, whereas "Reserved" variables cannot be.

Perhaps we could just override it to be back at \Program Files or \Program Files (x86) regardless of where the MSBuild binaries are.

This means that it might be possible to create a directory structure containing the relevant .targets and .props files necessary to build an F# project and distribute it with the project cracker.

Yes, though other imports in project files will look for that variable

[dsyme]

@rneatherway This is an awful dilemma, isn't it. Requiring the install of either Visual Studio or the standalone Build Tools installer on Windows is very awkward for editing and code formatting and other tools.

I don't see any way to have local copies of the MSBuild DLLs. However I'm wondering what happens if we adjust this code:

  let addMSBuildv14BackupResolution () =
    let onResolveEvent = new ResolveEventHandler(fun sender evArgs ->
      let requestedAssembly = AssemblyName(evArgs.Name)
      if requestedAssembly.Name.StartsWith("Microsoft.Build") &&
          not (requestedAssembly.Name.EndsWith(".resources")) then
        requestedAssembly.Version <- Version("14.0.0.0")
        Assembly.Load (requestedAssembly)
      else
        null)
    AppDomain.CurrentDomain.add_AssemblyResolve(onResolveEvent)

to use

        // If the version of MSBuild that we're using wasn't present on the machine, then 
        // just revert back to 12.0.0.0 since that's normally installed as part of the .NET 
        // Framework.
        requestedAssembly.Version <- Version("12.0.0.0")

as the fallback. Would that correctly load MSBuild on a system with .NET Framework installed, but not the Build Tools? That would likely work for the majority of cases (v12.0 was only failing for a few cases). But does this require a special app.config setting on the part of the user? Right now the App.config for the ProjectCrackerTool has been adjusted to this:

        <assemblyIdentity name="Microsoft.Build.Framework" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-14.0.0.0" newVersion="14.0.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.Build.Engine" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-14.0.0.0" newVersion="14.0.0.0" />
      </dependentAssembly>

Certainly the above change won't be any worse than what we have now (which is a static bind to 14.0.0.0). But, ignoring non-binary-compat functionality, is it valid to return a 12.0.0.0 assembly when the 14.0.0.0 was requested?

One problem is that I can't easily test this stuff since I don't have a machine without VS 2015 installed.

Thanks
don

[dsyme]

@rneatherway Please see here: https://github.com/fsharp/FSharp.Compiler.Service/pull/603/files#diff-95410d5b67f692571ebb551f0d2a1e4bL435 (the rest of that commit is just cleanup since we don't do TargetFrameworkVersion=4.0 any more)

[rneatherway]

Perhaps we could just override it to be back at \Program Files or \Program Files (x86) regardless of where the MSBuild binaries are.

I was thinking more of overriding it to be wherever the bundled MSBuild assemblies and target/props files are.

Anyway, I'm afraid that I do not think your approach (14 by default, 12 as a backup) will work in the MSBuild v12 case. I definitely considered it back when we first created ProjectCrackerTool. If I remember correctly, the problem is that the binding redirects you have in place will even affect the backup resolution, so if 14 fails and you try 12, that just gets redirected back to 14 again.

I hope that one day there might be a viable alternative to MSBuild for F#.

[Krzysztof-Cieslak]

I hope that one day there might be a viable alternative to MSBuild for F#.

I know some folks who would be interested at working on some alternative ;)

[cloudRoutine]

@rneatherway @dsyme I'm confused as to what we need the project cracker for. Maybe I'm missing something about it's functionality but it seems like the most important thing it does is resolving the assembly references and handling the binding redirects. Is it absolutely necessary for Msbuild to be used to accomplish this? Couldn't we do something similar to how they build the reference graph in monodevelop where we insert the redirects during the node processing?

Despite our explicit instructions to install them, these MsBuild toolchain dependencies have been a constant source of confusion and barrier to getting started for many Ionide users.

The project cracker has been a constant source of headaches for people trying to contribute to Ionide and users, so if there's anything we can do to replace it, that would be a huge benefit.

(I'm one of those people interested in working on an alternative)

[rneatherway]

@cloudRoutine The project cracker is used to resolve assembly references, that is absolutely right. It does this using MSBuild, because this is the most accurate way to reproduce what the application will actually be linked against, given that it will almost certainly be built with MSBuild. I implemented it this way because that is how the Visual Studio F# Tools do it. Unfortunately we don't have the luxury of being installed alongside a particular version of Visual Studio, and I agree it has caused a lot of problems.

The code you linked to is an modified version of code written by a colleague of mine (and hosted here) to order fully resolved assembly references so that they can be used in an F# script file with #r (as that requires them to be loaded in order). I'm not sure that it will immediately help here.

If the MonoDevelop plugin has a way of resolving these assembly references, then let's definitely try to use that, perhaps @nosami can comment on the current approach. Even something as simple as finding the GAC and having a look in there might be better than the current project cracker, although it would fail for complex MSBuild project files using <Otherwise> and things like that.

[cloudRoutine]

@rneatherway another possibility is to package msbuild with the project cracker. We could add https://github.com/Microsoft/msbuild as a git dependency with paket and then as a part of the build process the msbuild dlls and .exe (netcore for xplat) could be produced and then bundled with the project cracker.

or maybe Unofficial: Getting .Net Core MSBuild binaries without building the code could be used in the interim until they release a proper package?

And I suppose the last option is to crib the msbuild parsing from monodevelop and use that to get the project settings and configuration settings we need.

[rneatherway]

@rneatherway another possibility is to package msbuild with the project cracker. We could add https://github.com/Microsoft/msbuild as a git dependency with paket and then as a part of the build process the msbuild dlls and .exe (netcore for xplat) could be produced and then bundled with the project cracker.

We would also have to include all the .target and .props files (including those specifically for F#), see my comment above. Probably FSharp.Build.dll as well. I think this would eventually work, after all Mono essentially does this, but the question is whether we would end up needing to fork the targets files and/or bundle too large a number of files.

[dsyme]

Closing. Project Cracker is on MSBuild 14. We're aiming to remove or optionalize the dependency on MSBuid 12 for reference resolution