Projects don't work with dotnet CLI tooling

[mmiller678]

Hi,
Great job on putting these together. I have using SDK projects for our .NET 4.8 projects but had some issues with our MVC projects. With the help from the work you have done plus the long running thread on .NET project system site (dotnet/project-system#2670) I was able to get most of it going. One issue I did run into is while trying to implement MvcBuildViews support to run the dotnet tooling for use on our build server. While your projects do work in the Visual Studio they fail to work from dotnet cmd line tools.

dotnet build for instance results in:

C:\Users\xxx.nuget\packages\msbuild.sdk.systemweb\4.0.33\Sdk\Sdk.targets(16,3): error MSB4019: The imported project "C:\Program Files\dotnet\sdk\5.0.201\Microsoft\VisualStudio\v16.0\WebApplications\Microsoft.WebApplication.targets" was not found. Confirm that the expression in the Import declaration "C:\Program Files\dotnet\sdk\5.0.201\Microsoft\VisualStudio\v16.0\WebApplications\Microsoft.WebApplication.targets" is correct, and that the file exists on disk. [c:\Users\xxx\Downloads\MSBuild.SDK.SystemWeb-main\MSBuild.SDK.SystemWeb-main\samples\ExampleEmptyWebApplication\ExampleEmptyWebApplication.csproj]

Build FAILED.

If I remove this property group:

<PropertyGroup>
    <VisualStudioVersion Condition="'$(VisualStudioVersion)' == ''">15.0</VisualStudioVersion>
    <VSToolsPath Condition="'$(VSToolsPath)' == ''">$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v$(VisualStudioVersion)</VSToolsPath>
  </PropertyGroup>
  <Import Project="Sdk.targets" Sdk="Microsoft.NET.Sdk" />
  <Import Project="$(VSToolsPath)\WebApplications\Microsoft.WebApplication.targets" />
</Project>

The problem is resolved. Any ideas on how to get this working from the cmd line?

[CZEMacLeod]

@mmiller678 Because this is targeting net4x and requires the WebApplication targets file from Visual Studio (available by installing the ASP.Net 4.x workload) I don't think you will be able to build using dotnet build. I think you will need to use a build server with VS2019 Build Tools installed and build using MSBuild instead.

The only alternative would be to include the contents of C:\Program Files (x86)\Microsoft Visual Studio\2019\xxxx\MSBuild\Microsoft\VisualStudio\v16.0\WebApplications in the NuGet package next to the SDK files. These are copyright Microsoft and part of Visual Studio so I cannot include them. If it were allowed, it would mean you wouldn't need to install the ASP.NET 4.x workload though.

I'm not entirely sure what the issue with MvcBuildViews is though. That uses the AspNetCompiler MSBuild task that runs the dotnet 4.x Aspnet_compiler.exe executable installed in %windir%\Microsoft.NET\Framework64\v4.0.30319
This very much depends on the .Net 4.x Runtime being installed so I would not be using the dotnet cli in this case.

I did look at adding the MvcBuildViews target to the SDK and setting a default for the MvcBuildViews condition to true for Release configuration and false otherwise. However, if you are not using MVC then perhaps you wouldn't want this as default behaviour. It can be easily overridden of course so I may push and update with this.

The ASPNET Compiler is only needed if you want/need to precompile your markup files. In general I would recommend using this as a 'head' project and moving MVC views into a class library and using the excellent RazorGenerator.Mvc package along with the RazorGenerator.MsBuild package to compile the views into your class libraries avoiding the need for the MvcBuildViews task altogether.

Perhaps I need to clarify the documentation about using and building inside and outside VS.
One area I never got to work inside VS was publish (although it does appear to work correctly from the MSBuild command line).

[CZEMacLeod]

@mmiller678
I did a quick test to see if passing in the location of VSToolsPath would work, but it fails at a further step.

mkdir test
cd test
dotnet new systemwebfull
dotnet build -p:"VSToolsPath=C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Microsoft\VisualStudio\v16.0"

It fails with C:\Users\____\.nuget\packages\microsoft.codedom.providers.dotnetcompilerplatform\2.0.1\build\net46\Microsoft.CodeDom.Providers.DotNetCompilerPlatform.props(31,5): error MSB4801: The task factory "CodeTaskFactory" is not supported on the .NET Core version of MSBuild.
As I said previously - I think you would need to build with MSBuild.

mkdir test2
dotnet new systemwebfull
msbuild
msbuild /t:Publish /p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingeFile=true /p:PackageLocation="MyPublishDir"

succeeds from a VS2019 developer command prompt.

[FuncLun]

Forgive me for trying to re-open a closed by design/wontfix issue, but I believe an improvement can be made without any drawbacks.

While there are several reasons to continue to build with msbuild, I would like to use actions like dotnet test my.sln and dotnet pack my.sln without the web projects failing. A simple solution is to include the files in the repo and copy them into the $(VSToolsPath) path.

The dotnet tools do not work because the sdks do not include $(VSToolsPath)\WebApplications\Microsoft.WebApplication.targets or $(VSToolsPath)\WebApplications\Microsoft.WebApplication.Build.Tasks.dll. A simple solution is to copy those files from the msbuild folder into the dotnet sdk (i.e. C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VisualStudio\v16.0\WebApplications to C:\Agent\_work\_tool\dotnet\sdk\5.0.203\Microsoft\VisualStudio\v16.0\WebApplications\Microsoft.WebApplication.targets). This can be tricky when using Microsoft hosted build agents, expecially when a new version of the sdk is released. Ideally, you would configure the csproj to point the location targets files. But, updating $(VSToolsPath) would require all tools to be included in the repo.

However, updating the import to this:

  <PropertyGroup>
    <VSToolsPath_WebApplications Condition="'$(VSToolsPath_WebApplications)' == ''">$(VSToolsPath)\WebApplications</VSToolsPath_WebApplications>
  </PropertyGroup>
  <Import Project="$(VSToolsPath_WebApplications)\Microsoft.WebApplication.targets" />

Would allow the directory to be overridden in the csproj:

    <VSToolsPath_WebApplications>..\{SomeFolder}\WebApplications</VSToolsPath_WebApplications>

FYI: dotnet pack will fail if the solution contains a project with the old file format. Adding <Target Name="pack" /> to the old csproj file allows pack to not fail - it just won't pack that project.

[CZEMacLeod]

@FuncLun Unfortunately, even adding the .targets files etc. to the SDK won't work, as some of those targets use features that only work in MSBuild, and not under dotnet as they require .NETFramework tasks which are not compatible with .NETCore which is used when you try and run it under dotnet.
You can see the issue in my previous comment.
If you use Microsoft.CodeDom.Providers.DotNetCompilerPlatform version 3.6.0 instead of 2.0.1 as with the previous template, I believe it now has a custom DLL with the task in that package, instead of using the CodeTaskFactory. So it looks like this issue may be avoidable.

If you look at e.g. the VSSDK for Visual Studio Extensions nuget package you can see they basically do what you are talking about. That package includes the targets, and DLLs that would normally be installed with visual studio, so that you can compile on a build server which doesn't have the VSSDK workload installed.

<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup Label="VSSDK_NuGet_Configuration">
    <ThisPackageDirectory>$(MSBuildThisFileDirectory)..\</ThisPackageDirectory>
    <VSToolsPath>$(ThisPackageDirectory)\tools</VSToolsPath>
    <VsSDKInstall>$(VSToolsPath)\VSSDK</VsSDKInstall>
    <VsSDKIncludes>$(VsSDKInstall)\inc</VsSDKIncludes>
    <VsSDKToolsPath>$(VsSDKInstall)\bin</VsSDKToolsPath>
  </PropertyGroup>
</Project>

Unfortunately, there isn't an equivalent Microsoft.WebApplications.BuildTools type of package - and I believe the targets files and associated custom tasks and DLLs are copyrighted as part of VS and not public domain, and are not part of the redistributables allowed by the licence.
Check C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Microsoft\VisualStudio\v16.0\WebApplications and also the contents of C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Microsoft\VisualStudio\v16.0\Web which is imported from Microsoft.WebApplication.targets
This means we cannot wrap those into a nuget package, and/or the SDK.
I did find MSBuild.Microsoft.VisualStudio.Web_WebApplication.Targets but it is very out of date, is an unofficial release, and, I suspect, breaches the Microsoft copyright.

One solution to mixing dotnet pack with MSBuild.SDK.SystemWeb projects would be to use solution configurations, say ones called release.pack and debug.pack and set the project configurations as appropriate and specifically say not to build the web projects.
I do this in one of my solutions even using MSBuild to not pack my web projects and some libraries that haven't been converted to SDK format yet (my solution has 173 projects in it, including 3 old style web application projects and 5 MSBuild.SDK.SystemWeb projects)

dotnet pack -c Debug.Pack

It is perfectly possible to use msbuild /t:pack instead of dotnet pack anyway.

Alternatively, I suggest you use multiple solution files, one of which only contains projects that work with dotnet if you cannot work with msbuild.

Also, you can simply add the nuget package NuGet.Build.Tasks.Pack if you want to enable the pack target for a web application project.

    <PackageReference Include="NuGet.Build.Tasks.Pack" Version="5.10.0" />

You can then prevent it actually packing the project by adding the property

    <IsPackable>false</IsPackable>

dotnet pack -p:"VSToolsPath=C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Microsoft\VisualStudio\v16.0"

Microsoft (R) Build Engine version 17.0.0-preview-21302-02+018bed83d for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  All projects are up-to-date for restore.
  You are using a preview version of .NET. See: https://aka.ms/dotnet-core-preview

  Restore operation started...

  5 libraries restored in 0.22 seconds

  TestWeb -> R:\testweb\TestWeb\bin\TestWeb.dll
  Successfully created package 'R:\testweb\TestWeb\bin\TestWeb.1.0.0.nupkg'.

[CZEMacLeod]

@FuncLun A quick follow up - I found another unofficial nuget package which I think includes the VS2015 versions of the web.application.targets and associated files.
It seems this will work for building with dotnet, although I imagine it will have other issues when running under MSBuild and/or VS when you actually want to build and publish your web application. The condition should make it not import the old package when running under msbuild, so that still works properly. YMMV

  <PropertyGroup>
    <IsPackable>false</IsPackable>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="NuGet.Build.Tasks.Pack" Version="5.10.0" />
    <PackageReference Include="MSBuild.Microsoft.VisualStudio.Web.targets" Version="14.0.0.3" Condition="'$(MSBuildRuntimeType)' == 'Core'"/>
  </ItemGroup>

[bachratyg]

If your goal is building the app on a CI server without Visual Studio you can install the same workload from Build Tools: https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2019

[julealgon]

@CZEMacLeod does it make sense to mark this issue with the known limitation label?

[CZEMacLeod]

@julealgon It is tagged as by design, as using dotnet to build a msbuild based project that requires net4x for building is not part of the design of this project. It is intended to allow you to build and publish using the VS Build tools or VS, or edit using VS, where you can take advantage of the newer project style.
The issue title is also slightly misleading in that you can create new projects from the template with dotnet tooling, just not build and compile.
I'm not sure adding known limitation here is valid given that, and it might lead people to the wrong impression of the project as a whole.

[julealgon]

Fair enough, thanks for elaborating @CZEMacLeod 😉

I'll probably pursue this on my own later because being able to unify our build pipelines using the dotnet CLI would be a substantial gain.

[RussKie]

Unfortunately, there isn't an equivalent Microsoft.WebApplications.BuildTools type of package - and I believe the targets files and associated custom tasks and DLLs are copyrighted as part of VS and not public domain, and are not part of the redistributables allowed by the licence.
Check C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Microsoft\VisualStudio\v16.0\WebApplications and also the contents of C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Microsoft\VisualStudio\v16.0\Web which is imported from Microsoft.WebApplication.targets
This means we cannot wrap those into a nuget package, and/or the SDK.

I believe this still can be resolved by finding VS installation and importing the web targets from there.
Something like this:

<Project>

  <ItemGroup>
    <PackageReference Include="vswhere" GeneratePathProperty="true" />
  </ItemGroup>

  <Target Name="FindWebApplicationTargets">
    <PropertyGroup>
      <VSWherePath>$([MSBuild]::NormalizePath('$(Pkgvswhere)', 'tools'))</VSWherePath>
    </PropertyGroup>

    <!-- Work out VS installation path, so we can find MSBuild.exe -->
    <Exec
        Command="vswhere.exe -latest -prerelease -property installationPath -requires Microsoft.VisualStudio.Component.Web"
        WorkingDirectory="$(VSWherePath)"
        EchoOff="true"
        ConsoleToMsBuild="true"
        StandardOutputImportance="Low">
        <Output TaskParameter="ConsoleOutput" PropertyName="_VSInstallPath" />
    </Exec>
    <Error Text="Unable to find Visual Studio installation." Condition="!Exists($(_VSInstallPath))" />

    <PropertyGroup>
      <_MSBuildVsPath>$([MSBuild]::NormalizePath('$(_VSInstallPath)', 'MSBuild', 'Microsoft', 'VisualStudio'))</_MSBuildVsPath>
    </PropertyGroup>

    <ItemGroup>
      <_WebTargets Include="$(_MSBuildVsPath)\**\WebApplications\Microsoft.WebApplication.targets" />
    </ItemGroup>
    <Error Text="Unable to locate $(_MSBuildVsPath)\&lt;version&gt;\WebApplications\Microsoft.WebApplication.targets. Ensure Microsoft.VisualStudio.Component.Web workload is installed."
           Condition="@(_WebTargets->Count()) != 1"/>

    <PropertyGroup>
      <_WebApplicationTargetsPath>@(_WebTargets)</_WebApplicationTargetsPath>
    </PropertyGroup>
    <Message Text="Microsoft.WebApplication.targets: $(_WebApplicationTargetsPath)" Importance="High" />
  </Target>

I assume "Microsoft.VisualStudio.Component.Web" workload is responsible for Microsoft.WebApplication.targets file but I'm not 100% sure.

[CZEMacLeod]

@RussKie Thanks for that idea - it might be able to resolve the path to the dll, but I think there is still the problem that those DLLs refer to System.Web and require running under NetFramework, not NetCore. You might be able to do something with dotnet msbuild instead of dotnet build - and if you have already built the output and only want to dotnet publish again there might be scope for this use case.

I'd be willing to add the extra code in (or accept a PR) if it actually enables any usable scenarios.
I would suggest that setting the environment variable VSToolsPath outside of MSBuild in your ci script and use a command line version of VSWhere would be a better fit, as it wouldn't make the SDK 'heavier' for all other users by requiring the VSWhere package which wouldn't be used by the majority of people.

[RussKie]

The scenario is the ability to build the solution with dotnet from cli - i.e., use dotnet restore, dotnet build, dotnet test, etc. having converted projects in the solution to the SDK-style.

I'm currently "lucky" to work on an old solution which consists of a mix of .NET Framework, .NET Standard and .NET projects. Unless I convert the lot to the SDK-style I can't use dotnet to build the solution from cli.
I have converted all .NET Framework projects to the SDK-style and I was able to restore/build/test the whole solution from cli, however, with that conversion I lost the "F5" experience for the web project. With you magical solution I have the "F5" experience back but now I'm unable to build from cli...

I would suggest that setting the environment variable VSToolsPath outside of MSBuild in your ci script

Setting en envvar may work for a single dev, however, in a distributed environment it won't scale - i.e., every contributor to the repo will have to know to set the variable in order to build from cli.

...as it wouldn't make the SDK 'heavier' for all other users by requiring the VSWhere package which wouldn't be used by the majority of people.

IMO this is a weak argument. Look at how many NuGet packages a typical restore pulls in, one more or one less won't make a difference, however it delivers a significant devex benefit.
Also, it's quite debatable what majority of developers prefer or use :)

[CZEMacLeod]

@RussKie Fair enough on the comment about assuming what developers use. I was more meaning in the context of someone working with a legacy project including aspnet4, which is based on an msbuild / windows / visual studio ecosystem, trying to use tools designed for a different runtime (netcore/dotnet) is IMHO not the majority use case. The point of this SDK is to allow the use of an SDK style csproj file and its benefits - globbing, easy diffing, edit in place, etc. Being able to compile using dotnet was never really a goal.

As for the weight - true enough, but pulling in something that is only used in one scenario, which isn't really supported anyway, seems to be a bad decision - and adding the logic to detect building outside of Visual Studio (either in the IDE or via the command line) affects the performance for everyone. I will admit that performance isn't really a goal of this SDK either, but making the overall system slower when it isn't needed seems to be a mistake.

Now on the particular issue with VSToolsPath - I'm pretty sure that is must be set and evaluated before the build starts, as it is used in an import statement - thus the variable cannot be set in a target as you've suggested here.
The use of a third party nuget package allows you to use the packagepath property to calculate the path at the top level - but that uses copyright dlls and is out of date.

I would suggest that if you are happy using cli tools, that you create an appropriate build script file, running vswhere and passing the result either directly to the dotnet command or setting the env variable.

As mentioned further back, using tools designed for netcore to build a net4 project is not a goal of this project.

[CZEMacLeod]

@RussKie If you come up with a clean solution with minimal impact to the existing build system, please open a PR and we can go from there.