Spec Proposal: MSBuild Extension support for .NET Core #1756

Open
AArnott opened this Issue Feb 27, 2017 · 11 comments

Comments

Projects
None yet
6 participants
@AArnott
Member

AArnott commented Feb 27, 2017

NuGet and MSBuild should work more closely together for MSBuild extensions.

Creation of an MSBuild extension

I should be able to dotnet new buildextension to get a new project that:

  1. Includes an MSBuild Task that compiles to net45 and netcoreapp1.x
  2. Includes an MSBuild .props and .targets file in source code that invoke the Task from a target, choosing
  3. Packs to a NuGet package, that
    1. expresses package dependencies for package dependencies referenced in the csproj that compiled the MSBuild Task. No need to embed dependencies in the MSBuild extension package itself.
    2. Sets DevelopmentDependency=true automatically in the nuspec file.

Consumption of this MSBuild extension:

  1. Happens by adding a BuildExtensionReference item to the receiving project. This isn't PackageReference because its dependency graph does not blend into the project's dependency graph. But it isn't DotNetCliToolReference because those items do not contribute MSBuild .props and .targets files to the project. This BuildExtensionReference item may propagate across P2P references if desired (so that a single 'root' project in a solution may add the extension to all other projects that reference it). Or perhaps we can unify this new item with DotNetCliToolReference by calling it ProjectExtension (similar to MSBuild's special element type, but this one would appear inside an ItemGroup element.
  2. Upon NuGet Restore of the project, the MSBuild extension is downloaded to the machine package cache along with all its dependencies in their own respective packages.
  3. The project automatically imports the .props and .targets from the package.
  4. The project can build and invoke the Task both on MSBuild full and MSBuild Core, and dependencies the Task has can resolve from the package cache.
  5. The MSBuild Task controls its own dependencies independently of other Tasks that may run in that project because each one runs in its own AssemblyLoadContext in MSBuild Core and AppDomain in MSBuild Desktop.

Optional dotnet CLI tool as well:

An MSBuild extension may also want to provide convenient dotnet CLI invocation as well. Currently for an MSBuild extension to both modify the build but also make tools accessible by dotnet CLI the user must add both a PackageReference and a DotNetCliToolReference item to their project. This is cumbersome, especially when such an extension applies to all projects in a solution. So dotnet CLI should allow one package to offer both an MSBuild extension and a dotnet CLI tool.

An example is Nerdbank.GitVersioning which both modifies the build with special version semantics, as well as offers a couple of CLI tools to translate a commit to a version and vice versa.

As discussed with @nguerrera and @tmat on another issue.

@rainersigwald

This comment has been minimized.

Show comment
Hide comment
@rainersigwald

rainersigwald Feb 27, 2017

Contributor

Do you have an objection to splitting this into two issues: one for having a template, and one for the second half?

Contributor

rainersigwald commented Feb 27, 2017

Do you have an objection to splitting this into two issues: one for having a template, and one for the second half?

@AArnott

This comment has been minimized.

Show comment
Hide comment
@AArnott

AArnott Feb 27, 2017

Member

Thanks for looking, @rainersigwald.

There are a bunch of "issues" to resolve to deliver on this. I see this issue more of a spec for folks to agree to the vision of, then to link it to all the various issues (including across repos) that will deliver on it. Having a template won't be very compelling on its own if the rest of the story isn't solid.

But if you'd really prefer two top level specs, that may reference each other, I can break it up.

Member

AArnott commented Feb 27, 2017

Thanks for looking, @rainersigwald.

There are a bunch of "issues" to resolve to deliver on this. I see this issue more of a spec for folks to agree to the vision of, then to link it to all the various issues (including across repos) that will deliver on it. Having a template won't be very compelling on its own if the rest of the story isn't solid.

But if you'd really prefer two top level specs, that may reference each other, I can break it up.

@dasMulli

This comment has been minimized.

Show comment
Hide comment
@dasMulli

dasMulli Feb 27, 2017

Contributor

What's probably needed is a proper acquisition story for SDKs as has been discussed in #1493 and #1436, as well as tooling(/templates/defaults) to build & publish SDKs.

I've come across this recently when trying to patch together a few build utilities. While you can tell NuGet to pack the resulting dlls in a folder other than lib through <BuildOutputTargetFolder>build</BuildOutputTargetFolder> and mess with PrivateAssets="All" on the PackageReference item, you soon run out of luck trying to reference & include 3rd party nugets (unless you would emit items with a custom pack path directly out of the nuget packages folder which is also accessible through an msbulid property).

Another issue: At some point, you'll want to explicitly depend on a specific SDK being present.
E.g., if I consume SDK properties, I want to make sure that Microsoft.NET.Sdk is present in the build. If I do fancy F# code generation, I'd want to depend on the F# SDK.
Or maybe integrate with web-specific targets from the web SDK..

What currently works fine is to build packages that make consuming projects include props and targets files via convention (e.g. PackageName.targets):

<None Update="build\**\*" Pack="true" PackagePath="\build" />

Combined with <IncludeBuildOutput>false</…> This also helps replace a lot of nuspec use cases since dotnet pack doesn't directly support packing nuspec files anymore.

Contributor

dasMulli commented Feb 27, 2017

What's probably needed is a proper acquisition story for SDKs as has been discussed in #1493 and #1436, as well as tooling(/templates/defaults) to build & publish SDKs.

I've come across this recently when trying to patch together a few build utilities. While you can tell NuGet to pack the resulting dlls in a folder other than lib through <BuildOutputTargetFolder>build</BuildOutputTargetFolder> and mess with PrivateAssets="All" on the PackageReference item, you soon run out of luck trying to reference & include 3rd party nugets (unless you would emit items with a custom pack path directly out of the nuget packages folder which is also accessible through an msbulid property).

Another issue: At some point, you'll want to explicitly depend on a specific SDK being present.
E.g., if I consume SDK properties, I want to make sure that Microsoft.NET.Sdk is present in the build. If I do fancy F# code generation, I'd want to depend on the F# SDK.
Or maybe integrate with web-specific targets from the web SDK..

What currently works fine is to build packages that make consuming projects include props and targets files via convention (e.g. PackageName.targets):

<None Update="build\**\*" Pack="true" PackagePath="\build" />

Combined with <IncludeBuildOutput>false</…> This also helps replace a lot of nuspec use cases since dotnet pack doesn't directly support packing nuspec files anymore.

@natemcmaster

This comment has been minimized.

Show comment
Hide comment
@natemcmaster

natemcmaster Feb 28, 2017

Member

I'm interested in this idea too. ASP.NET's build system attempts to extend MSBuild with NuGet packages containing tasks/targets. PackageReference is close to a good solution, but imports happen too late for things like loggers, .NET Framework reference assemblies, and custom task assemblies.

I'm hoping the SDK acquisition experience helps.

And +1 for creating a csproj template for MSBuild tasks projects. Currently this requires knowing how to manipulate NuGet's internal pack targets to get assemblies and files into the right place.

Member

natemcmaster commented Feb 28, 2017

I'm interested in this idea too. ASP.NET's build system attempts to extend MSBuild with NuGet packages containing tasks/targets. PackageReference is close to a good solution, but imports happen too late for things like loggers, .NET Framework reference assemblies, and custom task assemblies.

I'm hoping the SDK acquisition experience helps.

And +1 for creating a csproj template for MSBuild tasks projects. Currently this requires knowing how to manipulate NuGet's internal pack targets to get assemblies and files into the right place.

@dasMulli

This comment has been minimized.

Show comment
Hide comment
@dasMulli

dasMulli Mar 8, 2017

Contributor

Another approach could be to (ab)use NuGet's package type string and introduce a new kind of reference item. Along with <PackageReference> and <DotNetCliToolReference> there could be sth like a <BuildToolReference> that - like the cli tool reference - does not affect the consuming project's dependency tree and does not flow into parent projects (so you can avoid the PrivateAssets="All" now typically used for build-only dependencies) but would just add props and targets files imports.

This would be a pure NuGet feature but would more closely align how distributing and consuming works with nuget packages than "SDK packages" (for which you probably want only one version across the solution and maybe not ship it with NuGet but register a custom SDK resolver etc.).

Contributor

dasMulli commented Mar 8, 2017

Another approach could be to (ab)use NuGet's package type string and introduce a new kind of reference item. Along with <PackageReference> and <DotNetCliToolReference> there could be sth like a <BuildToolReference> that - like the cli tool reference - does not affect the consuming project's dependency tree and does not flow into parent projects (so you can avoid the PrivateAssets="All" now typically used for build-only dependencies) but would just add props and targets files imports.

This would be a pure NuGet feature but would more closely align how distributing and consuming works with nuget packages than "SDK packages" (for which you probably want only one version across the solution and maybe not ship it with NuGet but register a custom SDK resolver etc.).

@stazz

This comment has been minimized.

Show comment
Hide comment
@stazz

stazz Jun 3, 2017

Hi,

I agree with @AArnott that there should be better support for NuGet package -based MSBuild tasks.
Overall this issue has some good points, and I hope that we will get support for BuildExtensionReference soon.

However, while the current state of MSBuild Extension support via NuGet packages is not optimal, I have created a custom MSBuild Task Factory, which will execute other MSBuild Tasks, which are NuGet package-based.
I've tested this against MSBuild 15.1 in .NET 4.6, and MSBuild 15.3-Preview in .NET Core (since task factories are not supported in MSBuild 15.1 for .NET Core), and it seems that things work out nicely in both scenarios.

There are no special requirements for developing tasks which are useable by this task factory, other than their target framework is suitable (you can just target .netstandard 1.3, and the task will work in both .NET Desktop and .NET Core MSBuild).
The task may reference other third-party NuGet packages freely - the task factory will take care of loading dependent assemblies on-the-fly.

More information available at UtilPack.NuGet.MSBuild, I hope this will help other people who develop complex NuGet package-based MSBuild tasks!

stazz commented Jun 3, 2017

Hi,

I agree with @AArnott that there should be better support for NuGet package -based MSBuild tasks.
Overall this issue has some good points, and I hope that we will get support for BuildExtensionReference soon.

However, while the current state of MSBuild Extension support via NuGet packages is not optimal, I have created a custom MSBuild Task Factory, which will execute other MSBuild Tasks, which are NuGet package-based.
I've tested this against MSBuild 15.1 in .NET 4.6, and MSBuild 15.3-Preview in .NET Core (since task factories are not supported in MSBuild 15.1 for .NET Core), and it seems that things work out nicely in both scenarios.

There are no special requirements for developing tasks which are useable by this task factory, other than their target framework is suitable (you can just target .netstandard 1.3, and the task will work in both .NET Desktop and .NET Core MSBuild).
The task may reference other third-party NuGet packages freely - the task factory will take care of loading dependent assemblies on-the-fly.

More information available at UtilPack.NuGet.MSBuild, I hope this will help other people who develop complex NuGet package-based MSBuild tasks!

@AArnott

This comment has been minimized.

Show comment
Hide comment
@AArnott

AArnott Jun 4, 2017

Member

That sounds very interesting, @stazz. Thanks for sharing. I hope to check it out when I get some time.

Member

AArnott commented Jun 4, 2017

That sounds very interesting, @stazz. Thanks for sharing. I hope to check it out when I get some time.

@stalek71

This comment has been minimized.

Show comment
Hide comment
@stalek71

stalek71 Oct 18, 2017

Very good idea @stazz :)
Thanks for sharing it with us.
I found something like this also:
https://blog.nuget.org/20170316/NuGet-now-fully-integrated-into-MSBuild.html

stalek71 commented Oct 18, 2017

Very good idea @stazz :)
Thanks for sharing it with us.
I found something like this also:
https://blog.nuget.org/20170316/NuGet-now-fully-integrated-into-MSBuild.html

@AArnott

This comment has been minimized.

Show comment
Hide comment
@AArnott

AArnott Oct 22, 2017

Member

@stazz said:

you can just target .netstandard 1.3, and the task will work in both .NET Desktop and .NET Core MSBuild

I'm afraid this is likely not true. Unless you've taken special care to load portable assemblies on desktop Framework. The problem is an MSBuild task that compiles against .netstandard1.3 will require facade assemblies at runtime when on .NETFramework -- assemblies that MSBuild (full) doesn't have, leading to at least some MSBuild Tasks failing at runtime.

That's why the MSBuild team's position is you have to dual compile tasks, targeting each of .NET Core and .NET Framework for it to work reliably on each platform.

Member

AArnott commented Oct 22, 2017

@stazz said:

you can just target .netstandard 1.3, and the task will work in both .NET Desktop and .NET Core MSBuild

I'm afraid this is likely not true. Unless you've taken special care to load portable assemblies on desktop Framework. The problem is an MSBuild task that compiles against .netstandard1.3 will require facade assemblies at runtime when on .NETFramework -- assemblies that MSBuild (full) doesn't have, leading to at least some MSBuild Tasks failing at runtime.

That's why the MSBuild team's position is you have to dual compile tasks, targeting each of .NET Core and .NET Framework for it to work reliably on each platform.

@stazz

This comment has been minimized.

Show comment
Hide comment
@stazz

stazz Oct 22, 2017

@stalek71 said:

Thanks for sharing it with us.

Glad to hear! :)

@AArnott said:

Unless you've taken special care to load portable assemblies on desktop Framework.

That's exactly what I had to do. :)

The problem is an MSBuild task that compiles against .netstandard1.3 will require facade assemblies at runtime when on .NETFramework

You're right - that's what happens on .NET Desktop. I encountered this long time ago, and had to do appropriate modifications to my code. But those modifications do work: for example, the CBAM.SQL.MSBuild task is compiled only against .NET Standard 1.3, but it runs successfully on both .NET Desktop and .NET Core. It is not a trivial task either, since it uses disk IO to read database configuration and SQL files, and network IO to communicate with the database. So the dual-compiling is not really mandatory.

stazz commented Oct 22, 2017

@stalek71 said:

Thanks for sharing it with us.

Glad to hear! :)

@AArnott said:

Unless you've taken special care to load portable assemblies on desktop Framework.

That's exactly what I had to do. :)

The problem is an MSBuild task that compiles against .netstandard1.3 will require facade assemblies at runtime when on .NETFramework

You're right - that's what happens on .NET Desktop. I encountered this long time ago, and had to do appropriate modifications to my code. But those modifications do work: for example, the CBAM.SQL.MSBuild task is compiled only against .NET Standard 1.3, but it runs successfully on both .NET Desktop and .NET Core. It is not a trivial task either, since it uses disk IO to read database configuration and SQL files, and network IO to communicate with the database. So the dual-compiling is not really mandatory.

@stazz

This comment has been minimized.

Show comment
Hide comment
@stazz

stazz Oct 22, 2017

Oh, one more thing. One special problem specific only to this issue is that when executing .NET Core MSBuild, the NuGet assemblies are part of trusted assembly set, and thus if e.g. task factory uses NuGet stuff, it won't see the NuGet libraries it was compiled against, but the NuGet libraries loaded by .NET Core SDK. Furthermore, even the minor version updates in NuGet libraries introduce binary-incompatible changes (the ILogger changes in 4.2.0 -> 4.3.0 update, and then the introduction of LocalNuspecCache and usage in RestoreCommandProviders.Create method in 4.3.0 -> 4.4.0 update).

This is why, for UtilPack.NuGet.MSBuild version 2.0.0, I had to introduce facade task factory for .NET Core assembly, which examines the version of NuGet library loaded by SDK, and uses appropriate actual task factory assembly (currently two: for NuGet version 4.3.0, and for NuGet version 4.4.0).

That is something I need to create a issue about once I get time. Not sure if anything can be done about that tho.

stazz commented Oct 22, 2017

Oh, one more thing. One special problem specific only to this issue is that when executing .NET Core MSBuild, the NuGet assemblies are part of trusted assembly set, and thus if e.g. task factory uses NuGet stuff, it won't see the NuGet libraries it was compiled against, but the NuGet libraries loaded by .NET Core SDK. Furthermore, even the minor version updates in NuGet libraries introduce binary-incompatible changes (the ILogger changes in 4.2.0 -> 4.3.0 update, and then the introduction of LocalNuspecCache and usage in RestoreCommandProviders.Create method in 4.3.0 -> 4.4.0 update).

This is why, for UtilPack.NuGet.MSBuild version 2.0.0, I had to introduce facade task factory for .NET Core assembly, which examines the version of NuGet library loaded by SDK, and uses appropriate actual task factory assembly (currently two: for NuGet version 4.3.0, and for NuGet version 4.4.0).

That is something I need to create a issue about once I get time. Not sure if anything can be done about that tho.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment