Proposal: Address Project Reunion goals by addressing Visual Studio restrictions

[Scottj1s]

Proposal: Address Project Reunion goals by addressing Visual Studio restrictions

Summary

Many of the barriers to code sharing and interop that Project Reunion seeks to remove are either active restrictions, or passive limitations, of Visual Studio. These have been surfaced with the release of new WinRT projections like C++/WinRT and C#/WinRT.

Rationale

Project Reunion is likely to highlight these Visual Studio restrictions even more, and so should address their elimination. These restrictions fall into build time and run time categories, detailed below. This proposal is something of a grab bag of known issues, and seeks to prioritize their resolution so that the end to end developer experience is more streamlined.

Build Time

C++ PackageReferences

C++ projects (*.vcxproj files) do not support Nuget PackageReferences, but only packages.config references, which have a number of limitations:

• They must target a specific package version, even with allowedVersions
• They don't participate with the project system, so can't import other artifacts or use replaceable parameters
• They don't support 'msbuild /t:restore', so a separate 'nuget restore' is necessary in build pipelines, etc

There is a longstanding open request to add this support:
Use PackageReference in vcxproj
And a related longstanding open PR to implement it:
add C++ PackageReference support

Desktop <--> UWP Project References

Project references cannot be added in Visual Studio between Desktop and UWP (Universal Windows) projects. Attempts to do so result in an error: A reference cannot be added because the two projects target different platforms.. This can be worked around with a project file edit, or introduction of a directory.build.* file, to manually add the project reference. In other words, this is strictly a tooling restriction, which prevents interop unless developers are aware of the workarounds.

Runtime

Unified CRT

Having enabled Desktop<-->UWP project references, the developer must then address the runtime conflict of C Runtime libraries used by both modules. A Desktop module imports C:\Windows\System32\vcruntime140.dll and friends, while a UWP module imports C:\Program Files\WindowsApps\Microsoft.VCLibs.140*\vcruntime140_app.dll and friends. This impedance mismatch can be addressed by using the VCRT Forwarders package. But this technique is not very discoverable (it's not obvious that CRT dynalink errors can be addressed with a special adapter Nuget package).

A better solution is the inclusion of both CRT flavors in a unified VCLibs.UWPDesktop framework package, usable by both packaged UWP and centennial modules. This would obsolete the need for the VCRT Forwarders package, which is an explicit user action that adds build and deploy complexity, and increases install image size. Visual Studio could ensure that all binaries are "chameleon linked" with the CRT, using the Desktop import entries. Selection between Desktop CRT and unified VCLibs framework package could then be deferred until runtime, enabling broad reusability of modules. Visual Studio could also automatically select the unified VCLibs framework package for debugging and packaging behavior of UWP projects.

Native WinRT Activation

Having addressed the CRT linkage issues, there are often WinRT activation issues. Historically, activation of user-defined WinRT components was restricted to packaged apps. With Windows 1903 (19H1), support was added for unpackaged app activation, based on a fusion manifest: Enhancing Non-packaged Desktop Apps using Windows Runtime Components. This requires additional user action that is not discoverable and is tedious - the manifest requires an entry for every activatable class. Visual Studio could include project templates and/or build-time customization to automatically generate and populate this manifest.

There is an effort to address the Windows 1903 requirement of the Reg-Free Activation feature above, using a Detours-based library: Undocked RegFree Winrt Activation. Ideally, this library could be delivered as a Nuget package, along with build-time customizations mentioned above, and included in Visual Studio project templates.

Managed WinRT Activation

The above discussion addresses activation of native components, from either native or managed code. There is some support in Visual Studio for the opposite - activating managed components from native code. But this is subject to a number of limitations.

For example, a packaged UWP app can include a project reference to a managed UWP component and Visual Studio will generate the necessary uwpshims.exe appx manifest entry for hosting the managed component. However, there is a bug in this support that requires targeting Windows SDK 15063 (RS2) or older: Enable C++ Hybrid apps to target .Net native 1.7 and 2.2 without workarounds.

For activating a managed component from a native Desktop (non-packaged) app, there is currently no support in Visual Studio. An approach similar to Reg-Free Activation above (using a fusion manifest, in the absence of any other app manifest), could be used to implement shim support similar to that for packaged apps above.

[stevenbrix]

So a) thank you for filing this issue Scott! This is a perfect summary of many of the tooling pain points our customers are facing. The good news is that I believe we are starting to address many of these. More details on each below.

C++ Package References

@davkean said in a meeting that dotnet/project-system#2491 would be in VS 16.8 and bring with it PackageReference support for C++. Is this still the plan of record? The lack of updates on the issue worries me a bit that this may not be coming.

Desktop <--> UWP Project References

This is being addressed right now with WinUI3. We've introduced a DesktopCompatible property that allows the ProjectReference to work. If you're using VS16.7, you can set this property in your .vcxproj.

Native WinRT Activation

We have a task to make this work for WinUI3 apps: https://microsoft.visualstudio.com/OS/_workitems/edit/25174345/

We'll be a shipping a custom MSBuild task in the Microsoft.WinUI nuget that will do this. If there is a better place that this code should live/ship so that it isn't only WinUI apps that can use it, then we're open to that.

Unified CRT

I love this, I don't have much to add other than needing to ensure that we support the self-contained xcopy deployment scenario that will place the appropriate vcruntime binaries next to the .exe

[stevenbrix]

I forgot a b...

and b) Hooray for Project Reunion! 🙌 😄

[mikebattista]

Unified CRT
Having enabled Desktop<-->UWP project references, the developer must then address the runtime conflict of C Runtime libraries used by both modules. A Desktop module imports C:\Windows\System32\vcruntime140.dll and friends, while a UWP module imports C:\Program Files\WindowsApps\Microsoft.VCLibs.140*\vcruntime140_app.dll and friends. This impedance mismatch can be addressed by using the VCRT Forwarders package. But this technique is not very discoverable (it's not obvious that CRT dynalink errors can be addressed with a special adapter Nuget package).

A better solution is the inclusion of both CRT flavors in a unified VCLibs.UWPDesktop framework package, usable by both packaged UWP and centennial modules. This would obsolete the need for the VCRT Forwarders package, which is an explicit user action that adds build and deploy complexity, and increases install image size. Visual Studio could ensure that all binaries are "chameleon linked" with the CRT, using the Desktop import entries. Selection between Desktop CRT and unified VCLibs framework package could then be deferred until runtime, enabling broad reusability of modules. Visual Studio could also automatically select the unified VCLibs framework package for debugging and packaging behavior of UWP projects.

@stwish-msft is working on a single VC libs package that works everywhere.

[Scottj1s]

@stevenbrix - DesktopCompatible sounds promising but it should not be specific to WinUI projects, and so should be implemented in VS project system support. PackageReference for C++ still has some resistance to overcome, so we'll see if it lands.

@mikebattista - Thanks for making that statement public. I wasn't sure if it had been.

[stevenbrix]

@stevenbrix - DesktopCompatible sounds promising but it should not be specific to WinUI projects, and so should be implemented in VS project system support.

@Scottj1s it is in the VS project system ☺️. It isn't WinUI specific.

[kennykerr]

Along the same lines, as limitations persist in Visual Studio, developers have increasingly gone elsewhere for their needs, building Windows apps with languages, toolchains, and IDEs that have nothing to do with traditional Visual Studio and msbuild. Some languages like C++ don't have a standard build system, but without good support from Visual Studio, developers are increasingly using cmake. Other languages like Rust include standard build support via cargo. In such cases, nuget and msbuild are completely foreign. Components written for Project Reunion should minimize the friction to make Windows approachable to more developers. A concession can be made to use nuget as the packaging format, while limiting the use of msbuild such that projects written in C++ or Rust can simply extract the component's winmd and DLLs without any further knowledge of the component's implementation and dependencies.

[davkean]

Having been around for Win8/UWP on the Visual Studio sides of things, these "restrictions" around referencing are more better described as Windows business rules (at the time) being enforced by Visual Studio.

There is also a type unification problem, in that .NET Framework projects do not get the WindowsRuntime bridge assemblies between C# <-> WinRT. Allowing the reference might appear to succeed, but without these three assemblies, various things won't work:

System.Runtime.InteropServices.WindowsRuntime.dll
System.Runtime.WindowsRuntime.dll
System.Runtime.WindowsRuntime.UI.Xaml.dll

Has this been addressed, or are there plans to address this?

@stevenbrix: Looking at code, it looks like DesktopCompatible is only respected by C++ and not C#. Is that intentional?

[jonwis]

NuGet vs vcxpkg vs cmake

Seems reasonable for us to have a "Using Project Reunion" wiki page here or sample directory in the repo showing how to integrate the built artifacts with non-Visual Studio / msbuild environments. The Project Reunion team will initially focus on Visual Studio to get going, and we welcome pulls from the community to help us learn about the other project systems.

I think @kennykerr was signing himself up for cmake/Rust integration? :)

[kennykerr]

The point is that we don't want to put the burden on component authors to have to support N different build systems. If we can follow certain simple conventions we can support a large variety of build systems and languages simply by virtue of not having packaging requirements that lock developers into using Visual Studio.

[sylveon]

I don't like the "fix" for the CRT. It only cares about packaged scenarios and unpackaged scenarios will still have to ship the forwarders. Doesn't fix the root issue either.

It is possible to force linking UWP project to the desktop CRT using a property that's undocumented in VS and I've been successfully using it for a while in an unpackaged XAML Island scenario with custom XAML controls (since for now the XAML compiler and tools only light up in UWP projects). No need to ship any forwarder, just the DLL containing the compiled XAML code.

In my opinion, we should fix the root issue of XAML tooling only available in UWP projects (support building XAML within a desktop project), and ship WinUI builds linked against the desktop CRT for use by both packaged and unpackaged Win32 scenarios (since those scenarios already depend on that CRT anyways). That would also remove the need for the DesktopCompatible property for many use cases (although it would still be useful for custom controls which support both Win32 and UWP). Also why did nobody tell me this I could use that property!

PackageReference support would be great. Its absence has been a major pain point to adopting NuGet for C++, and it's obvious that C++ is an afterthought of the NuGet team. I pick vcpkg over NuGet whenever possible for libraries (the only lib I use NuGet for currently is C++/WinRT, by lack of choice) because since C++ is the only actually supported language, the selection of libraries is much bigger, and it handles C++ specific scenarios that NuGet just doesn't (like customizing the library build or building from head). It doesn't handle versionning though which can be a bit annoying when your CI fails because an update was published to a library you depend on which broke the API.

There are also other glaring flaws in C++ support, like the XAML compiler often emitting code that outright doesn't compile when the same XAML in C# does: see microsoft/microsoft-ui-xaml#2721 and microsoft/microsoft-ui-xaml#2429. These should be fixed before the stable release of WinUI 3 because they're pretty basic scenarios that developers who adopt it will run into quickly (hell, I'm writing very simple GUIs in XAML Islands for a simple program and I ran into both without even trying)

These obvious issues make it feel like C++ is a second class citizen of the XAML compiler.

[sylveon]

As for the WinRT activation thing, how would auto generating the manifest as part of a build task work if the user already has a custom app manifest with other things (like Common Controls 6, per monitor v2 DPI, long path awareness, the supportedOS declaration, etc...).

Using Detours feels more like a hack than a proper solution.

[jonwis]

mt.exe knows how to merge multiple "snippets" into a final manifest during build.

[Scottj1s]

Another way to address manifest complexities is to make them optional. C++/WinRT, C#/WinRT and Rust/WinRT all provide reg-free AND manifest-free activation, by simulating what RoGetMetaDataFile does (described under WinRT type activation for C#/WinRT). System reg-free WinRT activation added in 19H1 should do likewise.

[stevenbrix]

There is also a type unification problem, in that .NET Framework projects do not get the WindowsRuntime bridge assemblies between C# <-> WinRT. Allowing the reference might appear to succeed, but without these three assemblies, various things won't work:

System.Runtime.InteropServices.WindowsRuntime.dll
System.Runtime.WindowsRuntime.dll
System.Runtime.WindowsRuntime.UI.Xaml.dll

Has this been addressed, or are there plans to address this?

@davkean I'm fairly sure we have parity, and everything in those assemblies has been moved to winrt.runtime.dll, but @jkoritzinsky or @Scottj1s would know for sure

I don't like the "fix" for the CRT. It only cares about packaged scenarios and unpackaged scenarios will still have to ship the forwarders. Doesn't fix the root issue either.

It is possible to force linking UWP project to the desktop CRT using a property that's undocumented in VS and I've been successfully using it for a while in an unpackaged XAML Island scenario with custom XAML controls (since for now the XAML compiler and tools only light up in UWP projects). No need to ship any forwarder, just the DLL containing the compiled XAML code.

@sylveon how do you get vcruntime onto the system? I'd think you'd have to either ship the forwarders, or the crt itself. I'm probably going to butcher this slightly, but for WinUI, the forwarders are needed because we link against vcruntime140_app.lib, and in desktop apps, vcruntime140.dll is used. So you need the forwarders to go back and forth. My understanding of the "unified" crt is that everyone links against vcruntime140.lib, and then the appropriate implementation dll (either vcruntime140_app.dll or vcruntime140.dll) is used at runtime when you reference the framework package from a packaged app. Unpackaged apps will always have to figure out a way to ensure the crt is on the machine, either through app local or the VCRedist MSI (or some other means I'm unaware of).

[jkoritzinsky]

Yes, all types in those three assemblies now live in WinRT.Runtime.dll.

[stevenbrix]

Looking at code, it looks like DesktopCompatible is only respected by C++ and not C#. Is that intentional?

@davkean hmm, I haven't had any problems referencing a C++ project from a C# project, so it seems to work?

[kennykerr]

Unpackaged apps will always have to figure out a way to ensure the crt is on the machine

A consumer should not be forced to figure out what dependencies a component has. A component should be responsible for its own dependencies. The CRT a component uses (if any) is an implementation detail of that component just like any other dependency. In practice, you can simply package up the CRT DLLs with your component (or use the static CRT which is more reliable).

[Scottj1s]

@stevenbrix - that's an accurate description of the CRT today, using the VCRT Forwarders package to unify things. The ultimate goal is for VS to link against the desktop imports (vcruntime140.lib and friends) and for the loader to resolve against those as well, without forwarding. That would be accomplished by including both desktop and app variants of the CRT dlls in the VCLibs.UWPDesktop framework package, which would then effectively obsolete the VCLibs framework package.

[Scottj1s]

I generally agree with @kennykerr about dependencies being implementation details, and so side with static CRT as the preferred approach. But many third party libraries provide both static and dynamic CRT linked variants - for good reasons. The latter passes along the responsibility of bundling the VCRedist. A component can't easily do that, and shouldn't.

[stevenbrix]

Unpackaged apps will always have to figure out a way to ensure the crt is on the machine

A consumer should not be forced to figure out what dependencies a component has. A component should be responsible for its own dependencies.

I couldn't agree more, although I'm not sure what it means in practice. Every Project Reunion component is written in c++ and has a dependency on the CRT. So, do we do something like WPF did with vcruntime140_cor3.dll and have each component ship a private copy of the crt? Then there will be a vcruntime140_winui.dll, vcruntime140_appmodel.dll, vcruntime140_xyz.dll, or is there a single vcruntime140_reunion.dll?

Like you said @kennykerr, dynamic crt is an implementation detail of the component, much like microsoft.ui.xaml.dll is an implementation of WinUI. As long as we make sure the crt assembly is located next the .exe file, then I think that works. I'd love to have your input on this @sylveon, we know this approach will work for .NET apps that use PublishSingleFile (with WPF as prior art), but we have much less information/data on users who are building unpackaged c++ apps.

[sylveon]

Of course I still have to get the desktop CRT onto the system, which is what my installer does.

Although I plan on shipping a MSIX version next update, with an unpackaged portable build (so I'll most likely end up using app-local CRT here). For those portable builds, not having to ship two CRTs (or forwarders) would be ideal, for among other things file size and build complexity. This is already achieved today with my UWP project linked against the desktop CRT, and AFAIK will be regressed by WinUI using the app CRT, because I'll end up shipping either forwarders or the app CRT in its entirety to use WinUI in those portable builds.

There are other advantages to UWP projects targeting desktop CRT (or unlocking the XAML tooling in desktop projects), like being able to use desktop family only Win32 methods within the XAML code behind, or linking with static libraries that target the desktop CRT.

However, a huge blocker to MSIX adoption for small developers like me is the need to purchase an expensive code signature certificate. I've been able to get one thanks to a generous donator but not everyone has my luck. The store doesn't always works as an alternative because not all consumers are fan of the store (I've heard it's also not liked so much internally), and some have outright uninstalled the store from their machine.

For the packaged scenarios, I don't really mind if WinUI links against the app CRT, as long as this doesn't need me to ship forwarders in my package (so the suggested universal VCLibs.UWPDesktop framework package would be good here).

[davkean]

Yes, all types in those three assemblies now live in WinRT.Runtime.dll.

How does that work on .NET Framework? Some of those assemblies contain projected types. I know the new stuff is removing projection, but we're talking about loosening references between UWP -> .NET Framework where projection still lives. A bit confused how the pieces fit together.

[davkean]

@stevenbrix

That image is of the SDK-based projects which is a different project-system, not .NET Framework projects.

[jkoritzinsky]

I missed that we were talking about Framework. Does Framework not have any of these assemblies? I'd expect them to be available given that Xaml Islands enables using APIs that touch some of the bridge types and that works to my knowledge.

[davkean]

.NET Framework physically contains those assemblies, they do not expose them in the targeting pack, ie the thing the developer compiles against which will be needed so that the compiler can find the types.

[stevenbrix]

@jkoritzinsky I didn't think we we were talking about framework either. I definitely assumed we were talking about core and missed that, sorry @davkean

I could be wrong, but I'm fairly sure we don't have any plans on investing in .NET Framework. But that's mostly because of the direction the .NET team is going.

[davkean]

There's an entire section referring to this in the bug:

Desktop <--> UWP Project References
Project references cannot be added in Visual Studio between Desktop and UWP (Universal Windows) projects. Attempts to do so result in an error: A reference cannot be added because the two projects target different platforms.. This can be worked around with a project file edit, or introduction of a directory.build.* file, to manually add the project reference. In other words, this is strictly a tooling restriction, which prevents interop unless developers are aware of the workarounds.

I have no idea what this is referring to if we're not talking about .NET Framework:

[davkean]

Oh, C++ only?

[sylveon]

Yes, currently the IDE refuses and you have to drop down to the .vcxproj file manually.

[sylveon]

[soumyamahunt]

Another small request is to enable developers modify .vcxproj file without doing "Unload project", as it can already be done with .csproj files.

[DrusTheAxe]

WinRT activation issues. Historically, activation of user-defined WinRT components was restricted to packaged apps.

Obligatory link to MSIX Dynamic Dependencies :-)

[Zingam]

Another reason to use other build systems besides VStudio limitations is cross-platform development. Such application could target Mac/Linux and CppWinRT/UWP on Windows and use Visual Studio Code as IDE/CMake to build (all required VSCode extensions are developed by Microsoft) .
This workflow is great but with CppWinRT/UWP, where there are limitations.

[MouriNaruto]

@kennykerr

The point is that we don't want to put the burden on component authors to have to support N different build systems. If we can follow certain simple conventions we can support a large variety of build systems and languages simply by virtue of not having packaging requirements that lock developers into using Visual Studio.

I hope C++ PackageReferences with TargetFrameworkMoniker support will added in the future. And I think it's useful for this issue.

I feel really angry when I read the reply in NuGet/NuGet.Client#3145 (comment).

For projects which uses some packages needs the complex build system like C++/WinRT (I really love C++/WinRT, it really saves me because I am a C++ UWP developer.) and etc. I DON'T THINK SUCH A VCPKG AND CMAKE CAN RESOLVE THAT. So I think it's necessary to improve the C++ support in MSBuild and NuGet. (I also hope we can use MSBuild to build C++ projects under Linux and macOS.)

[Scottj1s]

With WinAppSDK 1.0.2, all binaries will use "Hybrid CRT" linkage, which eliminates one source of pain - end user apps having to be responsible for the CRT redist:
#2143

[fredemmott]

On the cmake topic: C++/WinRT and WinUI3 are usable with just add_executable() - no add_custom_target() etc in self-contained unpackaged mode, but requires CMake 3.23 (April) and Windows App SDK 1.1

Example: https://github.com/fredemmott/cmake-cpp-winrt-winui3

Also, if I understood the hybrid CRT correctly (results look correct in Dependencies), that's simple enough too: https://github.com/fredemmott/cmake-cpp-winrt-winui3/blob/master/HybridCRT.cmake