Add option to generate out IntPtr for COM output pointer parameters

[wjk]

Is your feature request related to a problem? Please describe.

I am making heavy use of the .NET 5/6 ComWrappers class to make my Windows Forms codebases compatible with NativeAOT. I can create as many ComWrappers subclasses as I want, each of which knows how to to create bindings for a different set of COM interfaces. (I am using this Roslyn source generator to create the marshaling code.) However, only one ComWrappers instance can be used with ComImport interfaces, and it must know about every ComImport interface that is used, anywhere in the program and all of its libraries. There is no good way to do this a priori, and there is no way to specify an alternate ComWrappers instance for a ComImport type. The global instance must be used in this situation.

However, if I have a COM instance pointer as an IntPtr, I can use an instance of ComWrappers to create an AOT-compatible RCW for that instance. As long as I do not use UnmanagedType.IUnknown or any similar API, I can use whatever ComWrappers object I want, allowing me to pick one that understands the type of object I am giving it. However, there is currently no way I know of to have CsWin32 generate this kind of binding.

Describe the solution you'd like

Add a flag in NativeMethods.json that will generate code that uses out IntPtr as the type for arguments annotated as ComOutPtr. I can then use my custom ComWrappers code to call methods, or use Marshal.QueryInterface and Marshal.Release on the pointer directly.

Describe alternatives you've considered

Calling Marshal.GetIUnknownForObject() is not an option, as that call will occur after the default ComWrappers instance attempts to generate an RCW for the pointer.

Additional context

I will use extension methods to create overloads for these APIs that are easier to call.

As a side note: It would be really, really cool if there was some way to make WinFormsComInterop.SourceGenerator aware of CsWin32 and be able to generate marshaling stubs for all the interfaces automatically, but unfortunately you can’t use source generator output as input to another source generator. I must manually provide an attribute that triggers the WinFormsComInterop source generator for every interface used. However, one line of code per interface is far preferable to having to bind all of it myself. Thanks!

[kant2002]

@AArnott I would like to clarify why this is important not only for niche scenarios like NativeAOT.
.NET 6 has ability to disable built-in COM, and when publish trimmed application built-in COM is disabled and should be explicitly enabled.
To accommodate for these scenarios in .NET 5 was introduced ComWrappers API which allow develop to write their own RCW and CCW for COM interfaces.

Long-term for this project probably is that you should generate CCW/RCW for these interfaces too, like https://github.com/microsoft/CsWinRT since that's where .NET is moving.

I read in #167 that you under pressure to ship working product, so I will ask - would you consider contributions in that area?

[AArnott]

@wjk: I think the setting you want already exists.

CsWin32/src/Microsoft.Windows.CsWin32/settings.schema.json

Lines 40 to 44 in 207ce8b

	"allowMarshaling": {
	"title": "Emit COM interfaces instead of structs, and allow generation of non-blittable structs for the sake of an easier to use API.",
	"type": "boolean",
	"default": true
	},

Setting allowMarshaling: false in the NativeMethods.json file will produce structs instead of interfaces, using pointers everywhere and requiring you to call QI/AddRef/ReleaseRef everywhere yourself. Essentially, completely bypassing the .NET interop layer.

Did you look at that property and determine it doesn't suit you somehow? If so, can you elaborate?

@kant2002: CsWinRT and CsWin32 may do well to share more capabilities. We're looking into that soon.
And yes, I think we'd accept contributions that fix some of our interop bugs.

[wjk]

@AArnott I have considered that technique, but have found that it will not work. allowMarshaling=false currently only generates code for regular P/Invoke (managed ➪ COM). I cannot use the generated structures and pointer to perform reverse P/Invoke (COM ➪ managed), which is easily doable with ComWrappers.

Furthermore, for some reason when I disable allowMarshaling for some reason the source generator stops making its generated outputs visible to Visual Studio. ILSpy on the compiled binary confirms that they were emitted as expected, but they are not visible to the editor, Solution Explorer, or anywhere else in VS. This prevents me from effectively testing CsWin32, because I am getting tons of spurious symbol-not-found errors. This can only be avoided by turning allowMarshalling back on, and then reloading the solution to force VS to re-evaluate the code generators. I don’t know why this would happen, but I do remember it didn’t happen when I tried this library in the past. (I’m running VS 2022 Preview 3.1 and using CsWin32 0.1.506-preview.) Thanks!

[AArnott]

I cannot use the generated structures and pointer to perform reverse P/Invoke (COM ➪ managed), which is easily doable with ComWrappers.

Yes, that's true.

Furthermore, for some reason when I disable allowMarshaling for some reason the source generator stops making its generated outputs visible to Visual Studio

I see. That's very strange. I can repro it, and then when I switch back it doesn't fix it right away, as you say. that's worth a separate bug.