Productize LibraryImportGenerator for use outside of runtime

[AaronRobinsonMSFT]

Issues to address/consider after ship-ready work (#60212):

APIs:

• LibraryImportAttribute for p/invoke source generation #46822
  • Remove unsupported fields from GeneratedDllImport #61181
  • [DllImportGenerator] Update GeneratedDllImportAttribute handling of character set / encoding #61326
• Attributes and Analyzer Diagnostics to support custom struct marshalling in the DllImport Source Generator #46838
• [API Proposal]: Specify supported custom type marshalling features on CustomTypeMarshallerAttribute #66121
• [API Proposal]: Implementations of marshallers using CustomTypeMarshallerAttribute #66623

Reconciliation of blittable/unmanaged:

• [API Proposal]: System.Runtime.CompilerServices.RuntimeCompatibilityAttribute.DisableRuntimeMarshalling #60639
• Update design doc for user-defined type marshalling with the new DisableRuntimeMarshallingAttribute
  • Update StructMarshalling design now that DisableRuntimeMarshallingAttribute is approved #63765
• Update source generator for DisableRuntimeMarshallingAttribute and reconciled unmanaged/blittable
  • Source generator treats unmanaged as not requiring marshaling
  • Add marshalling info (NativeMarshalling or MarshalUsing) for types in runtime libraries that need it
  • Runtime libraries use DisableRuntimeMarshallingAttribute
  • Convert p/invokes using affected types to GeneratedDllImport - known p/invokes are marked // TODO: [DllImportGenerator] and #pragma warning disable DLLIMPORTGENANALYZER015
• Code fix to add DisableRuntimeMarshalling when required #67676

Custom marshallers:

• Add marshaller types to runtime libraries
  • Strings: [LibraryImportGenerator] Add/use CustomTypeMarshaller implementations for string marshalling #67635
  • Arrays: Expose ArrayMarshaller and PointerArrayMarshaller #68173

Release:

• Update diagnostics to match convention/prefix for runtime libraries
  • Use `SYSLIB1###' IDs and update doc
  • Update p/invoke source generator errors to use SYSLIB prefix #65983
• Localization - Enable localization for LibraryImportGenerator #67107
• Include DllImportGenerator in NETCoreApp ref-pack
  • See We need a repo convention to ship source generators as part of the ref pack #61321
  • Add LibraryImportGenerator to ref pack #68247
• Apply GeneratedCodeAttribute Interop source generator should insert [GeneratedCodeAttribute] #64562
• Enable trimming of attributes used for generation

Fixer:

A code fixer should be provided to help with the conversion to using generated p/invokes via the new attribute. We may not support all options based on added complexity versus usage.

• Basic conversion to new attribute where all options specified on DllImport are supported by source generator
• PreserveSig: change managed signature to return int and have an out param if necessary, add custom marshaller for return value to throw if necessary, update callers,
• ExactSpelling: different fix option to append suffix on EntryPoint property
• Add stubbed out implementation for custom marshallers

Issues:

• [DllImportGenerator] "MarshalUsing" ignores "ref" return type runtimelab#1650
• [DllImportGenerator] ICustomMarshaler ~support runtimelab#963
• Pin stage only accepts FixedStatementSyntaxes runtimelab#1653
• DllImportGenerator conversion code fix should make the containing types partial #62291
• DllImportGenerator should error if attribute is applied to an extern #62700
• [DllImportGenerator] Remove DLLIMPORTGENERATOR_ENABLED define option #63464
• [DllImportGenerator] DefaultDllImportSearchPathsAttribute should be forwarded to local p/invoke #65154

[kant2002]

Does attempt to add DllImportGenerator to WinForms require all outstanding tasks here to be completed?

[AaronRobinsonMSFT]

Does attempt to add DllImportGenerator to WinForms require all outstanding tasks here to be completed?

Not necessarily but we reserve the right to change any of the generated code or APIs that we are using. Additionally, we haven't put the source generator on a feed that would make consumption simple. From the Interop team's perspective, all products not in the dotnet/runtime repo are the same.

[kant2002]

There some version on the feed 7.0.0-preview.2.22110.4 (in addiional to experimental feed), but that package blocked by internal GeneratedDllImportAttribute.

I do understand that WinForms is external product for you. I would like to throw some boulder on WinForms by applying BuiltInComSupprted=false and DisableBuiltInMarshalling=true. I expect this to be problematic and prefer not to build custom runtime for that experiment. Is this somehow possible?

If not, can you tell me what's blocking the team from going preview mode?

[AaronRobinsonMSFT]

There some version on the feed 7.0.0-preview.2.22110.4

I didn't realize that. Perhaps something can be done here.

If not, can you tell me what's blocking the team from going preview mode?

The API reviews Reconciliation of blittable/unmanaged, and Release sections above are all mostly required. We then need to update the tooling with the new name and provide APIs that support the support for the generated, right now everything is internal since the API surface area and design are all in flux.

I should also note that the current plan is to productize this for .NET 7, but that is not something we're committed to. The only commitment is to ship the entire dotnet/runtime repo with generated source for all DllImports. Based on other factors this could slip and become productized in the next release. I don't think that is going to happen, but it is possible. The general point is there are a lot of moving parts to productizing this source generator because of expectations on how it works and support for assemblies containing the generated code. Even acknowledging that previews are not supported doesn't change much. We would like to ensure what we do release in a preview is something we can feasibly deliver so we don't need to remove it prior to an RC. The majority of the work here is around software support infrastructure and tenets that must be followed due to what the product is doing.

Other than hacking together a custom runtime, I'm not sure how easy this is going to be at present.

[kkukshtel]

Just as a question - is an intended use case of this to be something that can replace the ClangSharp P/Invoke generator? https://www.nuget.org/packages/ClangSharpPInvokeGenerator

[AraHaan]

Not at all, ChangSharp is meant for converting native header files into .NET PInvokes (like the Windows SDK files for TerraFX.Interop.Windows for example).

[DualBrain]

Is the source generator being built leveraging Roslyn SyntaxGenerators so that it will be consumable by C# and VB?

[AaronRobinsonMSFT]

Is the source generator being built leveraging Roslyn SyntaxGenerators so that it will be consumable by C# and VB?

That is a potential future option, however it is not going to work for VB.Net presently. The reason is lack of support for function pointers and other low-level .NET features in VB.Net.

The reason it was chosen in our case was for nicely formatting the source and it is simply easier to express/debug the intent if we aligned with how Roslyn expresses the tree.

[kant2002]

I would like to comeback and ask again what's happening with this feature. LibraryImportGenerator is missing from feeds right now and there no easy way to consume source generator. From what I see it was disabled after discussion here #65495 (comment) but that still does not explain why it is needed.

Which is sad, since that limit ability to play with feature in the preview mode without expecting it to be shipped. There couple other enthusiasts from the Discord who probably willing to put their dirty hands into this feature.

Is there something which I can do to unblock packaging (because from my understanding it was some build failures)?

[AaronRobinsonMSFT]

This is now consumable in preview 5. This source generator is in the ref pack for when targeting the preview 5 TFM - this is by design and that is the minimum TFM requirement.

See #68247.

[shaggygi]

@AaronRobinsonMSFT Any plans to add docs/examples on scenarios and how to use? I'm sure there already is and I'm overlooking. Just trying to understand feature and some of the basics to get started. Thx

[kant2002]

Thank you! This new made my day. So if I understand correctly LibraryImportGenerator is part of .NET SDK and would be shipped together with it. Correct?

[AaronRobinsonMSFT]

Any plans to add docs/examples on scenarios and how to use? I'm sure there already is and I'm overlooking. Just trying to understand feature and some of the basics to get started. Thx

There are plans and as with any product there will be docs and samples once we get out of preview. The repo does contain a sample - see https://github.com/dotnet/runtime/tree/main/src/samples/LibraryImportGeneratorSample.

Low-level design docs and how it works can be found under https://github.com/dotnet/runtime/tree/main/docs/design/features and in https://github.com/dotnet/runtime/tree/main/docs/design/libraries/LibraryImportGenerator.

This is a non-trivial feature that is still technically in flux since we are in preview. Please be patient with the low-level nature of the documentation and how the source generator is used from an end-user perspective. Proper higher-level walkthroughs are forth coming but will likely be closer to the RC. For now, the team is focused on ensuring correctness for non-netcoreapp scenarios and comprehensive testing. Also, the work to make anything an enterprise quality product is extensive - see this issue and all the non-functional work items - which is why the walkthroughs and samples are not yet ready.

[shaggygi]

@AaronRobinsonMSFT completely understand. thanks for the info and look forward to feature. keep up the good work 👍

[AraHaan]

This is now consumable in preview 5. This source generator is in the ref pack for when targeting the preview 5 TFM - this is by design and that is the minimum TFM requirement.

See #68247.

sad it cant be in a nuget package as well for those who want to use the generator downlevel (in .NET 6 projects).

[AaronRobinsonMSFT]

@AraHaan Unfortunately the source generator has strict requirements on .NET 7 features. However, the Roslyn team's source generator API permits writing the source to disk, which could then be modified as needed to target a .NET 6 project. This would likely require substantial care use down level as there are runtime semantics that aren't known to the compiler. See #60639.

Enabling source being written to disk can be done by adding the following MSBuild property to the project.

<EmitCompilerGeneratedFiles>true</EmitCompilerGeneratedFiles>

[elinor-fung]

Closing out this issue as the work to the LibraryImportGenerator exposed and consumable outside of the runtime repo is done. It is in the daily main builds and will be in .NET 7.0 Preview 5.

Please do file issues to let us know about problems you hit or questions you have when trying to consume this.