OpenApi XML comment cache gneration fails with System.ArgumentException: An item with the same key has already been added

[twenzel]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

I've mirgated to .NET 10 and enabled OpenApi and added the <InterceptorsNamespaces>$(InterceptorsNamespaces);Microsoft.AspNetCore.OpenApi.Generated</InterceptorsNamespaces> property to my project as the compiler told me to.

No I get an error System.ArgumentException: An item with the same key has already been added within the generated Mcrosoft.AspNetCore.OpenApi.Generated.XmlCommentCache class. Some elements occurs multiple times there.

Why?

Expected Behavior

No response

Steps To Reproduce

No response

Exceptions (if any)

No response

.NET Version

No response

Anything else?

No response

[twenzel]

Here's the reproduction project: https://github.com/twenzel/OpenApiIssueRepro

[martincostello]

Stack trace:

   D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\Microsoft.Extensions.ApiDescription.Server.targets(67,5): error System.AggregateException: One or more errors occurred. (An item with the same key has already been added. Key: T:OpenApiIssueRepro.Services.IdentityServerUserManager`1)
    D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\Microsoft.Extensions.ApiDescription.Server.targets(67,5): error  ---> System.ArgumentException: An item with the same key has already been added. Key: T:OpenApiIssueRepro.Services.IdentityServerUserManager`1
    D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\Microsoft.Extensions.ApiDescription.Server.targets(67,5): error    at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
    D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\Microsoft.Extensions.ApiDescription.Server.targets(67,5): error    at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
    D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\Microsoft.Extensions.ApiDescription.Server.targets(67,5): error    at Microsoft.AspNetCore.OpenApi.Generated.<OpenApiXmlCommentSupport_generated>FE9AC42871CBF450746D4365E99F2A745670A6C5FF9BDF56FEEA1B3CB6281171A__XmlCommentCache.GenerateCacheEntries() in D:\Coding\repros\twenzel\OpenApiIssueRepro\obj\Debug\net10.0\Microsoft.AspNetCore.OpenApi.SourceGenerators\Microsoft.AspNetCore.OpenApi.SourceGenerators.XmlCommentGenerator\OpenApiXmlCommentSupport.generated.cs:line 74
    D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\Microsoft.Extensions.ApiDescription.Server.targets(67,5): error    at Microsoft.AspNetCore.OpenApi.Generated.<OpenApiXmlCommentSupport_generated>FE9AC42871CBF450746D4365E99F2A745670A6C5FF9BDF56FEEA1B3CB6281171A__XmlCommentCache.get_Cache() in D:\Coding\repros\twenzel\OpenApiIssueRepro\obj\Debug\net10.0\Microsoft.AspNetCore.OpenApi.SourceGenerators\Microsoft.AspNetCore.OpenApi.SourceGenerators.XmlCommentGenerator\OpenApiXmlCommentSupport.generated.cs:line 67
    D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\Microsoft.Extensions.ApiDescription.Server.targets(67,5): error    at Microsoft.AspNetCore.OpenApi.Generated.<OpenApiXmlCommentSupport_generated>FE9AC42871CBF450746D4365E99F2A745670A6C5FF9BDF56FEEA1B3CB6281171A__XmlCommentOperationTransformer.TransformAsync(OpenApiOperation operation, OpenApiOperationTransformerContext context, CancellationToken cancellationToken) in D:\Coding\repros\twenzel\OpenApiIssueRepro\obj\Debug\net10.0\Microsoft.AspNetCore.OpenApi.SourceGenerators\Microsoft.AspNetCore.OpenApi.SourceGenerators.XmlCommentGenerator\OpenApiXmlCommentSupport.generated.cs:line 361
    D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\Microsoft.Extensions.ApiDescription.Server.targets(67,5): error    at Microsoft.AspNetCore.OpenApi.OpenApiDocumentService.GetOperationsAsync(IGrouping`2 descriptions, OpenApiDocument document, IServiceProvider scopedServiceProvider, IOpenApiOperationTransformer[] operationTransformers, IOpenApiSchemaTransformer[] schemaTransformers, CancellationToken cancellationToken)
    D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\Microsoft.Extensions.ApiDescription.Server.targets(67,5): error    at Microsoft.AspNetCore.OpenApi.OpenApiDocumentService.GetOpenApiPathsAsync(OpenApiDocument document, IServiceProvider scopedServiceProvider, IOpenApiOperationTransformer[] operationTransformers, IOpenApiSchemaTransformer[] schemaTransformers, CancellationToken cancellationToken)
    D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\Microsoft.Extensions.ApiDescription.Server.targets(67,5): error    at Microsoft.AspNetCore.OpenApi.OpenApiDocumentService.GetOpenApiDocumentAsync(IServiceProvider scopedServiceProvider, HttpRequest httpRequest, CancellationToken cancellationToken)
    D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\Microsoft.Extensions.ApiDescription.Server.targets(67,5): error    at Microsoft.Extensions.ApiDescriptions.OpenApiDocumentProvider.GenerateAsync(String documentName, TextWriter writer, OpenApiSpecVersion openApiSpecVersion)
    D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\Microsoft.Extensions.ApiDescription.Server.targets(67,5): error    --- End of inner exception stack trace ---
    D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\Microsoft.Extensions.ApiDescription.Server.targets(67,5): error    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
    D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\Microsoft.Extensions.ApiDescription.Server.targets(67,5): error    at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
    D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\Microsoft.Extensions.ApiDescription.Server.targets(67,5): error    at System.Threading.Tasks.Task.Wait(TimeSpan timeout, CancellationToken cancellationToken)
    D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\Microsoft.Extensions.ApiDescription.Server.targets(67,5): error    at Microsoft.Extensions.ApiDescription.Tool.Commands.GetDocumentCommandWorker.GetDocument(String documentName, String projectName, String outputDirectory, MethodInfo generateMethod, Object service, MethodInfo generateWithVersionMethod, String fileName)
    D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\Microsoft.Extensions.ApiDescription.Server.targets(67,5): error    at Microsoft.Extensions.ApiDescription.Tool.Commands.GetDocumentCommandWorker.GetDocuments(IServiceProvider services)
    D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\Microsoft.Extensions.ApiDescription.Server.targets(67,5): error    at Microsoft.Extensions.ApiDescription.Tool.Commands.GetDocumentCommandWorker.Process()
    D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\Microsoft.Extensions.ApiDescription.Server.targets(67,5): error MSB3073: The command "dotnet "D:\packages\nuget\microsoft.extensions.apidescription.server\10.0.0\build\../tools/dotnet-getdocument.dll" --assembly "D:\Coding\repros\twenzel\OpenApiIssueRepro\bin\Debug\net10.0\OpenApiIssueRepro.dll" --file-list "obj\OpenApiIssueRepro.OpenApiFiles.cache" --framework ".NETCoreApp,Version=v10.0" --output "D:\Coding\repros\twenzel\OpenApiIssueRepro\OpenApi" --project "OpenApiIssueRepro" --assets-file "D:\Coding\repros\twenzel\OpenApiIssueRepro\obj\project.assets.json" --platform "AnyCPU" " exited with code 11.

The code causing the exception:

private static Dictionary<string, XmlComment> GenerateCacheEntries()
{
    var cache = new Dictionary<string, XmlComment>();

    cache.Add(@"T:OpenApiIssueRepro.Services.IdentityServerUserManager`1", new XmlComment(@"User manager", null, null, null, null, false, null, null, null));
    cache.Add(@"T:OpenApiIssueRepro.Services.IdentityServerUserManager`1", new XmlComment(@"User manager", null, null, null, null, false, null, null, null));
    cache.Add(@"M:OpenApiIssueRepro.Controllers.Api.UsersController.GetUsers", new XmlComment(@"Get all users", null, null, @"", null, false, null, null, null));
    cache.Add(@"M:OpenApiIssueRepro.Services.IdentityServerUserManager`1.EnableIncludeDeletedUsers", new XmlComment(@"Searches for users including deleted", null, null, null, null, false, null, null, null));
    cache.Add(@"M:OpenApiIssueRepro.Services.IdentityServerUserManager`1.DisableIncludeDeletedUsers", new XmlComment(@"Searches for users including deleted", null, null, null, null, false, null, null, null));
    cache.Add(@"M:OpenApiIssueRepro.Services.IdentityServerUserManager`1.EnableIncludeDeletedUsers", new XmlComment(@"Searches for users including deleted", null, null, null, null, false, null, null, null));
    cache.Add(@"M:OpenApiIssueRepro.Services.IdentityServerUserManager`1.DisableIncludeDeletedUsers", new XmlComment(@"Searches for users including deleted", null, null, null, null, false, null, null, null));

    return cache;
}

The members for IdentityServerUserManager<TUser> are being added twice.

[desjoerd]

I've played with the repro. I could reduce it to this:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenApi();

var app = builder.Build();
app.MapOpenApi();

app.MapGet("/", () => new Bar());

app.Run();

/// <summary>
/// Should not be duplicated.
/// </summary>
public class GenericFoo<T>
{
}

public class Bar
{
    /// <summary>
    /// Will give duplicate xml comment on type GenericFoo. Only will happen when it's a property and generic.
    /// </summary>
    public GenericFoo<string> Foo { get; set; }
}

Which will emit

cache.Add(@"T:GenericFoo`1", new XmlComment(@"Should not be duplicated.", null, null, null, null, false, null, null, null));
cache.Add(@"T:GenericFoo`1", new XmlComment(@"Should not be duplicated.", null, null, null, null, false, null, null, null));
cache.Add(@"P:Bar.Foo", new XmlComment(@"Will give duplicate xml comment on type GenericFoo. Only will happen when it's a property and generic.", null, null, null, null, false, null, null, null));

From what I've tried the duplication only happens when it's a property. (I tried property, field, method and constructor).

[austindrenski]

Hi folks, I'm seeing this happen in projects referencing Jetbrains.Annotations.Source. Will share a full repro when I have a few, but wanted to flag this ASAP as I'm seeing it happen across member types (i.e., whereas @desjoerd was only seeing it with properties).

<!-- Directory.Build.props -->
<Project>

  <ItemGroup>
    <PackageReference Include="JetBrains.Annotations.Sources" />
  </ItemGroup>

</Project>

This happens despite JetBrains.Annotations.Source using the new(-ish) EmbeddedAttribute to mark provided types as local/private to the assembly, e.g.:

/// <summary>
/// Indicates that the value of the marked element could be <c>null</c> sometimes,
/// so checking for <c>null</c> is required before its usage.
/// </summary>
/// <example><code>
/// [CanBeNull] object Test() { ... }
/// 
/// void UseTest()
/// {
///   var p = Test();
///   var s = p.ToString(); // Warning: Possible 'System.NullReferenceException'
/// }
/// </code></example>
[AttributeUsage(
  AttributeTargets.Method
  | AttributeTargets.Parameter
  | AttributeTargets.Property
  | AttributeTargets.Delegate
  | AttributeTargets.Field
  | AttributeTargets.Event
  | AttributeTargets.Class
  | AttributeTargets.Interface
  | AttributeTargets.GenericParameter)]
[global::Microsoft.CodeAnalysis.EmbeddedAttribute]
internal sealed class CanBeNullAttribute : Attribute { }

fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
      An unhandled exception has occurred while executing the request.
      System.ArgumentException: An item with the same key has already been added. Key: T:JetBrains.Annotations.CanBeNullAttribute
         at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
         at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
         at Microsoft.AspNetCore.OpenApi.Generated.<OpenApiXmlCommentSupport_generated>FD6AD04B547C49881369CEBE736AB26141E4C9BB6128A6ACAAE98F45FB3D341E4__XmlCommentCache.GenerateCacheEntries() in ***/artifacts/obj/***/debug/Microsoft.AspNetCore.OpenApi.SourceGenerators/Microsoft.AspNetCore.OpenApi.SourceGenerators.XmlCommentGenerator/OpenApiXmlCommentSupport.generated.cs:line 761
         at Microsoft.AspNetCore.OpenApi.Generated.<OpenApiXmlCommentSupport_generated>FD6AD04B547C49881369CEBE736AB26141E4C9BB6128A6ACAAE98F45FB3D341E4__XmlCommentCache.get_Cache() in ***/artifacts/obj/***/debug/Microsoft.AspNetCore.OpenApi.SourceGenerators/Microsoft.AspNetCore.OpenApi.SourceGenerators.XmlCommentGenerator/OpenApiXmlCommentSupport.generated.cs:line 67
         at Microsoft.AspNetCore.OpenApi.Generated.<OpenApiXmlCommentSupport_generated>FD6AD04B547C49881369CEBE736AB26141E4C9BB6128A6ACAAE98F45FB3D341E4__XmlCommentOperationTransformer.TransformAsync(OpenApiOperation operation, OpenApiOperationTransformerContext context, CancellationToken cancellationToken) in ***/artifacts/obj/***/debug/Microsoft.AspNetCore.OpenApi.SourceGenerators/Microsoft.AspNetCore.OpenApi.SourceGenerators.XmlCommentGenerator/OpenApiXmlCommentSupport.generated.cs:line 4535
         at Microsoft.AspNetCore.OpenApi.OpenApiDocumentService.GetOperationsAsync(IGrouping`2 descriptions, OpenApiDocument document, IServiceProvider scopedServiceProvider, IOpenApiOperationTransformer[] operationTransformers, IOpenApiSchemaTransformer[] schemaTransformers, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.OpenApi.OpenApiDocumentService.GetOpenApiPathsAsync(OpenApiDocument document, IServiceProvider scopedServiceProvider, IOpenApiOperationTransformer[] operationTransformers, IOpenApiSchemaTransformer[] schemaTransformers, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.OpenApi.OpenApiDocumentService.GetOpenApiDocumentAsync(IServiceProvider scopedServiceProvider, HttpRequest httpRequest, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Builder.OpenApiEndpointRouteBuilderExtensions.<>c__DisplayClass0_0.<<MapOpenApi>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Http.Generated.<GeneratedRouteBuilderExtensions_g>F0399B8B15BE443123F4BECEE84B112C1912EE3F44D9DDD7BFA38E371FB917343__GeneratedRouteBuilderExtensionsCore.<>c__DisplayClass2_0.<<MapGet0>g__RequestHandler|5>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|7_0(Endpoint endpoint, Task requestTask, ILogger logger)
         at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
         at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)

Temporary workaround

In the meantime, if anyone else runs into this, I've been able to work around this by adding an MSBuild target to post-process the generated documentation file(s).

<!-- Directory.Build.targets -->
<Project>

  <!-- TODO: https://github.com/dotnet/aspnetcore/issues/64378 -->
  <Target Name="RemoveXmlDocsFromJetBrainsAnnotations"
    AfterTargets="CoreCompile"
    BeforeTargets="CoreBuild"
    Condition="'$(GenerateDocumentationFile)' == 'true'">

    <PropertyGroup>
      <_XslContent>
        <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
          <xsl:template match="@*|node()">
            <xsl:copy>
              <xsl:apply-templates select="@*|node()" />
            </xsl:copy>
          </xsl:template>
          <xsl:template match="member[starts-with(@name, 'F:JetBrains')]" />
          <xsl:template match="member[starts-with(@name, 'M:JetBrains')]" />
          <xsl:template match="member[starts-with(@name, 'P:JetBrains')]" />
          <xsl:template match="member[starts-with(@name, 'T:JetBrains')]" />
        </xsl:stylesheet>
      </_XslContent>
    </PropertyGroup>

    <XslTransformation
      OutputPaths="$(OutputPath)$(AssemblyName).xml.tmp"
      XmlInputPaths="$(OutputPath)$(AssemblyName).xml"
      XslContent="$(_XslContent)" />

    <Move
      DestinationFiles="$(OutputPath)$(AssemblyName).xml"
      SourceFiles="$(OutputPath)$(AssemblyName).xml.tmp" />

  </Target>

</Project>

[austindrenski]

I haven't had time to think through the implications, so perhaps there's good reason for these, but from a cursory glance, this issue seems to stem from one or more of the following design decisions:

1. using cache.Add("...", "...") instead of cache["..."] = "..."

   • aspnetcore/src/OpenApi/gen/XmlCommentGenerator.Emitter.cs

     Line 695 in 7387de9

     codeWriter.WriteLine($"cache.Add({FormatStringForCode(memberKey)}, {EmitSourceGeneratedXmlComment(comment)});");

2. not de-duplicating here:

   • aspnetcore/src/OpenApi/gen/XmlCommentGenerator.cs

     Lines 50 to 54 in 7387de9

     	var (generatedCommentsFromXmlFile, generatedCommentsFromCompilation) = output.Left;
     	var compiledXmlFileComments = !generatedCommentsFromXmlFile.IsDefaultOrEmpty
     	? string.Join("\n", generatedCommentsFromXmlFile)
     	: string.Empty;
     	Emit(context, compiledXmlFileComments, generatedCommentsFromCompilation, groupedAddOpenApiInvocations);

3. Something about this check (erroneously?) including types marked with [Embedded]

   • aspnetcore/src/OpenApi/gen/XmlCommentGenerator.Parser.cs

     Lines 122 to 124 in 7387de9

     	// Only include symbols that are declared in the application assembly or are
     	// accessible from the application assembly.
     	(SymbolEqualityComparer.Default.Equals(symbol.ContainingAssembly, compilation.Assembly) || symbol.IsAccessibleType()) &&

[desjoerd]

There are a couple of things.

1. From the compilation it does visit constructured generic types resulting in duplicate unconstrained generic types. Foo<string> and Foo<> are both emitting to Foo'1

2. From xml documents of referenced projects it does not de-duplicate (internal) types with the same namespace.

I think it's best to not have duplicates in the emitted code, so it's probably making sure we do not emit duplicates in the generator itself.