[BUG] System.InvalidOperationException System.ClientModel.Primitives.JsonPatch&

[rodriciru]

Describe the bug

This is an Open API project that make use of the OpenAI IA thought SemanticKernel

System.InvalidOperationException: The type 'System.ClientModel.Primitives.JsonPatch&' of property 'Patch' on type 'OpenAI.Responses.ResponseTokenUsage' is invalid for serialization or deserialization because it is a pointer type, is a ref struct, or contains generic parameters that have not been replaced by specific types.
   at System.Text.Json.ThrowHelper.ThrowInvalidOperationException_CannotSerializeInvalidType(Type typeToConvert, Type declaringType, MemberInfo memberInfo)
   at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.CreatePropertyInfo(JsonTypeInfo typeInfo, Type typeToConvert, MemberInfo memberInfo, NullabilityInfoContext nullabilityCtx, JsonSerializerOptions options, Boolean shouldCheckForRequiredKeyword, Boolean hasJsonIncludeAttribute)
   at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.AddMembersDeclaredBySuperType(JsonTypeInfo typeInfo, Type currentType, NullabilityInfoContext nullabilityCtx, Boolean constructorHasSetsRequiredMembersAttribute, PropertyHierarchyResolutionState& state)
   at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.PopulateProperties(JsonTypeInfo typeInfo, NullabilityInfoContext nullabilityCtx)
   at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.CreateTypeInfoCore(Type type, JsonConverter converter, JsonSerializerOptions options)
   at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.GetTypeInfo(Type type, JsonSerializerOptions options)
   at System.Text.Json.Serialization.Metadata.JsonTypeInfoResolverChain.GetTypeInfo(Type type, JsonSerializerOptions options)
   at System.Text.Json.Serialization.Metadata.JsonTypeInfoResolverWithAddedModifiers.GetTypeInfo(Type type, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializerOptions.GetTypeInfoNoCaching(Type type)
   at System.Text.Json.JsonSerializerOptions.CachingContext.CreateCacheEntry(Type type, CachingContext context)
--- End of stack trace from previous location ---
   at System.Text.Json.JsonSerializerOptions.GetTypeInfoInternal(Type type, Boolean ensureConfigured, Nullable`1 ensureNotNull, Boolean resolveIfMutable, Boolean fallBackToNearestAncestorType)
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo.Configure()
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo.ConfigureProperties()
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo.Configure()
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo.<EnsureConfigured>g__ConfigureSynchronized|174_0()
--- End of stack trace from previous location ---
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo.<EnsureConfigured>g__ConfigureSynchronized|174_0()
   at System.Text.Json.JsonSerializerOptions.GetTypeInfoInternal(Type type, Boolean ensureConfigured, Nullable`1 ensureNotNull, Boolean resolveIfMutable, Boolean fallBackToNearestAncestorType)
   at System.Text.Json.Schema.JsonSchemaExporter.GetJsonSchemaAsNode(JsonSerializerOptions options, Type type, JsonSchemaExporterOptions exporterOptions)
   at Microsoft.AspNetCore.OpenApi.OpenApiSchemaService.CreateSchema(OpenApiSchemaKey key)
   at Microsoft.AspNetCore.OpenApi.OpenApiSchemaService.GetOrCreateUnresolvedSchemaAsync(OpenApiDocument document, Type type, IServiceProvider scopedServiceProvider, IOpenApiSchemaTransformer[] schemaTransformers, ApiParameterDescription parameterDescription, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.OpenApi.OpenApiSchemaService.GetOrCreateSchemaAsync(OpenApiDocument document, Type type, IServiceProvider scopedServiceProvider, IOpenApiSchemaTransformer[] schemaTransformers, ApiParameterDescription parameterDescription, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.OpenApi.OpenApiDocumentService.GetJsonRequestBody(OpenApiDocument document, IList`1 supportedRequestFormats, ApiParameterDescription bodyParameter, IServiceProvider scopedServiceProvider, IOpenApiSchemaTransformer[] schemaTransformers, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.OpenApi.OpenApiDocumentService.GetRequestBodyAsync(OpenApiDocument document, ApiDescription description, IServiceProvider scopedServiceProvider, IOpenApiSchemaTransformer[] schemaTransformers, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.OpenApi.OpenApiDocumentService.GetOperationAsync(ApiDescription description, OpenApiDocument document, IServiceProvider scopedServiceProvider, IOpenApiSchemaTransformer[] schemaTransformers, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.OpenApi.OpenApiDocumentService.GetOperationsAsync(IGrouping`2 descriptions, OpenApiDocument document, IServiceProvider scopedServiceProvider, IOpenApiOperationTransformer[] operationTransformers, IOpenApiSchemaTransformer[] schemaTransformers, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.OpenApi.OpenApiDocumentService.GetOpenApiPathsAsync(OpenApiDocument document, IServiceProvider scopedServiceProvider, IOpenApiOperationTransformer[] operationTransformers, IOpenApiSchemaTransformer[] schemaTransformers, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.OpenApi.OpenApiDocumentService.GetOpenApiDocumentAsync(IServiceProvider scopedServiceProvider, HttpRequest httpRequest, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Builder.OpenApiEndpointRouteBuilderExtensions.<>c__DisplayClass0_0.<<MapOpenApi>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Http.Generated.<GeneratedRouteBuilderExtensions_g>F0399B8B15BE443123F4BECEE84B112C1912EE3F44D9DDD7BFA38E371FB917343__GeneratedRouteBuilderExtensionsCore.<>c__DisplayClass2_0.<<MapGet0>g__RequestHandler|5>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)

Steps to reproduce

Update to 2.6.0

Code snippets

OS

Windows

.NET version

9

Library version

2.6.0

[kurnakovv]

Code example

JsonSerializer.Serialize(
    ChatMessage.CreateAssistantMessage("test")
);

It's happens because serializer doesn't support ref https://github.com/openai/openai-dotnet/blob/main/src/Generated/Models/Chat/ChatMessage.cs#L28

And I can't find any solutions to fix it with JsonSerializerOptions

[stephentoub]

@m-nash, @joseharriaga, I realize the OpenAI types are intended to be serialized with the modelreaderwriter stuff. And it'd be really nice if all of the models could be made to "just work" with JsonSerializer, e.g. by putting a [JsonConverter] attribute on each that would delegate to the corresponding IJsonModel implementation.

But until that happens, could we at least put [JsonIgnore] on the newly-added Patch properties? System.Text.Json doesn't yet support ref-returning properties (we need to fix that, but it's unlikely to happen until at least .NET 11), and for anyone that was using JsonSerializer with these OpenAI model types (and getting whatever output they happened to get), the addition of the Patch properties now makes it throw. The presence of one of these model types in a graph that was previously serializable now makes the whole operation fail.

[KrzysztofCwalina]

I think it would be good to collaborate on how IJsonModel-JsonSerializer should interop/coexist. Maybe we can even add some features to JsonSerializer/BCL whose absence pushed us to use IJsonModel in some scenarios where otherwise we would not have to. i.e. I would love to see if we can simplify as opposed to having models with two parallel serialization technologies.

[stephentoub]

I agree. I'm just hoping for a more immediate unblocker. The addition of these Patch properties is breaking folks that have the OpenAI model types in an object graph that's getting serialized. Previously the graph would serialize, now it throws.

[KrzysztofCwalina]

@stephentoub, does JsonModelConverter not work for you? i.e. you can pass it to JsonSerializer and it will (should) intercept all IJsonModel serialization and do the right thing.

[jsquire]

@kurnakovv: The work-around for your snippet is to use ModelReaderWriter directly to serialize:

// MRW serialized to a BinaryData instance.
var binaryData = ModelReaderWriter.Write(ChatMessage.CreateAssistantMessage("test"), ModelReaderWriterOptions.Json);

// If you need the raw JSON string, just call ToString()
var jsonString = binaryData.ToString();

Alternatively, as Krzysztof mentions, you can use JsonModelConverter with JsonSerializer:

JsonSerializerOptions options = new JsonSerializerOptions()
{
    Converters = { new JsonModelConverter() }
};

var jsonString = JsonSerializer.Serialize(ChatMessage.CreateAssistantMessage("test"), options);

For more context, see: ModelReaderWriter samples.

[jsquire]

@rodriciru: Thanks for reaching out and we regret that you're experiencing difficulties. Please be aware that Semantic Kernel is a stand-alone product and not part of the OpenAI client library. The maintainers of the OpenAI client library are not able to offer direct support for other products here. Though Stephen kindly keeps an eye and offers help, the Semantic Kernel owners do not officially offer support from issues in our repository.

I'd suggest opening an issue in the Semantic Kernel repository to ensure that the right folks to assist have visibility.

[stephentoub]

That works if you're in direct control of the serialization / can influence the JsonSerializerOptions, but not if something else is doing so. In those cases, you have one of these objects that just happens to be in the object graph. You don't care about it, you're not particularly interested in the output for that one object, you don't need to roundtrip it, but you do need the graph to serialize. And the addition of these properties has made that whole operation blow up (not just produce undesirable output, but actually throw).

There are issues to be fixed here in JsonSerializer around supporting ref-returning properties, absolutely. But in the meantime, OpenAI can easily address this. It already references System.Text.Json and it'd be trivial to just add [JsonIgnore] to these new ref-returning, [Experimental], EditorBrowsable.Never properties, so that code that was working and now throws keeps working.

[jsquire]

@KrzysztofCwalina: I think there's merit in tagging [JsonIgnore] on Patch while we continue to look for ways to collaborate on improving interop between MRW and JsonSerializer. Any objections?