Implement JsonStringEnumConverter<TEnum> #87149

eiriktsarpalis · 2023-06-05T21:46:31Z

dotnet-issue-labeler · 2023-06-05T21:46:36Z

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

ghost · 2023-06-05T21:46:42Z

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

Fix #79311.

Author:	eiriktsarpalis
Assignees:	-
Labels:	`area-System.Text.Json`, `new-api-needs-documentation`
Milestone:	-

src/libraries/System.Text.Json/tests/Common/PropertyVisibilityTests.cs

eiriktsarpalis · 2023-06-05T21:56:16Z

/azp run runtime-extra-platforms

azure-pipelines · 2023-06-05T21:56:31Z

Azure Pipelines successfully started running 1 pipeline(s).

eerhardt · 2023-06-05T23:20:22Z

FYI - @stephentoub: will this will allow you to undo the workarounds in https://github.com/microsoft/semantic-kernel/pull/1284/files#diff-b2c336c6d6a74aa2671ea2afca585cdb1bf6a32ba4fdd0ddbfbaef516dd6686bR53?

src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonStringEnumConverter.cs

src/libraries/apicompat/ApiCompatBaseline.NetCoreAppLatestStable.xml

src/libraries/System.Text.Json/gen/Helpers/KnownTypeSymbols.cs

eerhardt · 2023-06-05T23:25:28Z

src/libraries/System.Text.Json/ref/System.Text.Json.cs

@@ -1186,6 +1187,7 @@ public static partial class JsonMetadataServices
        public static System.Text.Json.Serialization.JsonConverter<T> GetEnumConverter<T>(System.Text.Json.JsonSerializerOptions options) where T : struct { throw null; }
        public static System.Text.Json.Serialization.JsonConverter<T?> GetNullableConverter<T>(System.Text.Json.JsonSerializerOptions options) where T : struct { throw null; }
        public static System.Text.Json.Serialization.JsonConverter<T?> GetNullableConverter<T>(System.Text.Json.Serialization.Metadata.JsonTypeInfo<T> underlyingTypeInfo) where T : struct { throw null; }
+        public static System.Text.Json.Serialization.JsonConverter<T> GetStringEnumConverter<T>(System.Text.Json.Serialization.JsonStringEnumConverter converterFactory, System.Text.Json.JsonSerializerOptions options) where T : struct { throw null; }


Why doesn't this have where T : struct, Enum? That is what is in the approved API.

It would appear that the underlying converter doesn't have a : Enum constraint. Given that this matches the existing GetEnumConverter method, we should probably just update the proposal.

Hmm, actually this appears to be a case where the reference file is diverging from the implementation. Both the GetEnumConverter method shipped in .NET 6 and GetStringEnumConverter have an : Enum constraint but this isn't reflected in the ref file. Surprised this was never caught by our tooling.

cc @carlossanlop @ViktorHofer is it safe to update the ref file to match the implementation?

@eiriktsarpalis the code for APICompat which is used to compare the contract vs the implementation is located in the sdk repo: https://github.com/dotnet/sdk/tree/main/src/ApiCompat

Please file an issue so that we can track this bug / feature request (unsure yet what it is).

@carlossanlop @ViktorHofer is it safe to update the ref file to match the implementation?

Yes, that should be fine.

Dumb question, but did you make sure to autogenerate the ref file @eiriktsarpalis ? https://github.com/dotnet/runtime/blob/main/docs/coding-guidelines/updating-ref-source.md

I ask because we're not supposed to manually edit them. Unless of course, the tool that autogenerates it has a bug, which could be the case here, but it's not clear from the comments (at least for me) if the ref was autogenerated or not.

eerhardt · 2023-06-05T23:30:02Z

src/libraries/System.Text.Json/ref/System.Text.Json.cs

    public partial class JsonStringEnumConverter : System.Text.Json.Serialization.JsonConverterFactory
    {
+        [System.Diagnostics.CodeAnalysis.RequiresDynamicCodeAttribute("JsonStringEnumConverter cannot be statically analyzed and requires runtime code generation. Native AOT users should only use this type indirectly via JsonConverterAttribute annotations.")]


We should test this out, but I think this is still going to give warnings when being used in AOT apps. JsonConverterAttribute is annotated with [DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicParameterlessConstructor)]:

runtime/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Attributes/JsonConverterAttribute.cs

Line 25 in 2bf8f1a

public JsonConverterAttribute([DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicParameterlessConstructor)] Type converterType)

which means it is going to reflect against the parameterless constructor of the Type being specified. Here we are marking that constructor as "RequiresDynamicCode". The AOT compiler is going to warn that there is reflection being done against a constructor marked as "unsafe for AOT".

cc @MichalStrehovsky @vitek-karas

I think you're right. I'm not sure we would be able to reasonably change either the attribute or the converter annotation.

The only safe alternative I can think of is we expose a separate JsonStringEnumConverter<T> converter that isn't a factory and have the source generator emit a diagnostic prompting users to change their annotations to the new type.

FWIW I tested this code:

using System; using System.Reflection; using System.Text.Json; using System.Text.Json.Serialization; Console.WriteLine(JsonSerializer.Serialize(new MyPoco(), MyContext.Default.MyPoco)); public class MyPoco { [JsonConverter(typeof(JsonStringEnumConverter))] public BindingFlags Flags { get; set; } } [JsonSerializable(typeof(MyPoco))] public partial class MyContext : JsonSerializerContext { }

and it appears to not be warning on the attribute annotation location. The concern though is that future revisions of the AOT compiler could be warning there.

@vitek-karas pointed out that the above not warning is accidental because the attribute components get trimmed away. Once I include the following line:

Console.WriteLine(new JsonConverterAttribute(typeof(int)).ConverterType);

The compiler immediately warns. So it seems to me that using JsonStringEnumConverter (or any JsonConverterFactory for that matter) is simply not viable in source gen.

@vitek-karas pointed out that the above not warning is accidental because the attribute components get trimmed away.

Is there an issue logged for this? That seems like an analysis hole to me.

@vitek-karas pointed out that the above not warning is accidental because the attribute components get trimmed away.

Is there an issue logged for this? That seems like an analysis hole to me.

Vitek can correct me if I'm wrong for this case since he looked at it, but in general we don't analyze code that wasn't needed. So if the warning is for some piece of code that was trimmed, we'd not look at it.

This is NOT a bug - there is now reflection access to the Flags property in the repro, so the AOT compiler doesn't keep any metadata about the property (or even the fact that there is a property, it only keeps the accessors), that includes not having any attributes for it. And since it's not going to keep the attributes, it doesn't run any validation on it and thus doesn't warn.

Note that if this sample is modified to use RequiresUnreferencedCode (so that analyzer/linker will actually react to it), then both analyzer and linker would report the warning. Because they're not as precise in "trimming" as AOT compiler is.

So while this doesn't warn, I would argue that we don't want to rely on that behavior, since even defining what the behavior looks like is pretty difficult - and implementing it in illink and especially in the analyzer might be pretty tricky.

So the "trick" here is that no preserved code accesses the JsonConverterAttribute.ConverterType property? That allows the AOT compiler to not warn?

Does the ctor that is mark as RequiresDynamicCode still get preserved? Even though no one accesses the ConverterType property?

In the trimmer and AOT, if the .ctor is preserved we will report the RDC warning on it - always. The repro above actually does warn, but from the source generated code which calls the .ctor directly.

Analyzer currently doesn't correctly implement this behavior (combination of DAM and RDC) and will not report a warning (it will for DAM and RUC - don't ask ;-))

src/libraries/System.Text.Json/gen/JsonSourceGenerator.Emitter.cs

eiriktsarpalis · 2023-06-06T19:30:49Z

@layomia @eerhardt @stephentoub I just updated the PR to include the most recently approved API shape, PTAL.

src/libraries/System.Text.Json/gen/Resources/Strings.resx

src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonStringEnumConverter.cs

src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/EnumConverterTests.cs

eiriktsarpalis · 2023-06-06T20:31:45Z

src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonStringEnumConverter.cs

+    /// Reading is case insensitive, writing can be customized via a <see cref="JsonNamingPolicy" />.
+    /// </remarks>
+    /// <typeparam name="TEnum">The enum type that this converter targets.</typeparam>
+    public class JsonStringEnumConverter<TEnum> : JsonConverterFactory


@eerhardt This was brought up in API review, do we need to annotate TEnum with DynamicallyAccessedMembers in case the trimmer removes enum values?

Trimming never removes enum values. It cannot be done safely because ToString on enums relies on them - as soon as we see the enum getting boxed or it's typeof obtained, we keep all the fields. So this is fine without annotation - I assume there are no static analysis warnings either.

src/libraries/System.Text.Json/ref/System.Text.Json.cs

src/libraries/apicompat/ApiCompatBaseline.NetCoreAppLatestStable.xml

eerhardt · 2023-06-07T14:23:00Z

src/libraries/System.Text.Json/gen/JsonSourceGenerator.DiagnosticDescriptors.cs

@@ -72,6 +72,14 @@ internal static class DiagnosticDescriptors
                category: JsonConstants.SystemTextJsonSourceGenerationName,
                defaultSeverity: DiagnosticSeverity.Warning,
                isEnabledByDefault: true);
+
+            public static DiagnosticDescriptor JsonStringEnumConverterNotSupportedInAot { get; } = new DiagnosticDescriptor(


Do we need this new diagnostic? Isn't the RequiresDynamicCode attribute on JsonStringEnumConverter enough to devs that JsonStringEnumConverter is not supported in native AOT?

It was added in reaction to #87149 (comment) not triggering at all either at design time or at compile time. If we can get to the point that this always warns then I could remove that diagnostic.

That comment will not be a problem in publish - both trimmer and AOT compiler will correctly warn if necessary.
The only "hole" is in the analyzer - which will not report this warning currently.
That said - if the specific diagnostic will always provide better UX than the more general RDC warning (even if the analyzer did report it) - so it might be worth keeping this just for the better UX.

Downside is that if we fix the analyzer this might produce two warnings.

Can we deprecate the diagnostic in the future if the analyzer does get fixed?

Probably - we may need to figure out some versioning stuff, but it shouldn't be too hard.

Co-authored-by: Vitek Karas <10670590+vitek-karas@users.noreply.github.com>

eiriktsarpalis · 2023-06-07T18:16:28Z

I'll need to open a separate PR since github won't let me reopen this one after I've forced pushed changes.

Fix NativeAOT support for JsonStringEnumConverter.

efd71ca

dotnet-issue-labeler bot added area-System.Text.Json new-api-needs-documentation labels Jun 5, 2023

eiriktsarpalis requested review from layomia and krwq June 5, 2023 21:46

ghost assigned eiriktsarpalis Jun 5, 2023

eiriktsarpalis requested review from eerhardt and tarekgh June 5, 2023 21:46

eiriktsarpalis added this to the 8.0.0 milestone Jun 5, 2023

eiriktsarpalis commented Jun 5, 2023

View reviewed changes

src/libraries/System.Text.Json/tests/Common/PropertyVisibilityTests.cs Show resolved Hide resolved

Fix merge issue.

c4fb01f

layomia reviewed Jun 5, 2023

View reviewed changes

src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonStringEnumConverter.cs Outdated Show resolved Hide resolved

src/libraries/apicompat/ApiCompatBaseline.NetCoreAppLatestStable.xml Outdated Show resolved Hide resolved

eerhardt reviewed Jun 5, 2023

View reviewed changes

src/libraries/System.Text.Json/gen/JsonSourceGenerator.Emitter.cs Outdated Show resolved Hide resolved

build-analysis bot mentioned this pull request Jun 6, 2023

Tracking issue for CI build timeouts #76454

Closed

eiriktsarpalis mentioned this pull request Jun 6, 2023

Provide trim/AOT-safe mechanism to (de)serialize enums from strings with JSON source-gen #79311

Closed

Implement JsonStringEnumConverter<TEnum>.

1e45d00

eiriktsarpalis changed the title ~~Fix NativeAOT support for JsonStringEnumConverter.~~ Implement JsonStringEnumConverter<TEnum> Jun 6, 2023

eiriktsarpalis added 2 commits June 6, 2023 20:25

Remove leftover comments.

dfe0de6

Revert unnecessary suppression.

ca10408

eiriktsarpalis requested review from layomia and eerhardt June 6, 2023 19:30

stephentoub reviewed Jun 6, 2023

View reviewed changes

src/libraries/System.Text.Json/gen/Resources/Strings.resx Outdated Show resolved Hide resolved

stephentoub reviewed Jun 6, 2023

View reviewed changes

src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonStringEnumConverter.cs Show resolved Hide resolved

stephentoub reviewed Jun 6, 2023

View reviewed changes

src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/EnumConverterTests.cs Show resolved Hide resolved

eiriktsarpalis added 2 commits June 6, 2023 21:21

Update compatibility suppressions.

543b9e4

Add missing XML docs

d6ea143

eiriktsarpalis commented Jun 6, 2023

View reviewed changes

Fix capitalization.

37c6b6f

vitek-karas reviewed Jun 7, 2023

View reviewed changes

src/libraries/System.Text.Json/ref/System.Text.Json.cs Outdated Show resolved Hide resolved

eiriktsarpalis commented Jun 7, 2023

View reviewed changes

src/libraries/apicompat/ApiCompatBaseline.NetCoreAppLatestStable.xml Show resolved Hide resolved

eerhardt reviewed Jun 7, 2023

View reviewed changes

eiriktsarpalis and others added 2 commits June 7, 2023 18:20

add local project compatibility suppressions

aede061

Update src/libraries/System.Text.Json/ref/System.Text.Json.cs

a6a0475

Co-authored-by: Vitek Karas <10670590+vitek-karas@users.noreply.github.com>

eiriktsarpalis closed this Jun 7, 2023

eiriktsarpalis mentioned this pull request Jun 7, 2023

Implement JsonStringEnumConverter<TEnum> #87224

Merged

This was referenced Jun 8, 2023

[mono] Tests failing with Failed to find compatible device: arm64-v8a #87267

Closed

MsQuicPlatformDetectionTests failures in CI #87275

Closed

ghost locked as resolved and limited conversation to collaborators Jul 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement JsonStringEnumConverter<TEnum> #87149

Implement JsonStringEnumConverter<TEnum> #87149

eiriktsarpalis commented Jun 5, 2023 •

edited

dotnet-issue-labeler bot commented Jun 5, 2023

ghost commented Jun 5, 2023

eiriktsarpalis commented Jun 5, 2023

azure-pipelines bot commented Jun 5, 2023

eerhardt commented Jun 5, 2023

eerhardt Jun 5, 2023

eiriktsarpalis Jun 5, 2023

eiriktsarpalis Jun 6, 2023 •

edited

ViktorHofer Jun 6, 2023

carlossanlop Jun 7, 2023 •

edited

eerhardt Jun 5, 2023

eiriktsarpalis Jun 6, 2023 •

edited

eiriktsarpalis Jun 6, 2023

eiriktsarpalis Jun 6, 2023

eerhardt Jun 6, 2023

MichalStrehovsky Jun 7, 2023

vitek-karas Jun 7, 2023

eerhardt Jun 7, 2023

vitek-karas Jun 7, 2023

eiriktsarpalis commented Jun 6, 2023

eiriktsarpalis Jun 6, 2023

MichalStrehovsky Jun 7, 2023

eerhardt Jun 7, 2023

eiriktsarpalis Jun 7, 2023

vitek-karas Jun 7, 2023

eiriktsarpalis Jun 7, 2023

vitek-karas Jun 7, 2023

eiriktsarpalis commented Jun 7, 2023

Implement JsonStringEnumConverter<TEnum> #87149

Implement JsonStringEnumConverter<TEnum> #87149

Conversation

eiriktsarpalis commented Jun 5, 2023 • edited

dotnet-issue-labeler bot commented Jun 5, 2023

ghost commented Jun 5, 2023

eiriktsarpalis commented Jun 5, 2023

azure-pipelines bot commented Jun 5, 2023

eerhardt commented Jun 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eiriktsarpalis Jun 6, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlossanlop Jun 7, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eiriktsarpalis Jun 6, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eiriktsarpalis commented Jun 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eiriktsarpalis commented Jun 7, 2023

eiriktsarpalis commented Jun 5, 2023 •

edited

eiriktsarpalis Jun 6, 2023 •

edited

carlossanlop Jun 7, 2023 •

edited

eiriktsarpalis Jun 6, 2023 •

edited