Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't allow null converters and add Path support to NotSupportedException #32669

Merged

Conversation

steveharter
Copy link
Member

@steveharter steveharter commented Feb 21, 2020

Address feedback and issues from #2259 including

Note after this PR is in, there will be a follow-up PR that adds the correct XML doc regarding exceptions.

Details on JsonConverterFactory.CreateConverter()` should not return null.

After this PR we align with 3.0\3.1 behavior of not supporting null.

When a null value was returned from CreateConverter() the previous code (and in 3.0\3.1) threw ArgumentNullException. This was changed to InvalidOperationException and an exception message was added.

Background: allowing null from CreateConverter() was done in #2259 because before that PR the collection code threw a "nice" exception that contained the parent Type and property name. Before that PR, this could not be done in CreateConverter() because collections were not converters and didn't share that code path. With #2259 collections became converters, and thus needed a way to identify a "not supported" case so code further up the call stack could throw the "nice" exception message. If a NotSupportedException was thrown directly from within CreateConverter() then the message would only contain the name of the Type that is not supported, not the parent Type or the property name.

Per offline discussion, it was decided we would add Path information to NotSupportedException similarly like we do for JsonException. This means throwing a NotSupportedException directly from a converter (or converter factory) would re-throw the exception with Path appended to the original message.

Examples of the previous exceptions:
The type 'System.Int32[,]' on 'System.Text.Json.Serialization.Tests.ExceptionTests+ClassWithInvalidArray.UnsupportedArray' is not supported.
or if a converter threw its own NotSupportedException or the exception occurred in a collection element Type, not the property Type:
The type 'System.Int32[,]' is not supported.

The latter case is especially bad because it provided no way to determine where the unsupported Type exists in the JSON or the object graph.

Examples of exceptions with this PR:
The type 'System.Int32[,]' is not supported. The unsupported member type is located on type 'System.Text.Json.Serialization.Tests.GenericIEnumerableWrapper`1[System.Int32[]]'. Path: $.UnsupportedDictionary."
or this (for read cases):
The type 'System.Int32[,]' is not supported. The unsupported member type is located on type 'System.Text.Json.Serialization.Tests.GenericIEnumerableWrapper`1[System.Int32[]]'. Path: $.UnsupportedDictionary | LineNumber: 0 | BytePositionInLine: 10."

Details on extra boxing avoided

#2259 added an extra box+unbox for value types passed into the root. This PR fixes that. In addition, for structs that have their own converter (and thus are not treated as "POCO structs" with properties) no boxing occurs at all (previously and in 3.0\3.1 a box was done).

In addition, boxing operations were removed that helped reduce non-stack boxing. The result is less stack allocations and boxing\unboxing. There doesn't appear to be any significant CPU performance changes but there is some minor reductions in allocations:

WriteJson<SimpleStructWithProperties> before:

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
SerializeToString 289.5 ns 5.81 ns 6.69 ns 286.0 ns 282.2 ns 303.3 ns 0.0452 - - 288 B
SerializeToUtf8Bytes 270.9 ns 5.81 ns 6.70 ns 268.3 ns 263.8 ns 285.4 ns 0.0419 - - 264 B
SerializeToStream 360.0 ns 6.91 ns 7.96 ns 358.3 ns 349.8 ns 374.7 ns 0.0344 - - 216 B
SerializeObjectProperty 445.1 ns 10.86 ns 12.50 ns 441.3 ns 428.4 ns 473.0 ns 0.0880 - - 552 B
after (32 byte savings)
SerializeToString 286.8 ns 5.68 ns 6.08 ns 287.6 ns 279.3 ns 299.8 ns 0.0403 - - 256 B
SerializeToUtf8Bytes 262.7 ns 6.16 ns 7.09 ns 260.5 ns 255.1 ns 276.4 ns 0.0362 - - 232 B
SerializeToStream 352.7 ns 8.67 ns 9.99 ns 347.7 ns 342.5 ns 374.7 ns 0.0286 - - 184 B
SerializeObjectProperty 450.0 ns 10.13 ns 11.66 ns 445.5 ns 435.3 ns 475.0 ns 0.0868 - - 552 B

ReadJson<SimpleStructWithProperties> before:

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 384.2 ns 7.80 ns 8.98 ns 382.4 ns 373.2 ns 399.0 ns 0.0152 - - 96 B
DeserializeFromUtf8Bytes 336.7 ns 7.76 ns 8.93 ns 336.5 ns 324.4 ns 357.0 ns 0.0142 - - 96 B
DeserializeFromStream 592.5 ns 13.99 ns 16.11 ns 585.8 ns 574.5 ns 629.2 ns 0.0279 - - 176 B
after (32 byte savings)
DeserializeFromString 367.5 ns 6.93 ns 6.48 ns 366.4 ns 358.9 ns 379.2 ns 0.0089 - - 64 B
DeserializeFromUtf8Bytes 314.6 ns 2.63 ns 2.46 ns 314.0 ns 310.6 ns 319.8 ns 0.0091 - - 64 B
DeserializeFromStream 571.8 ns 1.89 ns 1.47 ns 571.6 ns 568.8 ns 574.3 ns 0.0221 - - 144 B

WriteJson<Int32> before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
SerializeToString 163.2 ns 3.48 ns 4.01 ns 162.9 ns 157.7 ns 171.4 ns 0.0331 - - 208 B
SerializeToUtf8Bytes 146.3 ns 4.16 ns 4.79 ns 144.4 ns 140.6 ns 155.6 ns 0.0327 - - 208 B
SerializeToStream 212.5 ns 4.65 ns 5.36 ns 209.8 ns 206.4 ns 222.6 ns 0.0277 - - 176 B
SerializeObjectProperty 257.8 ns 6.20 ns 7.13 ns 254.6 ns 248.8 ns 271.8 ns 0.0314 - - 200 B
after (24 byte savings)
SerializeToString 166.4 ns 3.39 ns 3.90 ns 166.1 ns 160.7 ns 173.9 ns 0.0291 - - 184 B
SerializeToUtf8Bytes 140.4 ns 0.99 ns 0.88 ns 140.3 ns 139.2 ns 142.2 ns 0.0289 - - 184 B
SerializeToStream 209.1 ns 1.68 ns 1.57 ns 209.2 ns 206.6 ns 212.3 ns 0.0236 - - 152 B
SerializeObjectProperty 256.4 ns 2.01 ns 1.88 ns 256.7 ns 252.9 ns 258.9 ns 0.0313 - - 200 B

ReadJson<Int32> before:

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 174.9 ns 3.64 ns 4.20 ns 173.6 ns 170.3 ns 183.3 ns 0.0035 - - 24 B
DeserializeFromUtf8Bytes 129.0 ns 2.48 ns 2.86 ns 129.5 ns 125.7 ns 133.8 ns 0.0037 - - 24 B
DeserializeFromStream 353.4 ns 7.16 ns 8.25 ns 355.9 ns 342.3 ns 365.9 ns 0.0151 - - 96 B
after (24 byte savings)
DeserializeFromString 171.6 ns 3.45 ns 3.23 ns 170.5 ns 168.4 ns 178.1 ns - - - -
DeserializeFromUtf8Bytes 121.4 ns 0.94 ns 0.88 ns 121.2 ns 120.0 ns 123.4 ns - - - -
DeserializeFromStream 343.6 ns 8.43 ns 9.37 ns 339.6 ns 332.3 ns 366.1 ns 0.0107 - - 72 B

Here are all benchmarks with >=3% threshold. No noted regression with latest changes. The benchmarks were run twice; below is the slightly less favorable results:

summary:
better: 19, geomean: 1.048
total diff: 19

No Slower results for the provided threshold = 2% and noise filter = 0.3ns.

| Faster                                                                           | base/diff | Base Median (ns) | Diff Median (ns) | Modality|
| -------------------------------------------------------------------------------- | ---------:| ----------------:| ----------------:| -------- |
| System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToStream     |      1.07 |           485.93 |           453.19 |         |
| System.Text.Json.Serialization.Tests.ReadJson<SimpleStructWithProperties>.Deseri |      1.07 |           336.51 |           313.97 |         |
| System.Text.Json.Serialization.Tests.ReadJson<Int32>.DeserializeFromUtf8Bytes    |      1.07 |           129.50 |           121.18 |         |
| System.Text.Json.Serialization.Tests.ReadJson<MyEventsListerViewModel>.Deseriali |      1.07 |        380413.09 |        357132.57 |         |
| System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>.Serializ |      1.06 |        417522.37 |        393750.16 |         |
| System.Text.Json.Serialization.Tests.WriteJson<Location>.SerializeToUtf8Bytes    |      1.06 |           878.60 |           832.25 |         |
| System.Text.Json.Serialization.Tests.WriteJson<ArrayList>.SerializeObjectPropert |      1.05 |         11980.63 |         11373.18 |         |
| System.Text.Json.Serialization.Tests.ReadJson<Location>.DeserializeFromUtf8Bytes |      1.05 |          1172.51 |          1113.40 |         |
| System.Text.Json.Serialization.Tests.ReadJson<BinaryData>.DeserializeFromUtf8Byt |      1.05 |           520.51 |           494.50 |         |
| System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>.Seria |      1.05 |         11909.04 |         11338.84 |         |
| System.Text.Json.Serialization.Tests.ReadJson<SimpleStructWithProperties>.Deseri |      1.04 |           382.43 |           366.40 |         |
| System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToUtf8Bytes  |      1.04 |           422.66 |           405.37 |         |
| System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeObjectProper |      1.04 |           741.50 |           711.32 |         |
| System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.Serial |      1.04 |           719.10 |           694.73 |         |
| System.Text.Json.Serialization.Tests.ReadJson<HashSet<String>>.DeserializeFromUt |      1.03 |         13756.11 |         13303.09 |         |
| System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.Serial |      1.03 |           751.40 |           729.69 | several?|
| System.Text.Json.Serialization.Tests.ReadJson<MyEventsListerViewModel>.Deseriali |      1.03 |        309202.87 |        300340.31 |         |
| System.Text.Json.Serialization.Tests.WriteJson<ArrayList>.SerializeToStream      |      1.03 |         11177.42 |         10880.01 |         |
| System.Text.Json.Serialization.Tests.ReadJson<LargeStructWithProperties>.Deseria |      1.03 |          1144.67 |          1114.37 |         |

@steveharter steveharter added NO-SQUASH The PR should not be squashed area-System.Text.Json tenet-performance Performance related issue labels Feb 21, 2020
@steveharter steveharter added this to the 5.0 milestone Feb 21, 2020
@steveharter steveharter self-assigned this Feb 21, 2020
@steveharter steveharter force-pushed the NullConvertersAndNotSupportedException branch 3 times, most recently from 78b056f to 3d21aca Compare February 26, 2020 20:41
@steveharter
Copy link
Member Author

Some value-typed benchmarks added in dotnet/performance#1211

Copy link
Contributor

@layomia layomia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feedback from offline review session (me, @steveharter, @jozkee), primarily around removing the delayed initialization for JsonClassInfo has been addressed.

Pls take a pass at filling code coverage for new code.

Options);

ClassType = converter.ClassType;
PolicyProperty = CreatePolicyProperty(Type, runtimeType, converter, Options);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you consider caching this lazily (like ElementClassInfo) as discussed, or is it guaranteed to be used by all ClassTypes anywhere they occur in the type graph? Maybe we could measure if/how much allocations could be saved.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This didn't change with this PR (PropertyPolicy was always created).

Currently PolicyProperty is always accessed for a given property, so deferring the construction will cause an extra if check at runtime instead of doing the creation up-front (which is cached).

Since there's a reference back to the parent, caching would only help for properties on a given type if they're the same Type. PropertyPolicy could pehaps be improved here, but not in scope for this PR anyway.

@steveharter steveharter force-pushed the NullConvertersAndNotSupportedException branch 2 times, most recently from f78ca13 to faa61a7 Compare March 4, 2020 20:08
Assert.Contains(typeof(int[,]).ToString(), ex.ToString());
Assert.Contains(typeof(ChildPocoWithNoConverterAndInvalidProperty).ToString(), ex.ToString());
Assert.Contains("Path: $.InvalidProperty | LineNumber: 0 | BytePositionInLine: 20.", ex.ToString());
Assert.Equal(2, ex.ToString().Split("Path:", StringSplitOptions.None).Length);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test failure - Serialization\CustomConverterTests.Callback.cs(139,58): error CS1503: (NETCORE_ENGINEERING_TELEMETRY=Build) Argument 2: cannot convert from 'System.StringSplitOptions' to 'char'

Here and below.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like that is supported in ns2.1 but not 2.0. Need to pass array instead of string.

Debug.Assert(writer != null);

if (value == null)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before we were not calling Initialize when the root was null; now we do and that causes a change in behavior for unsupported root types.

[Fact]
public static void QuickTest()
{
    int[,] arr = null;
    string json = JsonSerializer.Serialize(arr);
    Console.WriteLine(json);
}

on master, the previous example returns null. with these changes we will throw.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this can be fixed if you just keep this condition.

Copy link
Member Author

@steveharter steveharter Mar 6, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we want to support converters writing their own null we must ask the converter if they want to handle null. For example a custom String converter may want to write null as "". So we just can't write null here.

I'll add a test to assert this and track for any potential breaking changes.

Copy link
Member

@jozkee jozkee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than the pending discussion of moving the logic from the NotSupportingException catch to the helper #32669 (comment), and the change of behavior pointed on #32669 (comment); this looks good to me.

@@ -216,16 +211,86 @@ public static void AddExceptionInformation(in ReadStack state, in Utf8JsonReader
}
}

public static NotSupportedException GetNotSupportedException(in ReadStack state, in Utf8JsonReader reader, NotSupportedException ex)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
public static NotSupportedException GetNotSupportedException(in ReadStack state, in Utf8JsonReader reader, NotSupportedException ex)
[DoesNotReturn]
[MethodImpl(MethodImplOptions.NoInlining)]
public static void GetNotSupportedException(in ReadStack state, in Utf8JsonReader reader, NotSupportedException ex)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned earlier, we need to use plain "throw" statement to re-throw the current exception. We don't want to throw the same exception instance since that will create a new call stack unnecessarily. So this method can't be [DoesNotReturn]

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: added check for "Path" to caller and changed method to add [DoesNotReturn]. We still keep the original "throw" in the caller's catch statement in order to preserve the original call stack.

@steveharter steveharter force-pushed the NullConvertersAndNotSupportedException branch from 4e34106 to 417254e Compare March 6, 2020 16:32
@steveharter
Copy link
Member Author

Unrelated and previously hit CI failures in:
System.Diagnostics.PerformanceCounter.Tests
System.Net.Http.Functional.Tests.HttpClientTest

@steveharter steveharter merged commit d59f143 into dotnet:master Mar 6, 2020
@steveharter steveharter deleted the NullConvertersAndNotSupportedException branch March 6, 2020 22:46

return WriteAsyncCore(utf8Json, value, inputType, options, cancellationToken);
return WriteAsyncCore<object>(utf8Json, value!, inputType, options, cancellationToken);
Copy link
Member

@ahsonkhan ahsonkhan Mar 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is incorrect usage of nullability annotation. If value can be null (and looking at the null checks happening later, it can), the TValue value parameter on WriteAsyncCore should be correctly annotated to be nullable.

Avoid using ! to circumvent null warnings unless absolutely necessary, and instead annotate the implementation to reflect nullability accurately.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. This should be called with WriteAsyncCore<object?>.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a helpful note for reviewers: Make sure to pay close attention to uses of ! in the code :)

@layomia layomia added breaking-change Issue or PR that represents a breaking API or functional change over a prerelease. needs-breaking-change-doc-created Breaking changes need an issue opened with https://github.com/dotnet/docs/issues/new?template=dotnet labels Nov 5, 2020
@layomia
Copy link
Contributor

layomia commented Nov 5, 2020

This PR introduced a breaking change - #35422. See that issue for more info. I'll file a breaking change doc.

@layomia
Copy link
Contributor

layomia commented Nov 7, 2020

Filed a breaking change doc - dotnet/docs#21376.

@layomia
Copy link
Contributor

layomia commented Nov 9, 2020

@layomia layomia added breaking-change Issue or PR that represents a breaking API or functional change over a prerelease. and removed breaking-change Issue or PR that represents a breaking API or functional change over a prerelease. needs-breaking-change-doc-created Breaking changes need an issue opened with https://github.com/dotnet/docs/issues/new?template=dotnet labels Nov 9, 2020
@ghost ghost added the needs-breaking-change-doc-created Breaking changes need an issue opened with https://github.com/dotnet/docs/issues/new?template=dotnet label Nov 9, 2020
@layomia layomia removed the needs-breaking-change-doc-created Breaking changes need an issue opened with https://github.com/dotnet/docs/issues/new?template=dotnet label Nov 9, 2020
@ericstj
Copy link
Member

ericstj commented Nov 9, 2020

dotnet/docs#20823 was meant to cover #528 (comment) not this, but if they surface in the same API / behavior then agree we don't need two.

@layomia
Copy link
Contributor

layomia commented Nov 10, 2020

@ericstj on a second look, the breaking change was made in #528 but refactored in this PR. So this PR wasn't the breaking change.

@layomia layomia removed the breaking-change Issue or PR that represents a breaking API or functional change over a prerelease. label Nov 10, 2020
@ghost ghost locked as resolved and limited conversation to collaborators Dec 10, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-System.Text.Json NO-SQUASH The PR should not be squashed tenet-performance Performance related issue
Projects
None yet
6 participants