System.Text.Json.JsonException thrown when deserializing asynchronously to nullable types

[emonino]

Description

System.Text.Json.JsonSerializer.DeserializeAsync fails with exception
System.Text.Json.JsonException: 'The JSON value could not be converted to STJDeserializationException.DeserializeDto. Path: $[57] | LineNumber: 0 | BytePositionInLine: 19343.'

Deserializing the same data into the same object works when using the synchronous deserialization method, JsonSerializer.Deserialize.

Asynchronous deserialization, JsonSerializer.DeserializeAsync, works if the object T does not contain nullable parent types.

Reproduction Steps

I created a GitHub repo with the minimum classes to fully reproduce the behavior I am seeing - https://github.com/emonino/STJDeserializationException.

When we try to deserialize data into a simple object with a nullable parent type, DeserializeDto, deserialization fails with a System.Text.Json.JsonException. For example:
System.Text.Json.JsonException: 'The JSON value could not be converted to STJDeserializationException.DeserializeDto. Path: $[57] | LineNumber: 0 | BytePositionInLine: 19343.'

The exact method that throws is await JsonSerializer.DeserializeAsync<List<DeserializeDto>>(stream, options).

However, if we deserialize the exact same data into the same object using the synchronous deserialization method, deserialization works as expected. For example, the below method works:
JsonSerializer.Deserialize<List<DeserializeDto>>(dataStr)

I can also successfully use System.Text.Json if I modify the object I am trying to deserialize into to not include a nullable parent object as seen in NonNullable.DeserializeDto. Note that the only difference between DeserializeDto and NonNullable.DeserializeDto is that the Start parent object is nullable in DeserializeDto.

In other words, this code also works as expected:
await JsonSerializer.DeserializeAsync<List<STJDeserializationException.NonNullable.DeserializeDto>>(stream, options)

Expected behavior

Asynchronous deserialization to nullable objects should succeed

Actual behavior

Asynchronous deserialization fails with exception similar to
System.Text.Json.JsonException: 'The JSON value could not be converted to STJDeserializationException.DeserializeDto. Path: $[57] | LineNumber: 0 | BytePositionInLine: 19343.'

Regression?

We first noticed this error after upgrading from System.Text.Json 7.0.4 to 8.0.2. We have since upgraded to System.Text.Json 9.0.0. The exception is thrown less often now, but can still occur for large data sets.

Known Workarounds

Using JsonSerializer.Deserialize or making parent objects non-nullable.

Configuration

No response

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

[eiriktsarpalis]

I can reproduce this deterministically in .NET 8. Here's a minimal repro:

using System.Text;
using System.Text.Json;

string json = """
    [
      {
        "Start": {
          "Timestamp": "2024-12-05T00:00:00-05:00",
          "Value": 100.8728,
          "Questionable": null,
          "Substituted": null,
          "Annotated": null,
          "SystemStateCode": null,
          "DigitalStateName": null
        }
      },
      {
        "Start": {
          "Value": 100.13499
        }
      }
    ]
    """;

JsonSerializerOptions options = new() { DefaultBufferSize = 1 };
JsonSerializer.Deserialize<List<DeserializeDto>>(json, options); // success

using MemoryStream stream = new(Encoding.UTF8.GetBytes(json));
await JsonSerializer.DeserializeAsync<List<DeserializeDto>>(stream, options);
// System.Text.Json.JsonException: 'The JSON value could not be converted to DeserializeDto.
// Path: $[1] | LineNumber: 11 | BytePositionInLine: 3.' 

public class DeserializeDto
{
    public Start? Start { get; set; }
}

public struct Start // changing to class resolves the issue
{
    public float? Value { get; set; }
}

Although I didn't have any luck reproducing the issue in .NET 9. This is likely a bug with the state machine STJ implements for async deserialization. Although I doubt this meets the bar for servicing in .NET 8 or .NET 9, we should consider examining this reproduction to make sure that the current version of STJ isn't susceptible to other instances of the same bug.

[eiriktsarpalis]

I should add that these types of bugs are actually fairly deterministic once you fix the inputs and configuration. If you could provide us with a minimal reproduction like the above that demonstrates the same crash of .NET 9, that would be very helpful.

[emonino]

I apologize if my initial post was not clear. I was able to deterministically reproduce and included the models and code in a repo here - https://github.com/emonino/STJDeserializationException.

The bug is definitely less prevalent in .NET 9, but I can still hit it with a large enough data input. My repo uses .NET 9.

[eiriktsarpalis]

For whatever reason I couldn't reproduce the issue with your example, however after a few random runs I was able to condense a minimal reproduction that works with .NET 9:

using System.Text;
using System.Text.Json;

string json = """
    [{"Start":{"Padding1":"x","Value":100.86587,"Padding2":null}},{"Start":{"Padding1":"xxxxx","Value":100.08407,"Padding2":null}}]
    """;

// the following works
JsonSerializer.Deserialize<List<DeserializeDto>>(json);

// but this fails with System.Text.Json.JsonException: The JSON value could not be converted to DeserializeDto.
JsonSerializerOptions options = new() { DefaultBufferSize = 1 };
using MemoryStream stream = new(Encoding.UTF8.GetBytes(json));
await JsonSerializer.DeserializeAsync<List<DeserializeDto>>(stream, options);

public class DeserializeDto
{
    public Start? Start { get; set; }
}

public struct Start // Changing to a class removes the error
{
    public float? Value { get; set; }
}