Nullable JsonDocument property will deserialized to JsonValueKind.Null instead of null

[aco-mreble]

Description

After upgrading from dotnet 8 to dotnet 9 our applications don't work anymore. This is because we are using HttpClient generation with NSwag and unknown objects are serialized to JsonDocument? properties. This is a breaking change. I have tried a little bit with the new RespectNullableAnnotations and DefaultIgnoreCondition but this has not worked either.

Reproduction Steps

using System.Text.Json;

//lang=json
var json = """
           {
             "Properties": null
           }
           """;
var dto = JsonSerializer.Deserialize<MyTestClass>(json);

//Returns true in dotnet 8 and false in dotnet 9
Console.WriteLine(dto!.Properties is null);

//Returns false in dotnet 8 and true in dotnet 9
Console.WriteLine(dto!.Properties is JsonDocument { RootElement.ValueKind: JsonValueKind.Null });

public class MyTestClass()
{
	public JsonDocument? Properties { get; set; }
}

Expected behavior

Properties property in MyTestClass should be null

Actual behavior

Properties property in MyTestClass is JsonDocument with RootElement.ValueKind = JsonValueKind.Null

Regression?

It worked on dotnet 8

Known Workarounds

public class JsonDocumentNullableConverter : JsonConverter<JsonDocument?>
{
	public override JsonDocument? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
	{
		if (reader.TokenType == JsonTokenType.Null)
			return null;
		return JsonDocument.ParseValue(ref reader);
	}

	public override void Write(Utf8JsonWriter writer, JsonDocument? value, JsonSerializerOptions options)
	{
		if (value is null)
		{
			writer.WriteNullValue();
			return;
		}

		value.WriteTo(writer);
	}
}

Configuration

dotnet 9.0.100 SDK

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

[eiriktsarpalis]

The change was made intentionally in #97596 to address an inconsistency between JsonDocument.Parse which correcty maps null to JsonValueKind.Null and the built-in converter for JsonDocument.

[jeffhandley]

Thank you for reporting that you encountered this, @aco-mreble. I've referenced the code sample you provided in the breaking change documentation issue: dotnet/docs#43869.