Text.Json fails to serialize schema with properties of derived overloaded classes

[ptr727]

Description

Switching (trying) a project from Newtonsoft.Json to Text.Json, and encountered problem with some of the schemas failing to serialize.

The project uses versioned and inheriting record classes for schema management, with deserialization intantiating the correct version of the class, e.g. Schema2 : Schema3 : Schema1, and properties of the classes follows a similar inheritance history.

After several rounds of troubleshooting I have a reproducible case when a property in a class is overloaded in a derived class.

I reviewed the Newtonsoft.Json to Text.Json docs and did not see anything I am using that is not supported?

Example code at dotnetfiddle.

Reproduction Steps

using System;
using System.Collections.Generic;
using System.ComponentModel.DataAnnotations;
using System.Reflection;
using System.Text.Json;
using System.Text.Json.Serialization;
using System.Text.Json.Serialization.Metadata;
					
public class Program
{
	public static void Main()
	{
		// Create test object and populate it with some data
		var parent = new Parent2()
		{
			Child = new Child2()
			{
				List = [
					new() { Foo = "foo1", Bar = "bar1" },
					new() { Foo = "foo2", Bar = "bar2" }
					]
			}
		};

		// Convert to and from JSON using Newtonsoft
		var json1 = Newtonsoft.Json.JsonConvert.SerializeObject(parent, JsonSerializerSettingsWrite);
		Console.WriteLine("Newtonsoft : Schema to JSON:");
		Console.WriteLine(json1);
		var schema1 = Newtonsoft.Json.JsonConvert.DeserializeObject<Parent2>(json1, JsonSerializerSettingsRead);
		Console.WriteLine("Newtonsoft : JSON to Schema:");
		Console.WriteLine(schema1);

		// Convert to and from JSON using Text.Json
		var json2 = JsonSerializer.Serialize(parent, JsonSerializerOptionsWrite);
		Console.WriteLine("Text.Json : Schema to JSON:");
		Console.WriteLine(json2);
		var schema2 = JsonSerializer.Deserialize<Parent2>(json2, JsonSerializerOptionsRead);
		Console.WriteLine("Text.Json : JSON to Schema:");
		Console.WriteLine(schema2);

		// Convert to and from JSON using Text.Json (without special settings)
		json2 = JsonSerializer.Serialize(parent);
		Console.WriteLine("Text.Json : Schema to JSON (no settings):");
		Console.WriteLine(json2);
		schema2 = JsonSerializer.Deserialize<Parent2>(json2);
		Console.WriteLine("Text.Json : JSON to Schema (no settings):");
		Console.WriteLine(schema2);
}

	private static readonly Newtonsoft.Json.JsonSerializerSettings JsonSerializerSettingsRead = new Newtonsoft.Json.JsonSerializerSettings
	{
		ObjectCreationHandling = Newtonsoft.Json.ObjectCreationHandling.Reuse
	};

	private static readonly Newtonsoft.Json.JsonSerializerSettings JsonSerializerSettingsWrite = new Newtonsoft.Json.JsonSerializerSettings
	{
		Formatting = Newtonsoft.Json.Formatting.Indented,
		StringEscapeHandling = Newtonsoft.Json.StringEscapeHandling.EscapeNonAscii,
		NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore,
		ContractResolver = new ExcludeObsoletePropertiesResolver()
	};

	private static readonly JsonSerializerOptions JsonSerializerOptionsRead = new JsonSerializerOptions
	{
		PreferredObjectCreationHandling = JsonObjectCreationHandling.Populate,
		ReadCommentHandling = JsonCommentHandling.Skip,
		AllowTrailingCommas = true
	};
	
	private static readonly JsonSerializerOptions JsonSerializerOptionsWrite = new JsonSerializerOptions
	{
		WriteIndented = true,
		DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
		TypeInfoResolver = new DefaultJsonTypeInfoResolver()
			.WithAddedModifier(ExcludeObsoleteProperties)
	};

    private static void ExcludeObsoleteProperties(JsonTypeInfo typeInfo)
    {
        if (typeInfo.Kind != JsonTypeInfoKind.Object)
            return;
        foreach (var property in typeInfo.Properties)
        {
            if (property.AttributeProvider?.IsDefined(typeof(ObsoleteAttribute), true) == true)
                property.ShouldSerialize = (_, _) => false;
        }
    }
}

public record Parent1
{
	// Child1 replaced by Child2
    [Obsolete]
    public Child1 Child { internal get; set; } = new();
}

public record Parent2 : Parent1
{
    public Parent2() { }
    public Parent2(Parent1 parent) : base(parent) { }

	// Child2 replaced Child1
	// This seems to make Text.Json unhappy
    [Required]
    public new Child2 Child { get; set; } = new();
}

public record Sub
{
    public string Foo;
    public string Bar;
}

public record Child1
{
    [Obsolete]
    public string CSV { internal get; set; } = "";
}

public record Child2 : Child1
{
    public Child2() { }
    public Child2(Child1 child) : base(child) { }

    [Required]
    public List<Sub> List { get; set; } = [];
}

public class ExcludeObsoletePropertiesResolver : Newtonsoft.Json.Serialization.DefaultContractResolver
{
    protected override List<MemberInfo> GetSerializableMembers(Type objectType)
    {
        var memberList = base.GetSerializableMembers(objectType);
        memberList.RemoveAll(item => item.IsDefined(typeof(ObsoleteAttribute), true));
        return memberList;
    }
}

Expected behavior

Text.Json can serialize the class structure.

Actual behavior

Text.Json fails to serialize the class structure.

Regression?

No response

Known Workarounds

Revert changes and continue using Newtonsoft.Json :(

Configuration

.NET 8, Windows 11, x64

Other information

No response

[eiriktsarpalis]

System.Text.Json doesn't support field serialization by default, which impacts how your Sub type is being serialized. You can enable field serialization globally using the JsonSerializerOptions.IncludeFields property or locally for a given field using the JsonInclude attribute.

[ptr727]

Thank you, that behavior was totally unexpected.
What is the "definition" of a field in this context?

[eiriktsarpalis]

There is a formal definition that can be found here: https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/classes-and-structs/fields
An explainer comparing fields vs properties can be found here: https://stackoverflow.com/a/295109

[ptr727]

I made some progress, but now run into a new issue, seems that deserialization is not happening for a schema marked as [Obsolete] and internal get?

Demo code here:

using System.ComponentModel.DataAnnotations;
using System.Reflection;
using System.Text.Json;
using System.Text.Json.Serialization;
using System.Text.Json.Serialization.Metadata;

// Must contain something to keep the compiler happy
#pragma warning disable CS0219
int foo = 1;
#pragma warning restore CS0219

// Create test object and populate it with some data
/*
var parent1 = new Parent1()
{
    Child = new Child1()
    {
        List = [
            new() { Foo = "foo1", Bar = "bar1" },
            new() { Foo = "foo2", Bar = "bar2" }
            ]
    }
};
*/
var parent1Json = "{\r\n  \"Child\": {\r\n    \"List\": [\r\n      {\r\n        \"Foo\": \"foo1\",\r\n        \"Bar\": \"bar1\"\r\n      },\r\n      {\r\n        \"Foo\": \"foo2\",\r\n        \"Bar\": \"bar2\"\r\n      }\r\n    ]\r\n  }\r\n}";

// Convert to and from JSON using Newtonsoft
var parentN1 = Newtonsoft.Json.JsonConvert.DeserializeObject<Parent1>(parent1Json, JsonHelper.JsonSerializerSettingsRead);
var parentN2 = new Parent2(parentN1);
var jsonN = Newtonsoft.Json.JsonConvert.SerializeObject(parentN2, JsonHelper.JsonSerializerSettingsWrite);
Console.WriteLine(jsonN);

// Convert to and from JSON using Text.Json
// TODO: Parent1.Child.List is not deserialized
var parentT1 = JsonSerializer.Deserialize<Parent1>(parent1Json, JsonHelper.JsonSerializerOptionsRead);
var parentT2 = new Parent2(parentT1);
var jsonT = JsonSerializer.Serialize(parentT2, JsonHelper.JsonSerializerOptionsWrite);
Console.WriteLine(jsonT);

public static class JsonHelper
{
    public static readonly Newtonsoft.Json.JsonSerializerSettings JsonSerializerSettingsRead = new Newtonsoft.Json.JsonSerializerSettings
    {
        ObjectCreationHandling = Newtonsoft.Json.ObjectCreationHandling.Reuse
    };

    public static readonly Newtonsoft.Json.JsonSerializerSettings JsonSerializerSettingsWrite = new Newtonsoft.Json.JsonSerializerSettings
    {
        Formatting = Newtonsoft.Json.Formatting.Indented,
        StringEscapeHandling = Newtonsoft.Json.StringEscapeHandling.EscapeNonAscii,
        NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore,
        ContractResolver = new ExcludeObsoletePropertiesResolver()
    };

    public static readonly JsonSerializerOptions JsonSerializerOptionsRead = new JsonSerializerOptions
    {
        IncludeFields = true,
        PreferredObjectCreationHandling = JsonObjectCreationHandling.Populate,
        ReadCommentHandling = JsonCommentHandling.Skip,
        AllowTrailingCommas = true
    };

    public static readonly JsonSerializerOptions JsonSerializerOptionsWrite = new JsonSerializerOptions
    {
        IncludeFields = true,
        WriteIndented = true,
        DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
        TypeInfoResolver = new DefaultJsonTypeInfoResolver()
            .WithAddedModifier(ExcludeObsoleteProperties)
    };

    public static void ExcludeObsoleteProperties(JsonTypeInfo typeInfo)
    {
        if (typeInfo.Kind != JsonTypeInfoKind.Object)
            return;
        foreach (var property in typeInfo.Properties)
        {
            if (property.AttributeProvider?.IsDefined(typeof(ObsoleteAttribute), true) == true)
                property.ShouldSerialize = (_, _) => false;
        }
    }
}

public record Parent1
{
    [Obsolete]
    public Child1 Child { internal get; set; } = new();
}

public record Parent2 : Parent1
{
    public Parent2(Parent1 parent) : base(parent) 
    { 
        Upgrade(); 
    }

    [Required]
    public new Child2 Child { get; set; } = new();

#pragma warning disable CS0612 // Type or member is obsolete
    protected void Upgrade() 
    { 
        Parent1 parent1 = this;
        Child.NewList.AddRange(parent1.Child.List);
        parent1.Child.List.Clear();
    }
#pragma warning restore CS0612 // Type or member is obsolete
}

public record Sub
{
    public string Foo;
    public string Bar;
}

public record Child1
{
    public Child1() { }

    [Obsolete]
    public List<Sub> List { internal get; set; } = [];
}

public record Child2 : Child1
{
    public Child2() { }
    public Child2(Child1 child) : base(child) { }

    [Required]
    public List<Sub> NewList { get; set; } = [];
}

public class ExcludeObsoletePropertiesResolver : Newtonsoft.Json.Serialization.DefaultContractResolver
{
    protected override List<MemberInfo> GetSerializableMembers(Type objectType)
    {
        var memberList = base.GetSerializableMembers(objectType);
        memberList.RemoveAll(item => item.IsDefined(typeof(ObsoleteAttribute), true));
        return memberList;
    }
}

[elgonzo]

@ptr727

since the get accessor is not public, you need to annotate it with the [JsonInclude] attribute to include it in (de)serialization. E.g.:

public record Child1
{
    public Child1() { }

    [Obsolete]
    [JsonInclude]
    public List<Sub> List { internal get; set; } = [];
}

The documentation does not seem to be clear about this, but i suspect that the deserializer notices the lack of a public getter and thus excludes the property from deserialization. And because the deserialization logic doesn't acknowledge the public setter without considering the getter (similarily to it also not acknowledging setter-only properties), it therefore neither attempts creating a new List<Sub> instance.

[ptr727]

I "assumed" that setting global IncludeFields = true, would obviate the need for field level [JsonInclude], and IgnoreReadOnlyFields default is false?

I follow a pattern of making the getter internal so that the json schema generator tooling only includes current version properties.
With Newtonsoft's schema generator I have not found another way (other than internal) to skip [Obsolete] attributes on creating the "current" version schema.

I am investigating json-everything's schema generator, maybe I can exclude [Obsolete] similarly how I exclude writing [Obsolete] properties.

[eiriktsarpalis]

I "assumed" that setting global IncludeFields = true, would obviate the need for field level [JsonInclude]

Generally speaking STJ will only serialize public members. The IncludeFields property simply opts it in to public properties, whereas internals require explicit annotation on a per-member basis.

[ptr727]

Got it, so I either annotate or remove internal, internal was kinda hacky anyway, will keep looking for a way to create schemas without needing to use internal, i.e. let the schema generator ignore [Obsolete] fields.