JsonSerializer support for fields & non-public accessors

[layomia]

For streamlined API review and due to common API suggestions, this issue addresses non-public accessor support, and field support.

Motivation

Using non-public accessors for serialization and deserialization

From #29743:

Non-public setters for public properties are not used when deserializing. Other serializers have this feature. As a workaround, user's have to write custom converter's for such POCOs, which isn't trivial in case of a complex object. This feature provides an opt-in for the serializer to use non-public setters.

Enabling non-public getter usage is included for parity with Newtonsoft.Json which supports this when the JsonProperty attribute is used, and to prevent complicating the API surface if this is ever desired in the future.

This feature was scoped out of v1 (3.x), as the objective was to support simple POCOs. We elected to make this feature opt-in due to security concerns with non-public support.

This feature does not include support for non-public properties.

A related feature that allows deserialization of immutable objects through parameterized constructors was recently added in #33444.

Field support

From #876:

There is no way to serialize and deserialize fields using JsonSerializer.

While public fields are generally not recommended, they are used in .NET itself (see value tuples) and by users.

This feature was scoped out of v1 (3.x) due to lack of time, as we prioritized supporting simple POCOs with public properties.

We elected to have an opt-in model for field support because it would be a breaking change to support them by default, and also because public fields are generally not recommended. Other serializers, including Newtonsoft.Json, Utf8Json, and Jil, support this feature by default.

API proposal

Option 1

namespace System.Text.Json
{
    public partial class JsonSerializerOptions
    {
        /// <summary>
        /// Determines whether fields are included when serializing and deserializing.
        /// The default value is <see langword="false"/>.
        /// </summary>
        /// <remarks>
        /// Only public fields will be serialized and deserialized.
        /// </remarks>
        public bool IncludeFields { get; set; }
    }
}

namespace System.Text.Json.Serialization
{
    /// <summary>
    /// Provides options for how object members should be serialized and deserialized.
    /// </summary>
    [AttributeUsage(AttributeTargets.Class | Attributes.Struct, AllowMultiple = false)]
    public partial sealed class JsonObjectAttribute : JsonAttribute
    {
        /// <summary>
        /// Indicates whether non-public property getters and setters are ignored when serializing and deserializing.
        /// </summary>
        /// <remarks>
        /// The default value is <see langword="false"/>.
        /// <see cref="JsonIgnoreAttribute"> can be used to ignore individual properties.
        /// </remarks>
        public bool IgnoreNonPublicAccessors { get; set; }

        /// <summary>
        /// Indicates whether public fields are ignored when serializing and deserializing.
        /// </summary>
        /// <remarks>
        /// The default value is <see langword="false"/>.
        /// <see cref="JsonIgnoreAttribute"> can be used to ignore individual fields.
        /// </remarks>
        public bool IgnoreFields { get; set; }

        /// <summary>
        /// Initializes a new instance of <see cref="JsonObjectAttribute"/>.
        /// </summary>
        public JsonObjectAttribute() { }
    }

    /// <summary>
    /// When applied to public properties, non-public getters or setters will be used when serializing and deserializing.
    /// When applied to public fields, they will be included when serializing and deserializing.
    /// </summary>
    [AttributeUsage(AttributeTargets.Property | Attributes.Field, AllowMultiple = false)]
    public partial sealed class JsonMemberAttribute : JsonAttribute
    {
        /// <summary>
        /// Initializes a new instance of <see cref="JsonMemberAttribute"/>.
        /// </summary>
        public JsonMemberAttribute() { }
    }
}

Usage

Including fields and using non-public accessors

Given a Person class:

[JsonObject]
public class Person
{
    public string FirstName;
    
    public string LastName;

    public int Id { get; private set; }
}

Alternatively,

public class Person
{
    [JsonMember]
    public string FirstName;

    [JsonMember]
    public string LastName;

    [JsonMember]
    public int Id { get; private set; }

An instance can be serialized or deserialized:

string json = @"{""FirstName"":""Jet"",""LastName"":""Doe"",""Id"":123}";

Person person = JsonSerializer.Deserialize<Person>(json);
Console.WriteLine(person.FirstName); // Jet
Console.WriteLine(person.LastName); // Doe
Console.WriteLine(person.Id); // 123

json = JsonSerializer.Serialize(person);
Console.WriteLine(json); // {"FirstName":"Jet","LastName":""Doe","Id":123}

Including fields and ignoring non-public accessors

Given a Person class:

[JsonObject(IgnoreNonPublicAccessors = true)]
public class Person
{
    public string FirstName;
    
    public string LastName;

    public int Id { get; private set; }
}

Alternatively,

public class Person
{
    [JsonMember]
    public string FirstName;

    [JsonMember]
    public string LastName;

    public int Id { get; private set; }

An instance can be serialized or deserialized:

string json = @"{""FirstName"":""Jet"",""LastName"":""Doe"",""Id"":123}";

Person person = JsonSerializer.Deserialize<Person>(json);
Console.WriteLine(person.FirstName); // Jet
Console.WriteLine(person.LastName); // Doe
Console.WriteLine(person.Id); // 0

json = JsonSerializer.Serialize(person);
Console.WriteLine(json); // {"FirstName":"Jet","LastName":""Doe","Id":0}

Including fields globally but ignoring them for a particular type

Given an Account class:

public class Account
{
    public int Id;

    public AccountType Type;

    public Person Owner;
}

public enum AccountType
{
    Checking = 0,
    Saving = 1,
    MoneyMarket = 2
}

And a Person class:

[JsonObject(IgnoreFields = true)]
public class Person
{
    public string FirstName;
    
    public string LastName;

    public int Id { get; private set; }
}

Account instances can be serialized and deserialized:

JsonSerializerOptions options = new JsonSerializerOptions
{
    IncludeFields = true
};

string json = @"{
    ""Id"": 12345,
    ""Type"": 1,
    ""Owner"": {
        ""FirstName"":""Jet"",
        ""LastName"":""Doe"",
        ""Id"":123
    }
}";

Account account = JsonSerializer.Deserialize<Account>(json, options);
Console.WriteLine(account.Id); // 12345
Console.WriteLine(account.Type); // Saving

Person person = account.Person;
Console.WriteLine(person.FirstName) // null
Console.WriteLine(person.LastName) // null
Console.WriteLine(person.Id) // 123

// Prepare for serialization.
person.FirstName = "Jet";
person.LastName = "Doe";

json = JsonSerializer.Serialize(account, options);
Console.WriteLine(json); // {"Id":12345,"Type":1,"Owner":{"Id":123}}

• In the future, placing JsonObject on an enumerable could be used to make the serializer treat it like an object with properties: Provide option for the serializer to treat IEnumerables like objects with members #1808.
• Provides easy way to ignore non-public accessors and exclude fields for each type.

Option 2

namespace System.Text.Json
{
    public partial class JsonSerializerOptions
    {
        /// <summary>
        /// Determines whether fields are included when serializing and deserializing.
        /// The default value is false.
        /// </summary>
        /// <remarks>
        /// Only public fields will be serialized and deserialized.
        /// </remarks>
        public bool IncludeFields { get; set; }
    }
}

namespace System.Text.Json.Serialization
{
    /// <summary>
    /// When applied to types, non-public getters or setters will be used when serializing and deserializing all public properties.
    /// Use <see cref="JsonIgnoreAttribute"/> to opt-out.
    /// When applied to public properties, non-public getters or setters will be used when serializing and deserializing.
    /// </summary>
    [AttributeUsage(AttributeTargets.Class | AttributeTargets.Struct | AttributeTargets.Property, AllowMultiple = false)]
    public sealed class JsonPropertySerializableAttribute : JsonAttribute
    {
        /// <summary>
        /// Initializes a new instance of <see cref="JsonPropertySerializableAttribute"/>.
        /// </summary>
        public JsonPropertySerializableAttribute() { }
    }

    /// <summary>
    /// When applied to a type, all its public fields will be serialized and deserialized. Use <see cref="JsonIgnoreAttribute"/> to opt-out.
    /// When applied to a public field, it will be serialized and deserialized.
    /// </summary>
    /// <remarks>
    /// The absence of this attribute on a class, struct, or field, does not preclude the fields from serialization or deserialization, if <see cref="JsonSerializerOptions.IncludeFields"> is set to <see langword="true">.
    /// </remarks>
    [AttributeUsage(AttributeTargets.Class | AttributeTargets.Struct | AttributeTargets.Field, AllowMultiple = false)]
    public sealed class JsonFieldSerializableAttribute : JsonAttribute
    {
        /// <summary>
        /// Initializes a new instance of <see cref="JsonFieldSerializableAttribute"/>.
        /// </summary>
        public JsonFieldSerializableAttribute() { }
    }
}

Usage

Including fields and using non-public accessors

Given a Person class:

[JsonPropertySerializable]
[JsonFieldSerializable]
public class Person
{
    public string FirstName;
    
    public string LastName;

    public int Id { get; private set; }
}

Alternatively,

public class Person
{
    [JsonPropertySerializable]
    public string FirstName;

    [JsonPropertySerializable]
    public string LastName;

    [JsonFieldSerializable]
    public int Id { get; private set; }

An instance can be serialized or deserialized:

string json = @"{""FirstName"":""Jet"",""LastName"":""Doe"",""Id"":123}";

Person person = JsonSerializer.Deserialize<Person>(json);
Console.WriteLine(person.FirstName); // Jet
Console.WriteLine(person.LastName); // Doe
Console.WriteLine(person.Id); // 123

json = JsonSerializer.Serialize(person);
Console.WriteLine(json); // {"FirstName":"Jet","LastName":""Doe","Id":123}

Including fields and ignoring non-public accessors

Given a Person class:

[JsonFieldSerializable]
public class Person
{
    public string FirstName;
    
    public string LastName;

    public int Id { get; private set; }
}

Alternatively,

public class Person
{
    [JsonFieldSerializable]
    public string FirstName;

    [JsonFieldSerializable]
    public string LastName;

    public int Id { get; private set; }

An instance can be serialized or deserialized:

string json = @"{""FirstName"":""Jet"",""LastName"":""Doe"",""Id"":123}";

Person person = JsonSerializer.Deserialize<Person>(json);
Console.WriteLine(person.FirstName); // Jet
Console.WriteLine(person.LastName); // Doe
Console.WriteLine(person.Id); // 0

json = JsonSerializer.Serialize(person);
Console.WriteLine(json); // {"FirstName":"Jet","LastName":""Doe","Id":0}

Including fields globally but ignoring them for a particular type

Given an Account class:

public class Account
{
    public int Id;

    public AccountType Type;

    public Person Owner;
}

public enum AccountType
{
    Checking = 0,
    Saving = 1,
    MoneyMarket = 2
}

And a PersonClass:

public class Person
{
    [JsonIgnore]
    public string FirstName;
    
    [JsonIgnore]
    public string LastName;

    [JsonFieldSerializable]
    public int Id { get; private set; }
}

Alternatively,

[JsonFieldSerializable]
public class Person
{
    [JsonIgnore]
    public string FirstName;
    
    [JsonIgnore]
    public string LastName;

    public int Id { get; private set; }
}

Account instances can be serialized and deserialized:

JsonSerializerOptions options = new JsonSerializerOptions
{
    IncludeFields = true
};

string json = @"{
    ""Id"": 12345,
    ""Type"": 1,
    ""Owner"": {
        ""FirstName"":""Jet"",
        ""LastName"":""Doe"",
        ""Id"":123
    }
}";

Account account = JsonSerializer.Deserialize<Account>(json, options);
Console.WriteLine(account.Id); // 12345
Console.WriteLine(account.Type); // Saving

Person person = account.Person;
Console.WriteLine(person.FirstName) // null
Console.WriteLine(person.LastName) // null
Console.WriteLine(person.Id) // 123

// Prepare for serialization.
person.FirstName = "Jet";
person.LastName = "Doe";

json = JsonSerializer.Serialize(account, options);
Console.WriteLine(json); // {"Id":12345,"Type":1,"Owner":{"Id":123}}

To override JsonSerializer.IncludeFields when set to true, [JsonIgnore] would need to be set on each field to be ignored.

Notes

• The opt-in mechanism for non-public accessor is per property and per type, not "globally" on JsonSerializerOptions. This is to prevent non-public member access on types that are not owned by the user.

• Features involving the the use of non-public accessors, including constructors, getters, setters etc. may not be supported in the upcoming AOT/code-gen work due to the likely need to use runtime reflection for those scenarios.

• The options on JsonSerializerOptions that apply to properties will also apply to fields. This is intuitive and keeps the API surface clean:

  • IgnoreNullValues
  • PropertyNameCaseInsensitive
  • PropertyNamingPolicy

• Existing attributes that apply to properties will also apply to fields:

  • JsonConverterAttribute
  • JsonExtensionDataAttribute
  • JsonIgnoreAttribute
  • JsonPropertyNameAttribute

• Interfaces will not be added as a target until the implications for extended polymorphism support are understood. This is in keeping with JsonConverterAttribute where an interface is not a valid target: Add AttributeTargets.Interface to JsonConverterAttribute #33112.

  • Polymorphic deserialization: JsonSerializer polymorphic serialization and deserialization support #30083
  • Polymorphic serialization: Support polymorphic serialization through new option #29937

• As with public properties, public fields may also bind with constructor parameters during deserialization, with the same semantics.

[Symbai]

Features involving the the use of non-public accessors, including constructors, getters, setters etc. may not be supported in the upcoming AOT/code-gen work due to the likely need to use runtime reflection for those scenarios.

Why would it require runtime reflection and what speaks against it (Performance? Security?) and what speaks against it making it optional so in case of performance developers have a choice when performance / security isn't important?

[layomia]

Why would it require runtime reflection

In a design where serializers/converters are generated at build-time to reduce the serializer's start-up cost and reduce/eliminate the usage of reflection features that are not supported by some platforms, non-public usage may not be supported dependent on the output assembly of the generated code. This may force the serializer down existing code paths which employ such reflection features as a fallback, which may be problematic depending on the scenario/platform.

what speaks against it making it optional so in case of performance developers have a choice when performance / security isn't important?

This is a valid design possibility.

These are very early thoughts on potential serializer code-gen work in the future; it is not clear yet how non-public usage may affect this work.

[terrajobst]

Video

• Let's rename JsonMemberAttribute to JsonIncludeAttribute
• We should change JsonIgnoreAttribute to include fields
• We don't see a need for JsonObjectAttribute it should be a per property/field decisions
• We should treat read-only fields like get-only properties
• We can decide to support private/internal fields/properties by applying JsonInclude, but we need think through implications for AOT (generating the serialzier as part of the type would make this possible tho)

namespace System.Text.Json
{
    public partial class JsonSerializerOptions
    {
        public bool IncludeFields { get; set; }
    }
}
namespace System.Text.Json.Serialization
{
    [AttributeUsage(AttributeTargets.Property |
                    Attributes.Field, AllowMultiple = false)]
    public sealed class JsonIncludeAttribute : JsonAttribute
    {
        public JsonIncludeAttribute();
    }
}

[layomia]

Updated proposal approved offline:

namespace System.Text.Json
{
    public partial class JsonSerializerOptions
    {
        public bool IgnoreReadOnlyFields { get; set; }
        public bool IncludeFields { get; set; }
    }
}
namespace System.Text.Json.Serialization
{
    [AttributeUsage(AttributeTargets.Property |
                    Attributes.Field, AllowMultiple = false)]
    public sealed class JsonIncludeAttribute : JsonAttribute
    {
        public JsonIncludeAttribute();
    }
}

IgnoreReadOnlyFields was added for parity with IgnoreReadOnlyProperties.

---

The serializer's defintion for "read only" is as follows:

• for properties, "read only" means has a public getter but no public setter.
• for fields, "read only" means FieldInfo.IsInitOnly is true.

This definition may be updated following the upcoming C# lang init-only properties (& perhaps fields), dependent on the type/reflection metadata made available.

If init-only setters are available and public, it would be reasonable for the serializer to use them for deserialization. We would have to evaluate whether the serialization of such properties & fields should be influenced by options.IgnoreReadOnly[Properties | Fields]. Any change here may be breaking.

cc @terrajobst

[layomia]

Closing this composite issue as there are tracking issues for the sub-tasks:

• Using non-public getters and setters: JsonSerializer should support private setters as an opt-in feature #29743
• Field support: JsonSerializer should support field as well as properties #876