Should we add support to ignore cycles on serialization?

[jozkee]

Background and Motivation

Even though we have covered the blocking issue of not being able to (de)serialize reference loops (#30820, #29900) by adding ReferenceHandler.Preserve to S.T.Json, there is many asks for adding an option equivalent to Json.NET's ReferenceLoopHandling.Ignore.

Motivations for doing this are:

• ReferenceHandler.Preserve may be too cumbersome and increases payload size.
• ReferenceHandler.Preserve creates JSON incomprehensible by serializers other than S.T.Json and Json.NET.

Proposed API

namespace System.Text.Json.Serialization
{
    public abstract partial class ReferenceHandler
    {   
        public static ReferenceHandler Preserve { get; }
+        public static ReferenceHandler IgnoreCycle { get; }
    }
}

Usage Examples

class Node
{
    public string Description { get; set; }
    public Node Next { get; set; }
}

void Test()
{
    var node = new Node { Description = "Node 1" };
    node.Next = node;
    
    var opts = new JsonSerializerOptions { ReferenceHandler = ReferenceHandler.IgnoreCycle };
    
    string json = JsonSerializer.Serialize(node, opts);
    Console.WriteLine(json); // Prints: {"Description":"Node 1"}. 
    // Note property "Next" being ignored due to cycle detection.

}

Alternative Designs

This new API is being added to ReferenceHandler class since this can be considered as an alternative to deal with references that is more isolated to the circularity problem during serialization.

Comparison with Newtonsoft.Json

void Test()
{
    var node = new Node { Description = "Node 1" };
    node.Next = node;
    
    var settings = new JsonSerializerSettings { ReferenceLoopHandling = ReferenceLoopHandling.Ignore };
    
    string json = JsonConvert.SerializeObject(node, settings);
    Console.WriteLine(json); // Prints: {"Description":"Node 1"}.
}

Comparison with existing ReferenceHandler.Preserve setting in S.T.Json

void Test()
{
    var node = new Node { Description = "Node 1" };
    node.Next = node;
    
    var opts = new JsonSerializerOptions { ReferenceHandler = ReferenceHandler.Preserve };
    
    string json = JsonSerializer.Serialize(node, opts);
    Console.WriteLine(json); // Prints: {"$id":"1","Description":"Node 1","Next":{"$ref":"1"}}. 
}

Risks

One downside is that users are uanble to implement their own ReferenceHandler and cannot make their own "Ignore Cycle" handler. The discrimination between preserve and ignore would occur with an internal flag.

Concerns of adding this feature (users must be aware of these problems when opting-in for it):

• Silent loss of data given that the object where the loop is detected would be ignored from the JSON (see example: Should we add support to ignore cycles on serialization? #40099 (comment)).
• Unable to round-trip data. the JSON may differ depending on the order of serialization (e.g: object properties and dictionary elements enumeration is non-deterministic).

[jozkee]

From #30820 (comment):

I have to agree with many other concearning Ignore - and I'd like to try to add some arguments. Unfortunately video is not optimal for me, so I might have missed good arguments in the videos (but I've read through this issue/comments at least).

My arguments are based on that Preserve is the only good alternative. My reasoning for that: [JsonIgnore] is not feasible whenever you have a two-way relationship and you sometimes need to fetch either type of object (and you want the related objects with it, but not the self-references). Except for that there doesn't seem to be a good workaround?

• Like @diegogarber pointed out, it breaks backwards compatibility and can induce significant additional work (e.g. frontend wise)
• It causes a significantly larger payload. The original issues first two examples (Ignore vs Preserve) is pretty clear (~twice the amount of lines). For Web API scenarios (which might be the most common scenario for serializing JSON), that's pretty horrendous (even if the added stuff is quite short).
• The JSON structure is not obvious from a frontend perspective - which the Ignore structure is (imo). For scenarios with Ignore where you really need the references, you could easily identify that specific call (again, assuming a Web API) and do your ViewModel-layer instead. The Preserve structure on the other hand needs parsing - specifically turning lists (arrays) into collections (objects) will be quite confusing.

To sum up: To break backwards compatibility, cause a significant larger payload, as well as breaking the structure by introducing a middle layer (as in Subordinates not being a list/array of values), should be reason enough to consider having Ignore.

I believe the StackOverflow issue mentioned reflects the community's stand on this - the only reasonable way forward (unless for specific scenarios where you really need the C#/serialization performance) is to continue with Newtonsoft. To me, that should seen as a sign that this is not a reasonable way forward. It's been a long wait between 3.0 -> 5.0 (we're still waiting after all), and to still not having this resolved will definitely continue to stir up frustration.

[jozkee]

From #30820 (comment)

The problem with Ignore is that you end-up losing data, this will probably cause an issue even more severe than having a large payload or having to do extra parsing in the front-end.

e.g:

static void Main()
{
    var angela = new Employee { Name = "Angela" };
    var bob = new Employee { Name = "Bob", Manager = angela };

    angela.Subordinates = new List<Employee> { bob };

    var settings = new JsonSerializerSettings { ReferenceLoopHandling = ReferenceLoopHandling.Ignore };
    string json = JsonConvert.SerializeObject(bob, settings);
    Console.WriteLine(json);
    // {"Name":"Bob","Manager":{"Name":"Angela","Manager":null,"Subordinates":[]},"Subordinates":null}

    Employee bobCopy = JsonConvert.DeserializeObject<Employee>(json);
    Employee angelaCopy = bobCopy.Manager;

    Console.WriteLine("Number of subordinates for angela: " + angela.Subordinates.Count);
    // Number of subordinates for angela: 1
    Console.WriteLine("Number of subordinates for angelaCopy: " + angelaCopy.Subordinates.Count);
    // Number of subordinates for angelaCopy: 0
    Console.WriteLine(angelaCopy.Subordinates.Count == angela.Subordinates.Count);
    // False
}

@cjblomqvist How is that similar scenarios are not a concern for you or everyone else asking for Ignore?

[jozkee]

From #30820 (comment)

@jozkee thanks for your reply!

I believe a scenario which is probably quite common is that you have let's say 100 types, with various relationships between them. Some contian many, some not so many. Basically all are related to at least one other type one way or another, and you have a REST-like Web API to get them from a Web/Mobile application. To map out all relationships in each scenario and make sure you handle all self-referencing scenarios is quite cumbersome. So what happens is, the user is asking for a main type A, and then you ensure you also get all the related types that you believe the "user" might need, such as B, C, D, E, F, G (E, F, G might be sub-relationships). You do not filter this properly (i.e. you do not project out properties not needed) because of not wanting to "waste" time on it (as well as laziness). To map out this tree:

A
A -> B
A -> C
A -> D
A -> B -> E
A -> B -> E -> F
A -> D -> G

So far, all good. The problem is when you also have the following relationships:

C -> A
E -> A
D -> A
G -> A

It's cumbersome to properly keep track of them and filter them out (e.g. through JsonIgnore - which also has the problem of filtering out the relationships in all scenarios, not only for this particular call for A). It would also be cumbersome to handle it using preserve due to the A) wasted space/size to transmit over the network (might be less of a concearn in above scenario since obviously we're not so picky with what we're including and not), and B) the data structure does not map in a fully obvious way to JSON, but with a quite new-JSON-library-in-.NET specific way.

Then, to finally answer your question: The "user" (frontend consumer) simply does not care about the additional relationships [C, E, D, G] -> A - it only wants to know the original tree. The loss of data is irrelevant - because the relationship properties are not relevant.

I'm not say that Ignore is perfect by any means, but it is darn convenient to simply add it to a project and then not having to think about it anymore. Something like Preserve could be very useful, but the current implementation is too cumbersome for most cases (with all the downsides listed above) - it doesn't fulfil that simple-and-convenient-albeit-not-100%-correct scenario. I do understand that there are reasons behind Preserve being the way it is, so I'm not saying the Preserve is wrong. What I'm saying is that Preserve is good for one thing (100% correctness while still handling self-referencing), while Ignore is good for another (simple/convenient, but not 100% correct in some cases).

I do believe a lot of people in the community feel the same.

Something in between Ignore and Preserve might be very useful, especially that doesn't break the assumed data structure (to avoid the frontend parsing/unexpected data structure for lists/arrays) while still not loosing data, but I understand that there are difficulties/complexities with this, so Ignore might be good enough to solve the relevant actual real world scenarios.

[danmoseley]

@jozkee is this required to ship 5.0 (per milestone set here)?

[jozkee]

@danmosemsft I think it could disappoint many users if we don't ship this on 5.0.

[danmoseley]

@jozkee maybe, but master is a 6.0 branch in less than 3 weeks: if we add features, we're implicitly deciding to leave bugs un-fixed.

That's a general statement - I don't have context on this issue or JSON in general so it would be a good discussion to have with @ericstj .

[ericsampson]

@jozkee @ericstj FWIW when I heard that STJ would have loop handling in 5.0, I just assumed that would include something along the lines of Ignore. It would be disappointing to not have something along those lines in 5.0
Also FWIW I don't love the Newtonsoft options. Like another commenter mention above, it would be great to have an in-between option that didn't suffer the data loss of Ignore; one option that seems great space-wise would be to only emit reference tags where they are needed, although that might require backtracking/rewriting?

[snow-jallen]

What's the verdict? Is this going to ship in .net 5?