Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System.Text.Json Reference Loop Handling #38579

Open
MoienTajik opened this issue Jun 15, 2019 · 14 comments

Comments

@MoienTajik
Copy link

@MoienTajik MoienTajik commented Jun 15, 2019

One of the key features of JSON.NET serialization was the ReferenceLoopHandling which gives the ability to ignore the reference loops likes this :

public class Employee
{
    public string Name { get; set; }

    public Employee Manager { get; set; }
}

private static void Main()
{
    var joe = new Employee { Name = "Joe - User" };
    var mike = new Employee { Name = "Mike - Manager" };
    joe.Manager = mike;
    mike.Manager = mike;

    var json = JsonConvert.SerializeObject(joe, new JsonSerializerSettings
    {
        Formatting = Formatting.Indented,
        ReferenceLoopHandling = ReferenceLoopHandling.Ignore
    });
	
    Console.WriteLine(json);
}

And it produces the appropriate result :

{
  "Name": "Joe User",
  "Manager": {
    "Name": "Mike Manager"
  }
}

However, I couldn't find such a feature in System.Text.JSON, And when I've tried to Serialize the same object with JsonSerializer :

private static void Main()
{
    var joe = new Employee { Name = "Joe User" };
    var mike = new Employee { Name = "Mike Manager" };
    joe.Manager = mike;
    mike.Manager = mike;

    var json = JsonSerializer.ToString(joe, new JsonSerializerOptions
    {
        WriterOptions = new JsonWriterOptions
        {
            Indented = true,
        }
    });

    Console.WriteLine(json);
}

I've got this exception :
System.InvalidOperationException: 'CurrentDepth (1000) is equal to or larger than the maximum allowed depth of 1000. Cannot write the next JSON object or array.'

So, Is this feature exist in System.Text.Json now that I couldn't find it?
And if not, Are there any plans to support this?

@xsoheilalizadeh

This comment has been minimized.

Copy link

@xsoheilalizadeh xsoheilalizadeh commented Jun 16, 2019

This could make a lot of bugs, I have these self-reference models in my APIs and since now I never had Reference Loop issue with Json.NET. If we decided to replace the Json.NET with the new System.Text.Json we should consider this feature as soon as it is possible.

@ahsonkhan

This comment has been minimized.

Copy link
Member

@ahsonkhan ahsonkhan commented Jun 18, 2019

So, Is this feature exist in System.Text.Json now that I couldn't find it?
And if not, Are there any plans to support this?

This is a known limitation of System.Text.Json (we do not have this feature yet) and we have little runway left to design and implement this for 3.0. We plan to add this for vNext, for sure (there are quite a few features/capabilities in Json.NET that wouldn't work in System.Text.Json and given the constraints, we had to prioritize what made it in).

This could make a lot of bugs, I have these self-reference models in my APIs and since now I never had Reference Loop issue with Json.NET

Given we detect and throw for self-reference models, how could it cause bugs?

@ahsonkhan

This comment has been minimized.

Copy link
Member

@ahsonkhan ahsonkhan commented Jun 18, 2019

@AmrAlSayed0

This comment has been minimized.

Copy link

@AmrAlSayed0 AmrAlSayed0 commented Jun 24, 2019

Would this API be good?
This new property should be added to the JsonWriterOptions struct.

public struct JsonWriterOptions
{
    public ReferenceLoopHandlingOption ReferenceLoopHandling { get; set; }
}
public struct ReferenceLoopHandlingOption
{
    //A static member for easy access to pre-defined options.
   // This causes it to ignore all looped refrences.
    public static ReferenceLoopHandlingIgnore = new ReferenceLoopHandlingOption ( ReferenceLoopHandling.Ignore );
   //This causes the writer to output the looped reference only once after the first time.
   // {
   //   "Name": "Joe User",
   //   "Manager": {
   //     "Name": "Mike Manager",
   //     "Manager": {
   //       "Name":"Mike Manager"
   //     }
   //   }
   // }
    public static ReferenceLoopHandlingOnce = new ReferenceLoopHandlingOption ( ReferenceLoopHandling.Once );
    // Current (and default?) behavior.
    public static ReferenceLoopHandlingAll = new ReferenceLoopHandlingOption ( ReferenceLoopHandling.All );
    public ReferenceLoopHandlingOption ( ReferenceLoopHandling loophandling );
    // Also there is the option to specify how many times do you want the writer to write looped refrences (after the first one)
    // new ReferenceLoopHandlingOption ( ReferenceLoopHandling.Many , 3 );
    public ReferenceLoopHandlingOption ( ReferenceLoopHandling loophandling , int loopOutputTimes );
}
public enum ReferenceLoopHandling
{
    Ignore ,
    Once ,
    Many ,
    All
}
@ahsonkhan

This comment has been minimized.

Copy link
Member

@ahsonkhan ahsonkhan commented Aug 5, 2019

From @josundt (#40045 (comment)):

Object/collection graphs containing circular references (or multiple references to the same entity), can be handled in a few different ways when serializing/deserializing to/from JSON.

Best practice is naturally to avoid graphs with circular references when using JSON serialization, but techniques to deal with it exist, and some times it may be needed. We use it a few places in our software.

NewtonSoft.Json supports this, and I guess it is a goal for System.Text.Json to make the feature gap with NewtonSoft as small possible.

During serialization, when an object/collection that has already been serialized is referenced for the second time or more, the common technique I've seen is to replace the repeated objects/collection in the JSON with a

{ "$ref": "some-sort-of-pointer" }

Some use JSONPath to reference the object/array serialized previously in the JSON document.

NewtonSoft.Json uses a different techique with specific serialization settings: It adds an extra "$id" property to all serialized objects in the JSON document, and the "$ref" value is the "$id" property of the object/array serialized previously.

Example using NewtonSoft.Json:

using System;
using System.Collections.Generic;
using Newtonsoft.Json;

namespace CircularSerialization
{
    public class Bike
    {
        public List<Tire> Tires { get; set; } = new List<Tire>();
    }

    public class Tire
    {
        public Bike Bike { get; set; }
    }

    static class Program
    {
        static void Main()
        {
            var bike = new Bike();
            bike.Tires.Add(new Tire { Bike = bike });
            bike.Tires.Add(new Tire { Bike = bike });

            var settings = new JsonSerializerSettings
            {
                PreserveReferencesHandling = PreserveReferencesHandling.Objects,
                ReferenceLoopHandling = ReferenceLoopHandling.Ignore
            };
            var serialized = JsonConvert.SerializeObject(bike, settings);

            Console.Write(serialized);
            // {"$id":"1","Tires":[{"$id":"2","Bike":{"$ref":"1"}},{"$id":"3","Bike":{"$ref":"1"}}]}
        }
    }
}

I guess that if System.Text.Json.JsonSerializer will support circular/multiple references, some more memory allocation will be required during serialization/deserialization when the setting is switched on (so such a setting should naturally be off by default).

Is there a plan to support options for handling circular references in JsonSerializerSettings?
And if so, will you support the "$id" technique or the JSONPath techique, or maybe support both?

Reference:
https://github.com/douglascrockford/JSON-js/blob/master/cycle.js

@euclid47

This comment has been minimized.

Copy link

@euclid47 euclid47 commented Oct 7, 2019

Will the enhancement be released in 3.1?

@davidnmbond

This comment has been minimized.

Copy link

@davidnmbond davidnmbond commented Oct 27, 2019

Really need a fix for this one. It's not currently possible to serialize EF Core queries with populated navigation properties.

I'd suggest an approach similar to NewtonSoft's ReferenceLoopHandling:

https://www.newtonsoft.com/json/help/html/SerializationSettings.htm#ReferenceLoopHandling

@MagicAndre1981

This comment has been minimized.

Copy link

@MagicAndre1981 MagicAndre1981 commented Oct 28, 2019

Will the enhancement be released in 3.1?

Milestone is set to 5, so no it will be not part of 3.1 and so this is useless at all.

@julienGrd

This comment has been minimized.

Copy link

@julienGrd julienGrd commented Oct 28, 2019

Really need a fix for this one. It's not currently possible to serialize EF Core queries with populated navigation properties.

I'd suggest an approach similar to NewtonSoft's ReferenceLoopHandling:

https://www.newtonsoft.com/json/help/html/SerializationSettings.htm#ReferenceLoopHandling

You can for the moment switch to newtonsoft, im facing the same problem and i use this class extensions (adapt the settings to your needs)

public static class HttpClientExtension
    {
        public static async Task<T> PostJsonCustomAsync<T>(this HttpClient sender, string requestUrl, object postData)
        {
            sender.DefaultRequestHeaders.Add("Accept", "application/json");

            string stringPostData = JsonConvert.SerializeObject(postData, Settings);

            HttpContent body = new StringContent(stringPostData, Encoding.UTF8, "application/json");
            var response = await sender.PostAsync(requestUrl, body);

            string text = await response.Content.ReadAsStringAsync();

            T data = JsonConvert.DeserializeObject<T>(text, Settings);

            return data;
        }


        public static async Task<T> GetJsonCustomAsync<T>(this HttpClient sender, string requestUrl)
        {
            sender.DefaultRequestHeaders.Add("Accept", "application/json");

            var response = await sender.GetAsync(requestUrl);

            string text = await response.Content.ReadAsStringAsync();

            T data = JsonConvert.DeserializeObject<T>(text, Settings);

            return data;
        }

        public static async Task<T> PutJsonCustomAsync<T>(this HttpClient sender, string requestUrl, object putData)
            where T : new()
        {
            sender.DefaultRequestHeaders.Add("Accept", "application/json");

            string stringPutData = JsonConvert.SerializeObject(putData, Settings);

            HttpContent body = new StringContent(stringPutData, Encoding.UTF8, "application/json");
            var response = await sender.PutAsync(requestUrl, body);

            string text = await response.Content.ReadAsStringAsync();

            T data = JsonConvert.DeserializeObject<T>(text, Settings);

            return data;
        }

        private static JsonSerializerSettings Settings = new JsonSerializerSettings()
        {
            DateParseHandling = DateParseHandling.None,
            DateTimeZoneHandling = DateTimeZoneHandling.Unspecified
        };
    }
@ahsonkhan

This comment has been minimized.

Copy link
Member

@ahsonkhan ahsonkhan commented Nov 4, 2019

Folks on this thread who are requesting this feature, do you need it for deserialization as well as serialization? Most of the examples here are about serialization.

Also, how heavily (if at all) do you rely on Newtonsoft.Json's MetadataPropertyHandling.ReadAhead feature (to enable out-of-order metadata property support):
https://www.newtonsoft.com/json/help/html/DeserializeMetadataPropertyHandling.htm

I'd be interested in seeing current usages.

Also cc @ajcvickers, @Andrzej-W

@Andrzej-W

This comment has been minimized.

Copy link

@Andrzej-W Andrzej-W commented Nov 4, 2019

@ahsonkhan, people serialize objects to deserialize them later - we need support for loops in both of them. Now when we have Blazor Wasm (officially it will be released in May 2020) this is a typical scenario:

  • we have ASP.NET Web API service which is used to read some data from database and serialize it,
  • we read this response in Blazor and have to deserialize it.

Databases are usually designed in such a way that we can read child items when we have parent item and we can read parent item when we have one of its children. As a direct consequence we have bidirectional links between POCO classes used in Entity Framework. Simple example: in InvoiceHeader class we have a property with list of InvoiceLines and in every InvoiceLine we have a reference to InvoiceHeader. This is similar to Bike and Tire example in one of the posts above.

In my opinion this is such a basic requirement that it have to be implemented as soon as possible. Without support for reference loops it will be very hard to write any real world Blazor application or any other application which have to serialize and deserialize object graphs used in Entity Framework.

@ajcvickers

This comment has been minimized.

Copy link
Member

@ajcvickers ajcvickers commented Nov 4, 2019

@ahsonkhan Handling of cycles is required for serializing the results from EF (or any other OR/M) for the reasons stated by @Andrzej-W. However, I don't know which specific flags from Newtonsoft.Json map to this.

@BreyerW

This comment has been minimized.

Copy link

@BreyerW BreyerW commented Nov 4, 2019

Handling loops on both serialize and deserialize is required to be useful on many complex scenarios (EF was good example, i also use it privately). Best if you also provide something akin to ReferenceResolver like json.net do, because not everyone needs full ref loops and sometimes they want to restrict it to types that have guid as an example.

@Jozkee

This comment has been minimized.

Copy link
Member

@Jozkee Jozkee commented Nov 6, 2019

people serialize objects to deserialize them later - we need support for loops in both of them

I agree, having a way to support loops and the ability to round-trip them is a must; to address that, System.Text.Json should implement something similar to Json.Net's PreserveReferencesHandling.All and MetadataPropertyHandling.Default.

With that said, is there any value left in including a feature similar to ReferenceLoopHandling.Ignore?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.