What is the recomendation of using System.Text.Json and double NaN values ?

[pranavkm]

From @valeriob on Monday, September 30, 2019 12:49:01 PM

Hi,
we are upgrading some applications to aspnetcore 3.0, we have been bitten by something unexpected : System.Text.Json refuse to serialize double.NaN values.
This cause really nasty bugs because something that works suddenly does not because some result of some calculation is different, or some data changes in some way.
I see that the recommendation (https://docs.microsoft.com/en-us/aspnet/core/migration/22-to-30?view=aspnetcore-3.0&tabs=visual-studio#jsonnet-support) is to use Json.Net.

But how can ppl use this library if such a simple and common use case is not covered ?
I do not know any application that can live without this feature, i understand the fact that there is a specification, but it may very well be unpractical.
https://thefactotum.xyz/post/the-devil-is-in-the-json-details/
Python, Go, Javascript,Rust, Ruby handle it without making much fuss 😄

Thanks
Valerio

Copied from original issue: dotnet/aspnetcore#14571

[safern]

cc: @ahsonkhan

[ericstj]

@tannergooding can you weigh in on this?

[tannergooding]

This is a more complicated scenario as some floating-point values (+infinity, -infinity, and NaN) can't be represented as JSON "numbers".

The only way these can be successfully serialized/deserialized is by representing them by using an alternative format (such as strings).

[ericstj]

JSon.NET stores these as strings in the payload then automatically converts to double.

Console.WriteLine(JsonConvert.SerializeObject(new double [] { Double.PositiveInfinity, Double.NaN, 0.100, 1.0002, Math.PI }));

Writes

["Infinity","NaN",0.1,1.0002,3.141592653589793]

And the following succeeds:

JsonConvert.DeserializeObject<double[]>("[\"Infinity\",\"NaN\",0.1,1.0002,3.141592653589793]");

Is this something we should do automatically? If not, then what should OP do to handle this? Write a converter? /cc @steveharter

[pranavkm]

The linked document - https://thefactotum.xyz/post/the-devil-is-in-the-json-details/#javascript - includes examples of what browsers do when when these values are serialized:

> JSON.stringify({"nan": NaN})
'{"nan":null}'
> JSON.stringify({"inf": Infinity})
'{"inf":null}'

// My local tests:
JSON.stringify([Number.NEGATIVE_INFINITY, Number.POSITIVE_INFINITY, Number.NaN])
"[null,null,null]"

IMO, authoring a converter if these uncommon scenarios are important seem like the way to go.

[ahsonkhan]

Is this something we should do automatically? If not, then what should OP do to handle this? Write a converter?

Yes, write a converter to handle the special case.

IMO, authoring a converter if these uncommon scenarios are important seem like the way to go.

Agreed.

Here's an example of a converter you could write to be able to round-trip the special double values (like NaN/infinities) and also one to be compatible with Newtonsoft.Json's way of serializing special doubles for round-tripping:

[Fact]
public static void HandleSpecialDoubles_NewtonsoftCompat()
{
    string expectedJson = JsonConvert.SerializeObject(new double[] { double.PositiveInfinity, double.NegativeInfinity, double.NaN, 0.100, 1.0002, Math.PI });
    Console.WriteLine(expectedJson);
    double[] expected = JsonConvert.DeserializeObject<double[]>("[\"Infinity\",\"-Infinity\",\"NaN\",0.1,1.0002,3.141592653589793]");

    var options = new JsonSerializerOptions();
    options.Converters.Add(new HandleSpecialDoublesAsStrings_NewtonsoftCompat());

    string actualJson = JsonSerializer.Serialize(new double[] { double.PositiveInfinity, double.NegativeInfinity, double.NaN, 0.100, 1.0002, Math.PI }, options);
    Console.WriteLine(actualJson);
    double[] actual = JsonSerializer.Deserialize<double[]>("[\"Infinity\",\"-Infinity\",\"NaN\",0.1,1.0002,3.141592653589793]", options);

    Assert.Equal(expectedJson, actualJson);
    Assert.Equal(expected, actual);

    options = new JsonSerializerOptions();
    options.Converters.Add(new HandleSpecialDoublesAsStrings());

    actualJson = JsonSerializer.Serialize(new double[] { double.PositiveInfinity, double.NegativeInfinity, double.NaN, 0.100, 1.0002, Math.PI }, options);
    // Note, this will write the infinities escaped (as "\u221E")
    // You can override the encoder within options, for example: options.Encoder = JavaScriptEncoder.Create(UnicodeRanges.All);
    Console.WriteLine(actualJson);
    actual = JsonSerializer.Deserialize<double[]>("[\"∞\",\"-∞\",\"NaN\",0.1,1.0002,3.141592653589793]", options);
}

private class HandleSpecialDoublesAsStrings : JsonConverter<double>
{
    public override double Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        if (reader.TokenType == JsonTokenType.String)
        {
            return double.Parse(reader.GetString());
        }
        return reader.GetDouble();
    }

    public override void Write(Utf8JsonWriter writer, double value, JsonSerializerOptions options)
    {
        if (double.IsFinite(value))
        {
            writer.WriteNumberValue(value);
        }
        else
        {
            writer.WriteStringValue(value.ToString());
        }
    }
}

private class HandleSpecialDoublesAsStrings_NewtonsoftCompat : JsonConverter<double>
{
    public override double Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        if (reader.TokenType == JsonTokenType.String)
        {
            string specialDouble = reader.GetString();
            if (specialDouble == "Infinity")
            {
                return double.PositiveInfinity;
            }
            else if (specialDouble == "-Infinity")
            {
                return double.NegativeInfinity;
            }
            else
            {
                return double.NaN;
            }
        }
        return reader.GetDouble();
    }

    private static readonly JsonEncodedText s_nan = JsonEncodedText.Encode("NaN");
    private static readonly JsonEncodedText s_infinity = JsonEncodedText.Encode("Infinity");
    private static readonly JsonEncodedText s_negativeInfinity = JsonEncodedText.Encode("-Infinity");

    public override void Write(Utf8JsonWriter writer, double value, JsonSerializerOptions options)
    {
        if (double.IsFinite(value))
        {
            writer.WriteNumberValue(value);
        }
        else
        {
            if (double.IsPositiveInfinity(value))
            {
                writer.WriteStringValue(s_infinity);
            }
            else if (double.IsNegativeInfinity(value))
            {
                writer.WriteStringValue(s_negativeInfinity);
            }
            else
            {
                writer.WriteStringValue(s_nan);
            }
        }
    }
}

[layomia]

@steveharter @tannergooding @pranavkm @valeriob I believe this issue should be considered as part of https://github.com/dotnet/corefx/issues/39473 which deals with (de)serialization support for quoted numbers. Please re-open if needed.