JsonSerializer.Deserialize<T> can't work out my array of dictionaries

[HughPH]

My class is designed to get the items from an Amazon DynamoDB query so that they can be converted to T with a class that implements IEnumerable

    public class DynamoQueryResult<T> where T : class, new()
    {
        public int Count;
        public Dictionary<string, DynamoValue>[] Items;
    }

Newtonsoft can deserialize into this class with no difficulty at all, but JsonSerializer.Deserialize just leaves the array null.

Example json:

{
  "ConsumedCapacity": {
    "CapacityUnits": 1,
    "TableName": "Reply"
  },
  "Count": 2,
  "Items": [
    {
      "ReplyDateTime": {"S": "2015-02-18T20:27:36.165Z"},
      "PostedBy": {"S": "User A"},
      "Id": {"S": "Amazon DynamoDB#DynamoDB Thread 1"}
    },
    {
      "ReplyDateTime": {"S": "2015-02-25T20:27:36.165Z"},
      "PostedBy": {"S": "User B"},
      "Id": {"S": "Amazon DynamoDB#DynamoDB Thread 1"}
    }
  ],
  "ScannedCount": 2
}

Don't blame me for the insane JSON, blame Amazon. This example is lifted directly out of their DynamoDB documentation (and therefore forms part of my unit tests)

Changing from Dictionary<string, DynamoValue>[] to ExpandoObject[] also didn't help, and now I feel dirty.

[davidfowl]

Why is it an array of dictionaries? Which field becomes the string key?(brainfart) Also what's the generic argument for? I don't see it being used in your type definition?

I tried this but got a null ref:

public class DynamoQueryResult
{
    public int Count { get; set; }
    public Dictionary<string, JsonElement>[] Items { get; set; }
}

[HughPH]

Yeah I get a null ref from there.

The generic argument is irrelevant, I just forgot to cut it from my previous issue which was about it being impossible(?) to deserialize into an IEnumerable of my own design. To widen the view: DynamoValue has several fields: S, N, BOOL, SS, NS, B, BS which are for holding data of different types. The example JSON just shows String fields. If the field contains a String, it goes into the S field. If it's a Number, it goes into the N field. If it's a Number Set (array), it goes into the NS field, and so forth. Yeah, I would probably have had a Type and a Value field, but that's not what the DynamoDB developers decided. Once I have all these multi-typed objects, keyed by their names, I then 'collapse' the Dictionary<string, DynamoValue> into a POCO where the name is the POCO's field, and the value is inferred from a combination of the DynamoValue and the POCO field's type.

[HughPH]

Ultimately my workaround has been to get the data as a JsonElement and then write my own stuff to shuffle the JSON into whatever type I've passed. I also added an implicit operator, so I can just assign a JsonElement to a DynamoQueryResult which is once again IEnumerable

[ahsonkhan]

Ultimately my workaround has been to get the data as a JsonElement and then write my own stuff to shuffle the JSON into whatever type I've passed.

When deserializing, we explicitly avoid having the payload define the type to deserialize as (in this case, the DynamoValue is a union of several types and the payload tells you which type it should be returned as), which is why capturing the data as JsonElement and handling it explicitly should work. Your workaround makes sense to me. Out of curiosity, @HughPH, what does your DynamoValue object definition look like? Also, I would be interested in seeing your workaround with using JsonElement, if you could share that code sample.

Yeah I get a null ref from there.

    public class DynamoQueryResult<T> where T : class, new()
    {
        public int Count;
        public Dictionary<string, DynamoValue>[] Items;
    }

Fields are not supported in the current version of the JSON deserializer, which is why neither Count nor Items is being set here (and hence the Items array is null). But this scenario does bring up an issue with the deserializer (once the type being deserialized is changed to contain properties).

I tried this but got a null ref

Yep, that's a bug in how we are handling array of dictionaries.

        [Fact]
        public static void DeserializeToDictionaryArrayDynamo()
        {
            string jsonStr =
@"{
  ""ConsumedCapacity"": {
    ""CapacityUnits"": 1,
    ""TableName"": ""Reply""
  },
  ""Count"": 2,
  ""Items"": [
    {
      ""ReplyDateTime"": { ""S"": ""2015-02-18T20:27:36.165Z""},
      ""PostedBy"": { ""S"": ""User A""},
      ""Id"": { ""S"": ""Amazon DynamoDB#DynamoDB Thread 1""}
    },
    {
      ""ReplyDateTime"": {""S"": ""2015-02-25T20:27:36.165Z""},
      ""PostedBy"": { ""S"": ""User B""},
      ""Id"": { ""S"": ""Amazon DynamoDB#DynamoDB Thread 1""}
    }
  ],
  ""ScannedCount"": 2
}";

            DynamoQueryResult result = JsonSerializer.Deserialize<DynamoQueryResult>(jsonStr);
            Assert.Equal(2, result.Count);
            Assert.Equal(2, result.Items.Length);
        }

        public class DynamoQueryResult
        {
            public int Count { get; set; }
            public Dictionary<string, DynamoValue>[] Items { get; set; }
        }

        public class DynamoValue
        {
            public string S { get; set; }
            public int N { get; set; }
            public int[] NS { get; set; }
            public bool BOOL { get; set; }
        }

System.NullReferenceException : Object reference not set to an instance of an object.
        Stack Trace:
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Serialization\JsonPropertyInfoCommon.cs(96,0): at System.Text.Json.JsonPropertyInfoCommon`4.SetValueAsObject(Object obj, Object value)
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.HandleDictionary.cs(86,0): at System.Text.Json.JsonSerializer.HandleStartDictionary(JsonSerializerOptions options, Utf8JsonReader& reader, ReadStack& state)
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.cs(73,0): at System.Text.Json.JsonSerializer.ReadCore(JsonSerializerOptions options, Utf8JsonReader& reader, ReadStack& readStack)
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.Helpers.cs(17,0): at System.Text.Json.JsonSerializer.ReadCore(Type returnType, JsonSerializerOptions options, Utf8JsonReader& reader)
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.String.cs(74,0): at System.Text.Json.JsonSerializer.ParseCore(String json, Type returnType, JsonSerializerOptions options)
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.String.cs(31,0): at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
          E:\GitHub\Fork\corefx\src\System.Text.Json\tests\Serialization\Value.ReadTests.cs(62,0): at System.Text.Json.Serialization.Tests.ValueTests.ReadPrimitives()

This occurs whenever we have an array of Dictionary<string, TValue> where TValue is another POCO or non-primitive type (like JsonElement). Here's a simplified repro outside of the context of DynamoDB:

        [Fact]
        public static void DeserializeToDictionaryArray()
        {
            string jsonStr =
@"{
  ""Items"": [
    {
      ""property1"": {""Nested"": ""value1""}
    },
    {
      ""property2"": {""Nested"": ""value2""}
    }
  ]
}";
            DictionaryArray result = JsonSerializer.Deserialize<DictionaryArray>(jsonStr);
            Assert.Equal(2, result.Items.Length);
        }

        public class DictionaryArray
        {
            public Dictionary<string, NestedType>[] Items { get; set; }
        }
        public class NestedType
        {
            public string Nested { get; set; }
        }

System.Text.Json.Serialization.Tests.ValueTests.DeserializeToDictionaryArray [FAIL]
        System.NullReferenceException : Object reference not set to an instance of an object.
        Stack Trace:
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Serialization\JsonPropertyInfoCommon.cs(96,0): at System.Text.Json.JsonPropertyInfoCommon`4.SetValueAsObject(Object obj, Object value)
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.HandleDictionary.cs(86,0): at System.Text.Json.JsonSerializer.HandleStartDictionary(JsonSerializerOptions options, Utf8JsonReader& reader, ReadStack& state)
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.cs(73,0): at System.Text.Json.JsonSerializer.ReadCore(JsonSerializerOptions options, Utf8JsonReader& reader, ReadStack& readStack)
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.Helpers.cs(17,0): at System.Text.Json.JsonSerializer.ReadCore(Type returnType, JsonSerializerOptions options, Utf8JsonReader& reader)
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.String.cs(74,0): at System.Text.Json.JsonSerializer.ParseCore(String json, Type returnType, JsonSerializerOptions options)
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.String.cs(31,0): at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
          E:\GitHub\Fork\corefx\src\System.Text.Json\tests\Serialization\Value.ReadTests.cs(28,0): at System.Text.Json.Serialization.Tests.ValueTests.DeserializeToDictionaryArray()

In the following code, the Set Action is null. This is because, from the looks of it, we are incorrectly identifying this case as "ProcessingDictionary" (i.e. IsProcessingDictionary is true) rather than processing it as an array/IEnumerable (where IsProcessingEnumerable would be true).

https://github.com/dotnet/corefx/blob/4ca1feeeb484e8a7089ce8a9d377703ad5b8a53e/src/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.cs#L70-L79

https://github.com/dotnet/corefx/blob/a308e93b1f1303be7eab5ce2b8ed11e66cc8e01e/src/System.Text.Json/src/System/Text/Json/Serialization/JsonPropertyInfoCommon.cs#L89-L98

cc @steveharter, @layomia

Changing from Dictionary<string, DynamoValue>[] to ExpandoObject[] also didn't help, and now I feel dirty.

Here's an issue for ExpandoObject (just as an FYI): https://github.com/dotnet/corefx/issues/38007

[HughPH]

Thanks Ahson!

My code is behind the edit in my previous reply. The real thing has been cleaned up a little, but I still need to address sets (e.g. BS, SS, NS)

Cheers

[ahsonkhan]

Here's a similar issue from https://github.com/dotnet/corefx/issues/41198 by @mmosca

System.Text.Json fail with a NullReferenceException trying to deserialized nested objects/arrays to IEnumerable<IDictionary<>> type.

This sample program will trigger this issues on .Net Core 3.0-rc1:

using System;
using System.Collections.Generic;
using System.Text.Json;

namespace CustomViewSplit
{
    class Program
    {
        static void Main(string[] args)
        {
            string workingJson = @"[
  {
    ""a"": ""A"",
    ""b"" :  ""B""
  }
]";

            string brokenJson = @"[
  {
    ""a"": ""A"",
    ""b"" :  {} 
  }
]";


            foreach (string json in new[] {workingJson, brokenJson})
            {
                byte[] bytes = System.Text.Encoding.UTF8.GetBytes(json);
                ReadOnlySpan<byte> readonlySpan = new ReadOnlySpan<byte>(bytes);

                // This will throw a NullReferenceException for brokenJson and alsoBroken
                var enumerable = JsonSerializer.Deserialize<IEnumerable<IDictionary<string, object>>>(readonlySpan);
            }
        }
    }
}

[layomia]

I can no longer repro the issue with (de)serializing an array of dictionaries. This is likely due to various bug fixes and refactoring efforts we have made since 3.x.

Field support to satisfy the initial concern is being tracked here: #876

[Fact]
public static void DeserializeToDictionaryArrayDynamo()
{
    string jsonStr =
@"{
""ConsumedCapacity"": {
""CapacityUnits"": 1,
""TableName"": ""Reply""
},
""Count"": 2,
""Items"": [
{
""ReplyDateTime"": { ""S"": ""2015-02-18T20:27:36.165Z""},
""PostedBy"": { ""S"": ""User A""},
""Id"": { ""S"": ""Amazon DynamoDB#DynamoDB Thread 1""}
},
{
""ReplyDateTime"": {""S"": ""2015-02-25T20:27:36.165Z""},
""PostedBy"": { ""S"": ""User B""},
""Id"": { ""S"": ""Amazon DynamoDB#DynamoDB Thread 1""}
}
],
""ScannedCount"": 2
}";

    DynamoQueryResult result = JsonSerializer.Deserialize<DynamoQueryResult>(jsonStr);
    Assert.Equal(2, result.Count);
    Assert.Equal(2, result.Items.Length);

    string json = JsonSerializer.Serialize(result, new JsonSerializerOptions { WriteIndented = true });
    Console.WriteLine(json);
}

public class DynamoQueryResult
{
    public int Count { get; set; }
    public Dictionary<string, DynamoValue>[] Items { get; set; }
}

public class DynamoValue
{
    public string S { get; set; }
    public int N { get; set; }
    public int[] NS { get; set; }
    public bool BOOL { get; set; }
}

/*
The output is
{
  "Count": 2,
  "Items": [
    {
      "ReplyDateTime": {
        "S": "2015-02-18T20:27:36.165Z",
        "N": 0,
        "NS": null,
        "BOOL": false
      },
      "PostedBy": {
        "S": "User A",
        "N": 0,
        "NS": null,
        "BOOL": false
      },
      "Id": {
        "S": "Amazon DynamoDB#DynamoDB Thread 1",
        "N": 0,
        "NS": null,
        "BOOL": false
      }
    },
    {
      "ReplyDateTime": {
        "S": "2015-02-25T20:27:36.165Z",
        "N": 0,
        "NS": null,
        "BOOL": false
      },
      "PostedBy": {
        "S": "User B",
        "N": 0,
        "NS": null,
        "BOOL": false
      },
      "Id": {
        "S": "Amazon DynamoDB#DynamoDB Thread 1",
        "N": 0,
        "NS": null,
        "BOOL": false
      }
    }
  ]
  }
*/