Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add benchmarks for (de)serializing dictionaries in System.Text.Json #530

Merged
merged 7 commits into from
Jul 18, 2019
Merged
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using BenchmarkDotNet.Attributes;
using MicroBenchmarks;
using System.Collections.Generic;
using System.Collections.Immutable;

namespace System.Text.Json.Serialization.Tests
{
public class ReadDictionary
{
private const string _jsonString = @"{""Hello"":""World"",""Hello2"":""World2""}";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be useful to see perf of JSON with a lot more key/value pairs as well (say 100).

Copy link
Member

@adamsitnik adamsitnik Jun 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also strongly recommend a bigger dictionary.

If we try to deserialize a very small one then most probably most of the time will be spend not in the deserializaiton of the dictionary but elsewhere: in searching for the type information, properies metadata etc.

BTW you can use

public static T[] ArrayOfUniqueValues<T>(int count)
to get unique strings and populate the dictionary:

Dictionary<string, string> dictionary = ValuesGenerator.ArrayOfUniqueValues<string>(100).ToDictionary(value => value);


[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark]
public Dictionary<string, string> DeserializeDict()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe there is a way to have a generic benchmark method with the type of the dictionary being a type parameter. All these tests are quite similar.

@adamsitnik, any suggestions on how we could set that up (or do you think the tests are fine as is)?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@layomia have you considered adding the Dictionary and ImmutableDictionary to existing JSON serializer benchmarks?

You would need to add the type here:

[GenericTypeArguments(typeof(LoginViewModel))]
[GenericTypeArguments(typeof(Location))]
[GenericTypeArguments(typeof(IndexViewModel))]
[GenericTypeArguments(typeof(MyEventsListerViewModel))]
public class WriteJson<T>

and implement the data generation in:
https://github.com/dotnet/performance/blob/master/src/benchmarks/micro/Serializers/DataGenerator.cs

If you do that you can quickly add the deserialization benchmarks to https://github.com/dotnet/performance/blob/master/src/benchmarks/micro/corefx/System.Text.Json/Serializer/ReadJson.cs

{
return JsonSerializer.Parse<Dictionary<string, string>>(_jsonString);
}

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark]
public IDictionary<string, string> DeserializeIDict()
{
return JsonSerializer.Parse<IDictionary<string, string>>(_jsonString);
}

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark]
public IReadOnlyDictionary<string, string> DeserializeIReadOnlyDict()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is IReadOnlyDictionary testing a different code path than IDictionary or Dictionary? If it's the same code path on the serializer side, I would stay only with Dictionary

{
return JsonSerializer.Parse<IReadOnlyDictionary<string, string>>(_jsonString);
}

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark]
public ImmutableDictionary<string, string> DeserializeImmutableDict()
{
return JsonSerializer.Parse<ImmutableDictionary<string, string>>(_jsonString);
}

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark]
public IImmutableDictionary<string, string> DeserializeIImmutableDict()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above: what is the difference in executed code between ImmutableDictionary and IImmutableDictionary?

{
return JsonSerializer.Parse<IImmutableDictionary<string, string>>(_jsonString);
}

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark]
public ImmutableSortedDictionary<string, string> DeserializeImmutableSortedDict()
{
return JsonSerializer.Parse<ImmutableSortedDictionary<string, string>>(_jsonString);
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using BenchmarkDotNet.Attributes;
using MicroBenchmarks;
using System.Collections.Generic;
using System.Collections.Immutable;

namespace System.Text.Json.Serialization.Tests
{
public class WriteDictionary
{
private Dictionary<string, string> _dict = new Dictionary<string, string>() { { "Hello", "World" }, { "Hello2", "World2" } };
private static IDictionary<string, string> _iDict = new Dictionary<string, string>() { { "Hello", "World" }, { "Hello2", "World2" } };
private IReadOnlyDictionary<string, string> _iReadOnlyDict = new Dictionary<string, string>() { { "Hello", "World" }, { "Hello2", "World2" } };
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly, writing a larger dictionary would be good to see. You could set it up in the GlobalSetup as well, maybe with a Params for KeyCount, with 2 and 100 as reasonable values.


private static ImmutableDictionary<string, string> _immutableDict;
private static IImmutableDictionary<string, string> _iimmutableDict;
private static ImmutableSortedDictionary<string, string> _immutableSortedDict;

[GlobalSetup]
public void Setup()
{
_immutableDict = ImmutableDictionary.CreateRange(_dict);
_iimmutableDict = ImmutableDictionary.CreateRange(_dict);
_immutableSortedDict = ImmutableSortedDictionary.CreateRange(_dict);
}

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark]
public byte[] SerializeDict_ToUtf8Bytes()
{
return JsonSerializer.ToUtf8Bytes(_dict);
}

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark]
public byte[] SerializeIDict_ToUtf8Bytes()
{
return JsonSerializer.ToUtf8Bytes(_iDict);
}

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark]
public byte[] SerializeIReadOnlyDict_ToUtf8Bytes()
{
return JsonSerializer.ToUtf8Bytes(_iReadOnlyDict);
}

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark]
public string SerializeDict_ToString()
{
return JsonSerializer.ToString(_dict);
}

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark]
public string SerializeIDict_ToString()
{
return JsonSerializer.ToString(_iDict);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the interest of avoiding test bloat (and only testing a single dimension), it's probably not useful to add the ToString() tests for all the specialized test cases (like WriteJson.Dictionary), when we have the ToUtf8Bytes tests already. I think the overhead of transcoding the end result to UTF-16 is well understood (and captured by the existing tests).

}

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark]
public string SerializeIReadOnlyDict_ToString()
{
return JsonSerializer.ToString(_iReadOnlyDict);
}

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark]
public byte[] SerializeImmutableDict_ToBytes()
{
return JsonSerializer.ToUtf8Bytes(_immutableDict);
}

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark]
public byte[] SerializeIImmutableDict_ToBytes()
{
return JsonSerializer.ToUtf8Bytes(_iimmutableDict);
}

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark]
public byte[] SerializeImmutableSortedDict_ToBytes()
{
return JsonSerializer.ToUtf8Bytes(_immutableSortedDict);
}

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark]
public string SerializeImmutableDict_ToString()
{
return JsonSerializer.ToString(_immutableDict);
}

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark]
public string SerializeIImmutableDict_ToString()
{
return JsonSerializer.ToString(_iimmutableDict);
}

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark]
public string SerializeImmutableSortedDict_ToString()
{
return JsonSerializer.ToString(_immutableSortedDict);
}
}
}