Add benchmarks for (de)serializing dictionaries in System.Text.Json #530

layomia · 2019-05-31T14:30:36Z

ahsonkhan

Otherwise, LGTM. Can you please share the perf results from a local run?

I would be interested to see the ToString/ToUtf8Bytes difference just as a data point before we drop one.

ahsonkhan · 2019-06-04T00:34:52Z

src/benchmarks/micro/corefx/System.Text.Json/Serializer/ReadJson.Dictionary.cs

+
+        [BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
+        [Benchmark]
+        public Dictionary<string, string> DeserializeDict()


Maybe there is a way to have a generic benchmark method with the type of the dictionary being a type parameter. All these tests are quite similar.

@adamsitnik, any suggestions on how we could set that up (or do you think the tests are fine as is)?

@layomia have you considered adding the Dictionary and ImmutableDictionary to existing JSON serializer benchmarks?

You would need to add the type here:

performance/src/benchmarks/micro/corefx/System.Text.Json/Serializer/WriteJson.cs

Lines 13 to 17 in 2e05afc

[GenericTypeArguments(typeof(LoginViewModel))]

[GenericTypeArguments(typeof(Location))]

[GenericTypeArguments(typeof(IndexViewModel))]

[GenericTypeArguments(typeof(MyEventsListerViewModel))]

public class WriteJson<T>

and implement the data generation in:
https://github.com/dotnet/performance/blob/master/src/benchmarks/micro/Serializers/DataGenerator.cs

If you do that you can quickly add the deserialization benchmarks to https://github.com/dotnet/performance/blob/master/src/benchmarks/micro/corefx/System.Text.Json/Serializer/ReadJson.cs

ahsonkhan · 2019-06-04T00:35:45Z

src/benchmarks/micro/corefx/System.Text.Json/Serializer/ReadJson.Dictionary.cs

+{
+    public class ReadDictionary
+    {
+        private const string _jsonString = @"{""Hello"":""World"",""Hello2"":""World2""}";


It would be useful to see perf of JSON with a lot more key/value pairs as well (say 100).

I would also strongly recommend a bigger dictionary.

If we try to deserialize a very small one then most probably most of the time will be spend not in the deserializaiton of the dictionary but elsewhere: in searching for the type information, properies metadata etc.

BTW you can use

performance/src/harness/BenchmarkDotNet.Extensions/ValuesGenerator.cs

Line 27 in 2e05afc

public static T[] ArrayOfUniqueValues<T>(int count)

to get unique strings and populate the dictionary:

Dictionary<string, string> dictionary = ValuesGenerator.ArrayOfUniqueValues<string>(100).ToDictionary(value => value);

ahsonkhan · 2019-06-04T00:37:45Z

src/benchmarks/micro/corefx/System.Text.Json/Serializer/WriteJson.Dictionary.cs

+        [Benchmark]
+        public string SerializeIDict_ToString()
+        {
+            return JsonSerializer.ToString(_iDict);


In the interest of avoiding test bloat (and only testing a single dimension), it's probably not useful to add the ToString() tests for all the specialized test cases (like WriteJson.Dictionary), when we have the ToUtf8Bytes tests already. I think the overhead of transcoding the end result to UTF-16 is well understood (and captured by the existing tests).

ahsonkhan · 2019-06-04T00:38:54Z

src/benchmarks/micro/corefx/System.Text.Json/Serializer/WriteJson.Dictionary.cs

+    {
+        private Dictionary<string, string> _dict = new Dictionary<string, string>() { { "Hello", "World" }, { "Hello2", "World2" } };
+        private static IDictionary<string, string> _iDict = new Dictionary<string, string>() { { "Hello", "World" }, { "Hello2", "World2" } };
+        private IReadOnlyDictionary<string, string> _iReadOnlyDict = new Dictionary<string, string>() { { "Hello", "World" }, { "Hello2", "World2" } };


Similarly, writing a larger dictionary would be good to see. You could set it up in the GlobalSetup as well, maybe with a Params for KeyCount, with 2 and 100 as reasonable values.

layomia · 2019-06-04T03:04:36Z

Can you please share the perf results from a local run?

See these sections over at dotnet/corefx#37710:

Deserialization and serialization of Dictionary, IDictionary, IReadOnlyDictionary
Deserialization and serialization of ImmutableDictionary, IImmutableDictionary, ImmutableSortedDictionary

ahsonkhan · 2019-06-04T04:05:48Z

See these sections over at dotnet/corefx#37710:

Gotcha: dotnet/corefx#37710 (comment)

S.T.Json:

SerializeDict	817.5 ns

Json.Net:

SerializeDict_ToString	593.5 ns

How come we are slower for serialization?

It also looks like we are slower for serializing and deserializing immutable collections.
dotnet/corefx#37710 (comment)

We might be able to improve the performance of (de)serializing ImmutableDicitonary, IImmutableDictionary, and ImmutableSortedDictionay by using IL emit to generate high performance CreateRange methods.

We should consider doing this or find some other ways to optimize this. We are quite a bit slower (almost 50%). @steveharter, any ideas?

layomia · 2019-06-04T11:14:05Z

How come we are slower for serialization?

I can spend some time looking at this as part of testing/validation next week and beyond. @steveharter might be able to provide some insight now.

We should consider doing this or find some other ways to optimize this.

Agreed. Json.NET uses IL emit extensively for immutable collections: https://github.com/JamesNK/Newtonsoft.Json/blob/master/Src/Newtonsoft.Json/Utilities/DynamicReflectionDelegateFactory.cs.
My understanding is since immutable collections are not a mainstream scenario for us, we should defer efforts here to any buffer time after testing/validation efforts for mainstream scenarios.

adamsitnik · 2019-06-04T19:35:14Z

src/benchmarks/micro/corefx/System.Text.Json/Serializer/ReadJson.Dictionary.cs

+
+        [BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
+        [Benchmark]
+        public IReadOnlyDictionary<string, string> DeserializeIReadOnlyDict()


is IReadOnlyDictionary testing a different code path than IDictionary or Dictionary? If it's the same code path on the serializer side, I would stay only with Dictionary

adamsitnik · 2019-06-04T19:37:14Z

src/benchmarks/micro/corefx/System.Text.Json/Serializer/ReadJson.Dictionary.cs

+
+        [BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
+        [Benchmark]
+        public IImmutableDictionary<string, string> DeserializeIImmutableDict()


same as above: what is the difference in executed code between ImmutableDictionary and IImmutableDictionary?

adamsitnik · 2019-06-06T20:06:21Z

@layomia ping

layomia · 2019-06-07T06:06:02Z

@layomia ping

Thanks. I'll address the feedback on this shortly and apply it to related PRs.

…ucing new types

adamsitnik · 2019-07-18T11:01:06Z

@layomia similar to other stale PRs (#478, #589) I have cloned your fork, updated the code and pushed some fixes. I need to make sure that we don't regress it for 3.0. PTAL at my changes and the doc that I've recently written https://github.com/dotnet/performance/blob/master/docs/microbenchmark-design-guidelines.md

adamsitnik · 2019-07-18T11:02:19Z

sample results:

dotnet run -c Release -f netcoreapp3.0 --filter *WriteJson<*Dictionary*>* *ReadJson<*Dictionary*>* --join

BenchmarkDotNet=v0.11.3.1003-nightly, OS=Windows 10.0.18362
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=3.0.100-preview7-012697
  [Host]     : .NET Core 3.0.0-preview7-27826-20 (CoreCLR 4.700.19.32603, CoreFX 4.700.19.32613), 64bit RyuJIT
  Job-WTATHE : .NET Core 3.0.0-preview7-27826-20 (CoreCLR 4.700.19.32603, CoreFX 4.700.19.32613), 64bit RyuJIT

IterationTime=250.0000 ms  MaxIterationCount=20  MinIterationCount=15  
WarmupCount=1

Type	Method	Mean	Error	StdDev	Median	Min	Max	Gen 0/1k Op	Gen 1/1k Op	Gen 2/1k Op	Allocated Memory/Op
ReadJson<Dictionary<String, String>>	DeserializeFromString	35.56 us	0.6456 us	0.6039 us	35.69 us	34.61 us	36.92 us	3.9238	0.4204	-	31.06 KB
ReadJson<ImmutableDictionary<String, String>>	DeserializeFromString	82.70 us	1.0845 us	0.9613 us	82.64 us	81.07 us	84.20 us	5.3191	0.6649	-	40.87 KB
ReadJson<ImmutableSortedDictionary<String, String>>	DeserializeFromString	220.47 us	2.6522 us	2.3511 us	219.98 us	217.16 us	224.24 us	6.1620	-	-	47.56 KB
WriteJson<Dictionary<String, String>>	SerializeToString	41.84 us	1.1219 us	1.2470 us	41.38 us	40.39 us	44.78 us	3.1250	-	-	24.66 KB
WriteJson<ImmutableDictionary<String, String>>	SerializeToString	72.75 us	1.1743 us	1.0984 us	72.21 us	71.58 us	74.52 us	3.8507	-	-	31.18 KB
WriteJson<ImmutableSortedDictionary<String, String>>	SerializeToString	65.01 us	1.2080 us	1.1299 us	65.44 us	62.93 us	66.69 us	3.9226	-	-	31.08 KB
ReadJson<Dictionary<String, String>>	DeserializeFromUtf8Bytes	33.07 us	0.5040 us	0.4714 us	32.95 us	32.43 us	33.87 us	3.2538	0.4067	-	25.26 KB
ReadJson<ImmutableDictionary<String, String>>	DeserializeFromUtf8Bytes	79.04 us	1.4724 us	1.3773 us	79.07 us	77.26 us	81.08 us	4.4643	0.3189	-	35.06 KB
ReadJson<ImmutableSortedDictionary<String, String>>	DeserializeFromUtf8Bytes	224.46 us	2.8615 us	2.6767 us	224.78 us	219.11 us	227.70 us	5.4348	-	-	41.75 KB
WriteJson<Dictionary<String, String>>	SerializeToUtf8Bytes	40.16 us	0.7985 us	0.8544 us	40.01 us	38.84 us	42.33 us	2.3615	-	-	18.89 KB
WriteJson<ImmutableDictionary<String, String>>	SerializeToUtf8Bytes	73.09 us	0.7413 us	0.6934 us	73.23 us	71.06 us	73.88 us	3.1977	-	-	25.41 KB
WriteJson<ImmutableSortedDictionary<String, String>>	SerializeToUtf8Bytes	62.99 us	1.1375 us	1.0640 us	62.60 us	61.54 us	64.70 us	3.2762	-	-	25.3 KB
ReadJson<Dictionary<String, String>>	DeserializeFromStream	34.06 us	0.3728 us	0.3488 us	34.08 us	33.52 us	34.62 us	3.1944	0.4167	-	25.33 KB
ReadJson<ImmutableDictionary<String, String>>	DeserializeFromStream	80.15 us	1.2857 us	1.2027 us	79.80 us	78.84 us	82.15 us	4.5337	0.3238	-	35.13 KB
ReadJson<ImmutableSortedDictionary<String, String>>	DeserializeFromStream	223.03 us	3.4515 us	3.2285 us	223.85 us	217.72 us	228.55 us	5.4348	-	-	41.82 KB
WriteJson<Dictionary<String, String>>	SerializeToStream	39.64 us	0.7748 us	0.6470 us	39.58 us	38.77 us	40.57 us	1.6108	-	-	13.09 KB
WriteJson<ImmutableDictionary<String, String>>	SerializeToStream	74.32 us	1.4165 us	1.3250 us	73.81 us	72.72 us	76.63 us	2.3585	-	-	19.6 KB
WriteJson<ImmutableSortedDictionary<String, String>>	SerializeToStream	62.31 us	1.0369 us	0.9700 us	62.76 us	60.89 us	63.57 us	2.5202	-	-	19.5 KB

layomia · 2019-07-18T18:53:44Z

@adamsitnik LGTM, thanks!

layomia added 3 commits May 30, 2019 16:15

Add micro benchmarks for (de)serializing dictionaries in S.T.Json

3580c18

Merge remote-tracking branch 'upstream/master' into dict

e4e8ed5

Add serialize ToString benchmarks

e5727ad

layomia mentioned this pull request May 31, 2019

Add support for immutable dictionaries dotnet/corefx#37710

Merged

layomia changed the title ~~Add benchmarks for (de)serializing dictionaries in S.T.Json~~ Add benchmarks for (de)serializing dictionaries in System.Text.Json May 31, 2019

Remove extraneous namespace prefix

b0bc1b8

ahsonkhan approved these changes Jun 4, 2019

View reviewed changes

adamsitnik reviewed Jun 4, 2019

View reviewed changes

ahsonkhan mentioned this pull request Jun 5, 2019

Add benchmarks for ISet<T> (de)serialization and general framework fo… #540

Merged

adamsitnik added 3 commits July 18, 2019 12:33

Merge remote-tracking branch 'upstream/master' into dict

3c50bba

reflect the API changes

0f27991

use existing ReadJson and WriteJson generic classes instead od introd…

5f71efa

…ucing new types

adamsitnik merged commit a6c5770 into dotnet:master Jul 18, 2019

adamsitnik mentioned this pull request Jul 18, 2019

Add benchmarks for (de)serializing non-generic IEnumerable, IList, ICollection #547

Merged

layomia deleted the dict branch July 18, 2019 18:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add benchmarks for (de)serializing dictionaries in System.Text.Json #530

Add benchmarks for (de)serializing dictionaries in System.Text.Json #530

layomia commented May 31, 2019 •

edited

ahsonkhan left a comment

ahsonkhan Jun 4, 2019

adamsitnik Jun 4, 2019

ahsonkhan Jun 4, 2019

adamsitnik Jun 4, 2019 •

edited

ahsonkhan Jun 4, 2019

ahsonkhan Jun 4, 2019

layomia commented Jun 4, 2019

ahsonkhan commented Jun 4, 2019

layomia commented Jun 4, 2019

adamsitnik Jun 4, 2019

adamsitnik Jun 4, 2019

adamsitnik commented Jun 6, 2019

layomia commented Jun 7, 2019

adamsitnik commented Jul 18, 2019

adamsitnik commented Jul 18, 2019

layomia commented Jul 18, 2019

	[GenericTypeArguments(typeof(LoginViewModel))]
	[GenericTypeArguments(typeof(Location))]
	[GenericTypeArguments(typeof(IndexViewModel))]
	[GenericTypeArguments(typeof(MyEventsListerViewModel))]
	public class WriteJson<T>

Add benchmarks for (de)serializing dictionaries in System.Text.Json #530

Add benchmarks for (de)serializing dictionaries in System.Text.Json #530

Conversation

layomia commented May 31, 2019 • edited

ahsonkhan left a comment

Choose a reason for hiding this comment

ahsonkhan Jun 4, 2019

Choose a reason for hiding this comment

adamsitnik Jun 4, 2019

Choose a reason for hiding this comment

ahsonkhan Jun 4, 2019

Choose a reason for hiding this comment

adamsitnik Jun 4, 2019 • edited

Choose a reason for hiding this comment

ahsonkhan Jun 4, 2019

Choose a reason for hiding this comment

ahsonkhan Jun 4, 2019

Choose a reason for hiding this comment

layomia commented Jun 4, 2019

ahsonkhan commented Jun 4, 2019

layomia commented Jun 4, 2019

adamsitnik Jun 4, 2019

Choose a reason for hiding this comment

adamsitnik Jun 4, 2019

Choose a reason for hiding this comment

adamsitnik commented Jun 6, 2019

layomia commented Jun 7, 2019

adamsitnik commented Jul 18, 2019

adamsitnik commented Jul 18, 2019

layomia commented Jul 18, 2019

layomia commented May 31, 2019 •

edited

adamsitnik Jun 4, 2019 •

edited