Skip to content
This repository has been archived by the owner on Jan 23, 2023. It is now read-only.
/ corefx Public archive

Populate non-immutable collections directly on deserialize #41482

Merged
merged 12 commits into from
Oct 19, 2019
Merged

Populate non-immutable collections directly on deserialize #41482

merged 12 commits into from
Oct 19, 2019

Conversation

layomia
Copy link
Contributor

@layomia layomia commented Oct 1, 2019 •
edited
Loading

This PR improves performance collections previously deserialized with the use of converters. This includes types derived from native (System.Collections[.Generic]) collections, and native collections that don't implement ILIst, IDictionary, or IDictionary<>. See #41482 (comment) for the improvements.

The deserialization logic for collections is as follows:

  • Types implementing IList, or are assignable from List<>, are populated in one pass using IList(<>).Add
  • Non dictionary enumerables (do not implement IDictionary, IDictionary<.>, or IReadOnlyDictionary<>) that implement ICollection<>, or are assignable from HashSet<>, are populated with reflection using the void ICollection<>.Add method
  • Non dictionary enumerables that implement Stack<>, Queue<>, Stack, or Queue are populated with reflection using the corresponding void .Push or .Enqueue method.
  • All other non-dictionary enumerables are not supported for deserialization, and the serializer will throw a NSE if deserialization is attempted.
  • Dictionary enumerables that implement one of IDictionary, or IDictionary<,>, or are assignable from Dictionary<,>, are populated using the dictionary indexer.
  • All other dictionaries e.g. types implementing only IReadOnlyDictionary<,>, are not supported for deserialization, and the serializer will throw a NSE if deserialization is attempted.

Fixes https://github.com/dotnet/corefx/issues/41427 - higher order inheritance for derived types is now supported
Fixes https://github.com/dotnet/corefx/issues/40479 - reflection is now used to support more types.
Fixes https://github.com/dotnet/corefx/issues/41034 - types that implement IDictionary are now supported for serialization.

@layomia layomia added this to the 5.0 milestone Oct 1, 2019
@layomia layomia self-assigned this Oct 1, 2019
@steveharter
Copy link
Member

@layomia can you please sync to latest code and include the benchmark results (before\after). Thanks

@layomia
Copy link
Contributor Author

layomia commented Oct 3, 2019 •
edited
Loading

Deserialization

StringToStringDictionaryWrapper (type that implements Dictionary<string, string>)

29% faster, 34.18% decrease in allocations.

Before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 31.89 us 0.147 us 0.138 us 31.89 us 31.65 us 32.16 us 9.3458 1.0104 - 38.38 KB

After

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 24.79 us 0.165 us 0.155 us 24.77 us 24.55 us 25.08 us 6.1435 - - 25.26 KB

ImmutableSortedDictionary

11% faster, no change in allocations

Before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 151.3 us 1.31 us 1.22 us 151.2 us 149.8 us 153.5 us 9.6502 1.2063 - 41.63 KB

After

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 136.5 us 0.36 us 0.34 us 136.4 us 136.0 us 137.1 us 9.7035 1.0782 - 41.63 KB

Hashtable

7% faster, 10.54% decrease in allocations.

Before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 76.98 us 0.772 us 0.722 us 77.31 us 75.60 us 77.67 us 12.1507 - - 49.72 KB

After

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 72.38 us 0.216 us 0.202 us 72.35 us 72.13 us 72.87 us 10.8478 1.7128 - 44.48 KB

Stack<string>

6% faster, 18.04% decrease in allocations.

Before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 13.75 us 0.053 us 0.050 us 13.75 us 13.68 us 13.84 us 2.9248 - - 12.03 KB

After

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 12.98 us 0.056 us 0.050 us 12.98 us 12.89 us 13.06 us 2.3976 0.0521 - 9.86 KB

ICollection

4% faster, no change to allocations

Before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 55.73 us 0.247 us 0.219 us 55.72 us 55.45 us 56.28 us 7.2218 0.2257 - 30.29 KB

After

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 53.71 us 0.158 us 0.140 us 53.72 us 53.49 us 53.93 us 7.3387 0.2158 - 30.29 KB

BinaryData

4% faster, no change to allocations.

Before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 705.0 ns 9.12 ns 8.53 ns 705.7 ns 689.8 ns 718.0 ns 0.2550 - - 1.05 KB

After

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 679.3 ns 6.70 ns 5.94 ns 680.4 ns 669.2 ns 687.5 ns 0.2542 - - 1.05 KB

ArrayList

3% faster, 5.17% decrease in allocations

Before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 57.28 us 0.300 us 0.281 us 57.29 us 56.65 us 57.77 us 7.7750 - - 31.93 KB

After

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 55.40 us 0.220 us 0.206 us 55.40 us 55.11 us 55.75 us 7.2321 0.2192 - 30.29 KB

LoginViewModel

3% faster, no change to allocations

Before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 555.9 ns 5.94 ns 5.56 ns 555.4 ns 548.7 ns 565.5 ns 0.0378 - - 168 B

After

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 540.3 ns 4.97 ns 4.65 ns 538.9 ns 534.8 ns 550.7 ns 0.0387 - - 168 B

Queue

3% faster, 5.02% decrease in allocations.

Before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 57.44 us 0.161 us 0.134 us 57.41 us 57.25 us 57.72 us 7.7063 - - 31.67 KB

After

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 55.83 us 0.182 us 0.170 us 55.80 us 55.54 us 56.17 us 7.1445 - - 30.08 KB

WrapperForIList (custom type that implements IList)

3% faster, 3.75% decrease in allocations

Before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 57.61 us 0.187 us 0.166 us 57.63 us 57.24 us 57.90 us 7.7679 0.2285 - 32.49 KB

After

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 56.23 us 0.602 us 0.533 us 56.04 us 55.71 us 57.65 us 7.3155 - - 30.31 KB

ISet

No chance to execution time, 12.70% decrease in allocations.

Before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 18.04 us 0.068 us 0.063 us 18.02 us 17.90 us 18.15 us 4.1724 0.2877 - 17.09 KB

After

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 18.27 us 0.109 us 0.102 us 18.24 us 18.12 us 18.50 us 3.6478 0.2918 - 14.92 KB

WrapperForIList (type that implements IList)

No change to execution time, 6.71% decrease in allocations

Before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 57.61 us 0.187 us 0.166 us 57.63 us 57.24 us 57.90 us 7.7679 0.2285 - 32.49 KB

After

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 56.23 us 0.602 us 0.533 us 56.04 us 55.71 us 57.65 us 7.3155 - - 30.31 KB

No significant perf changes to ICollection<string>, Location, MyEventsListerViewModel, WrapperForStringList (type that implements List<string>), ImmutableDictionary<string, string>, List<string>, IImmutableList<string>, ImmutableArray<string>, IndexViewModel

SortedList

4% slower, 16.60% decrease in allocations

Before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 94.16 us 0.525 us 0.492 us 94.28 us 93.35 us 95.23 us 11.6541 - - 48.07 KB

After

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
DeserializeFromString 98.22 us 0.256 us 0.240 us 98.16 us 97.81 us 98.76 us 9.5238 1.5873 - 40.09 KB

Serialization

ArrayList

6% faster, no change in allocations

Before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
SerializeToString 17.30 us 0.073 us 0.064 us 17.29 us 17.22 us 17.44 us 1.4557 - - 6 KB

After

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
SerializeToString 16.39 us 0.060 us 0.050 us 16.37 us 16.34 us 16.51 us 1.4417 - - 6 KB

SortedList

4% faster, no change to allocations.

Before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
SerializeToString 32.41 us 0.166 us 0.139 us 32.39 us 32.19 us 32.68 us 4.3911 0.1291 - 18.04 KB

After

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
SerializeToString 31.09 us 0.089 us 0.084 us 31.07 us 30.95 us 31.26 us 4.3679 0.1248 - 18.04 KB

MyEventsListerViewModel

3% faster, no change to allocations

Before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
SerializeToString 568.5 us 3.77 us 3.34 us 568.2 us 562.6 us 574.8 us 92.3788 46.1894 46.1894 386.49 KB

After

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
SerializeToString 548.9 us 1.01 us 0.89 us 549.0 us 547.2 us 550.2 us 92.5110 46.2555 46.2555 386.49 KB

ImmutableSortedDictionary

3% faster, no change to allocations

Before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
SerializeToString 42.04 us 0.103 us 0.096 us 42.03 us 41.88 us 42.25 us 4.3912 - - 18.2 KB

After

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
SerializeToString 40.90 us 0.137 us 0.121 us 40.87 us 40.70 us 41.19 us 4.4017 - - 18.2 KB

ImmutableArray<string>

3% slower, no change to allocations

Before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
SerializeToString 12.54 us 0.055 us 0.052 us 12.53 us 12.48 us 12.68 us 1.4532 - - 6.01 KB

After

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
SerializeToString 12.86 us 0.055 us 0.046 us 12.86 us 12.77 us 12.94 us 1.4346 - - 6.01 KB

LinkedList<string

4% slower

Before

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
SerializeToString 13.26 us 0.036 us 0.034 us 13.26 us 13.21 us 13.32 us 1.4324 - - 6 KB

After

Method Mean Error StdDev Median Min Max Gen 0 Gen 1 Gen 2 Allocated
SerializeToString 13.73 us 0.040 us 0.033 us 13.74 us 13.69 us 13.82 us 1.4251 - - 6 KB

No significant perf changes to serializing Queue, ICollection, ISet<string>, WrapperForListString, Stack<string>, ImmutableDictionary<string, string>, StringICollectionWrapper (type that implements ICollection), StringToStringDictionaryWrapper, HashSet<string>, ICollection<string>.

Execution time summary

Faster

Faster base/diff Base Median (ns) Diff Median (ns) Modality
System.Text.Json.Serialization.Tests.ReadJson.D 1.29 31885.20 24773.73
System.Text.Json.Serialization.Tests.ReadJson<ImmutableSortedDictionary<String, 1.11 151214.23 136378.11
System.Text.Json.Serialization.Tests.ReadJson.DeserializeFromString 1.07 77308.23 72353.13
System.Text.Json.Serialization.Tests.ReadJson<Stack>.DeserializeFromStri 1.06 13751.54 12983.48
System.Text.Json.Serialization.Tests.WriteJson.SerializeToString 1.06 17287.58 16373.20
System.Text.Json.Serialization.Tests.WriteJson.SerializeToString 1.04 32386.89 31074.00
System.Text.Json.Serialization.Tests.WriteJson.SerializeToString 1.04 33390.37 32181.77
System.Text.Json.Serialization.Tests.ReadJson.DeserializeFromString 1.04 55717.64 53715.31
System.Text.Json.Serialization.Tests.ReadJson.DeserializeFromString 1.04 705.66 680.36
System.Text.Json.Serialization.Tests.WriteJson.Serializ 1.03 568226.21 549017.73
System.Text.Json.Serialization.Tests.ReadJson.DeserializeFromString 1.03 57291.56 55403.68
System.Text.Json.Serialization.Tests.ReadJson.DeserializeFromStr 1.03 555.44 538.91
System.Text.Json.Serialization.Tests.ReadJson.DeserializeFromString 1.03 57411.67 55795.76
System.Text.Json.Serialization.Tests.ReadJson.DeserializeFromSt 1.03 57633.80 56039.46
System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, 1.03 42032.97 40874.54

No significant change

Same diff/base Base Median (ns) Diff Median (ns) Modality
System.Text.Json.Serialization.Tests.WriteJson<Stack>.SerializeToString 1.01 13520.86 13327.96
System.Text.Json.Serialization.Tests.ReadJson<ICollection>.DeserializeFr 1.01 11715.28 11565.19
System.Text.Json.Serialization.Tests.ReadJson.DeserializeFromString 1.01 1406.12 1388.53
System.Text.Json.Serialization.Tests.ReadJson.Deseriali 1.01 405604.72 401985.10
System.Text.Json.Serialization.Tests.WriteJson<ImmutableDictionary<String, Strin 1.01 51750.93 51332.06
System.Text.Json.Serialization.Tests.WriteJson.SerializeToString 1.01 427.78 424.38
System.Text.Json.Serialization.Tests.WriteJson.SerializeToString 1.01 1018.08 1010.95
System.Text.Json.Serialization.Tests.WriteJson<List>.SerializeToString 1.01 13306.19 13220.26
System.Text.Json.Serialization.Tests.WriteJson.SerializeToString 1.01 19018.37 18904.12
System.Text.Json.Serialization.Tests.ReadJson<ImmutableArray>.Deserializ 1.01 14556.76 14753.08
System.Text.Json.Serialization.Tests.ReadJson<ISet>.DeserializeFromStrin 1.01 18021.83 18241.62
System.Text.Json.Serialization.Tests.WriteJson.Seriali 1.01 12987.11 13123.09
System.Text.Json.Serialization.Tests.WriteJson<IImmutableList>.Serialize 1.01 19581.13 19749.45
System.Text.Json.Serialization.Tests.ReadJson<List>.DeserializeFromStrin 1.01 11689.88 11785.58
System.Text.Json.Serialization.Tests.WriteJson.SerializeToStrin 1.01 19399.50 19551.79
System.Text.Json.Serialization.Tests.ReadJson<ImmutableDictionary<String, String 1.01 63694.96 64026.68
System.Text.Json.Serialization.Tests.ReadJson.DeserializeF 1.00 11780.75 11825.79
System.Text.Json.Serialization.Tests.WriteJson.SerializeToString 1.02 19278.72 18832.19
System.Text.Json.Serialization.Tests.WriteJson<ISet>.SerializeToString 1.02 13610.87 13376.28
System.Text.Json.Serialization.Tests.WriteJson.SerializeTo 1.02 13335.29 13113.11
System.Text.Json.Serialization.Tests.ReadJson<IList>.DeserializeFromStri 1.02 11499.77 11770.64
System.Text.Json.Serialization.Tests.WriteJson<ICollection>.SerializeToS 1.02 13342.26 13614.54
System.Text.Json.Serialization.Tests.WriteJson<HashSet>.SerializeToStrin 1.02 13560.58 13793.01
System.Text.Json.Serialization.Tests.WriteJson. 1.02 22324.25 22683.23
System.Text.Json.Serialization.Tests.ReadJson.DeserializeFromStr 1.02 35036.74 35586.78

Slower

Slower diff/base Base Median (ns) Diff Median (ns) Modality
System.Text.Json.Serialization.Tests.ReadJson.DeserializeFromString 1.04 94277.78 98155.60
System.Text.Json.Serialization.Tests.WriteJson<LinkedList>.SerializeToSt 1.04 13257.40 13735.20
System.Text.Json.Serialization.Tests.WriteJson<ImmutableArray>.Serialize 1.03 12529.79 12863.22

steveharter
steveharter reviewed Oct 16, 2019
View reviewed changes
...xt.Json/src/System/Text/Json/Serialization/Converters/DefaultImmutableDictionaryConverter.cs Outdated Show resolved Hide resolved
...xt.Json/src/System/Text/Json/Serialization/Converters/DefaultImmutableEnumerableConverter.cs Show resolved Hide resolved
src/System.Text.Json/src/System/Text/Json/Serialization/JsonClassInfo.AddProperty.cs Show resolved Hide resolved
src/System.Text.Json/src/System/Text/Json/Serialization/JsonClassInfo.AddProperty.cs Show resolved Hide resolved
src/System.Text.Json/src/System/Text/Json/Serialization/JsonClassInfo.AddProperty.cs Outdated Show resolved Hide resolved
src/System.Text.Json/tests/NewtonsoftTests/DateTimeConverterTests.cs Outdated Show resolved Hide resolved
src/System.Text.Json/tests/Serialization/Array.ReadTests.cs Outdated Show resolved Hide resolved
src/System.Text.Json/tests/Serialization/CacheTests.cs Outdated Show resolved Hide resolved
src/System.Text.Json/tests/Serialization/NullableTests.cs Outdated Show resolved Hide resolved
src/System.Text.Json/tests/Serialization/Null.WriteTests.cs Outdated Show resolved Hide resolved
@steveharter steveharter mentioned this pull request Oct 18, 2019
Merged
@steveharter
Copy link
Member

steveharter commented Oct 18, 2019 •
edited
Loading

@layomia per #41482 I didn't realize Stack<T> was broken in 3.0 (can't round-trip but that is also the same semantics as Newtonsoft Json.NET).

So support non-generic Stack the same (broken) in this PR for consistency with Stack<T> is probably the best choice.

We need to consider what to do for 5.0 then per that issue - add an option to have "correct" Stack semantics for example or have a breaking change (probably not feasible)

@layomia layomia merged commit 2c06259 into dotnet:master Oct 19, 2019
@layomia layomia mentioned this pull request Oct 22, 2019
Closed
layomia added a commit to layomia/corefx that referenced this pull request Oct 22, 2019
@layomia
Populate non-immutable collections directly on deserialize (dotnet#41482
8e0ad81 
)

* Populate non-immutable collections directly on deserialize

(rather than storing in temporary collections and using converters to
create and populate instances)

* Fix deserilizing nested dictionaries

* Don't get add method for types we can populate without reflection; fix failing tests

* Misc perf changes

* Additional changes

* More changes

* Address review feedback

* Address feedback
@layomia layomia mentioned this pull request Oct 22, 2019
Closed
@layomia layomia mentioned this pull request Nov 9, 2019
Closed
@layomia layomia mentioned this pull request Feb 1, 2020
Closed
@jozkee jozkee mentioned this pull request Feb 4, 2020
Closed
picenka21 pushed a commit to picenka21/runtime that referenced this pull request Feb 18, 2022
@layomia
Populate non-immutable collections directly on deserialize (dotnet/co…
ed51d11 
…refx#41482)

* Populate non-immutable collections directly on deserialize

(rather than storing in temporary collections and using converters to
create and populate instances)

* Fix deserilizing nested dictionaries

* Don't get add method for types we can populate without reflection; fix failing tests

* Misc perf changes

* Additional changes

* More changes

* Address review feedback

* Address feedback


Commit migrated from dotnet/corefx@2c06259
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@JamesNK JamesNK JamesNK left review comments

@jozkee jozkee jozkee left review comments

@Youssef1313 Youssef1313 Youssef1313 left review comments

@steveharter steveharter steveharter approved these changes

@ahsonkhan ahsonkhan Awaiting requested review from ahsonkhan

Assignees

@layomia layomia

Labels
area-System.Text.Json
Projects
None yet
Milestone
5.0
5 participants
@layomia @steveharter @JamesNK @jozkee @Youssef1313