Fix to #30604 - Implement JSON serialization/deserialization via Utf8JsonReader/Utf8JsonWriter

[maumar]

Using Utf8JsonReader to read JSON data rather than caching it using DOM. This should reduce allocations significantly. Tricky part is that entity materializers are build in a way that assumes we have random access to all the data we need. This is not the case here.
We read JSON data sequentially and can only do it once, and we don't know the order in which we get the data. This is somewhat problematic in case where entity takes argument in the constructor. Those could be at the very end of the JSON string, so we must read all the data before we can instantiate the object, and populate it's properties and do navigation fixup.
This requires us reading all the JSON data, store them in local variables, and only when we are done reading we instantiate the entity and populate all the properties with data stored in those variables. This adds some allocations (specifically navigations).

We also have to disable de-duplication logic - we can't always safely re-read the JSON string, and definitely can't start reading it from arbitrary position, so now we have to add JSON string for every aggregate projected, even if we already project it's parent.

Serialization implementation (i.e. Utf8JsonWriter) is pretty straighforward.

Also fix to #30993 - Query/Json: data corruption for tracking queries with nested json entities, then updating nested entities outside EF and re-querying

Fix is to recognize and modify shaper in case of tracking query, so that nav expansions are not skipped when parent entity is found in Change Tracker. This is necessary to fix alongside streaming, because now we throw exception from reader (unexpected token) if we don't process the entire stream correctly. Before it would be silently ignored apart from the edge case described in the bug.

Fixes #30604
Fixes #30993

[maumar]

@roji @ajcvickers this is ready for review now, there are still some minor cleanups to be done but overall logic is complete. Good luck ;)

[ajcvickers]

This looks reasonable to me, as best as I can tell for this kind of code, but I think @roji should take a look too.

@maumar Can you post some expression debug views, or similar, showing the materialization expressions we are creating?

[roji]

I'll be reviewing this in the coming days for sure.

Can you post some expression debug views, or similar, showing the materialization expressions we are creating?

These would actually be useful as doc comments in the source code too (where necessary/reasonable).

[maumar]

I added fragments that are generated here and there - will add more. But I think the entire output is not feasible as comments in the code - it's 3+ pages of code for the simplest nested scenario - will add some samples for various scenarios on the pr tho

[maumar]

most basic scenario, no tracking

ss.Set<JsonEntityBasic>().Select(x => x.OwnedReferenceRoot.OwnedReferenceBranch.OwnedReferenceLeaf).AsNoTracking()

(queryContext, dataReader, resultContext, resultCoordinator) => 
{
    Stream namelessParameter{0};
    JsonReaderData namelessParameter{1};
    Utf8JsonReaderManager namelessParameter{2};
    object[] namelessParameter{3};
    JsonOwnedLeaf namelessParameter{4};
    namelessParameter{0} = dataReader.IsDBNull(0) ? default(MemoryStream) : new MemoryStream(Encoding.UTF8.GetBytes((string)dataReader.GetFieldValue<object>(0)));
    namelessParameter{1} = namelessParameter{0} == default(MemoryStream) ? default(JsonReaderData) : new JsonReaderData(namelessParameter{0});
    namelessParameter{1} != default(JsonReaderData) ? 
    {
        namelessParameter{2} = new Utf8JsonReaderManager(namelessParameter{1});
        namelessParameter{2}.MoveNext();
        namelessParameter{2}.CaptureState();
    } : default(void);
    namelessParameter{3} = new object[]{ (object)dataReader.GetInt32(1) };
    namelessParameter{4} = ShaperProcessingExpressionVisitor.MaterializeJsonEntity<JsonOwnedLeaf>(
        queryContext: queryContext, 
        keyPropertyValues: namelessParameter{3}, 
        jsonReaderData: namelessParameter{1}, 
        nullable: False, 
        shaper: (queryContext, namelessParameter{5}, namelessParameter{6}) => 
        {
            JsonOwnedLeaf namelessParameter{7};
            return namelessParameter{7} = 
            {
                MaterializationContext materializationContext1;
                IEntityType entityType1;
                JsonOwnedLeaf instance1;
                materializationContext1 = new MaterializationContext(
                    ValueBuffer, 
                    queryContext.Context
                );
                instance1 = null;
                namelessParameter{5}[0] != null ? 
                {
                    ValueBuffer shadowValueBuffer1;
                    shadowValueBuffer1 = ValueBuffer;
                    entityType1 = EntityType: JsonEntityBasic.OwnedReferenceRoot#JsonOwnedRoot.OwnedReferenceBranch#JsonOwnedBranch.OwnedReferenceLeaf#JsonOwnedLeaf CLR Type: JsonOwnedLeaf Owned;
                    instance1 = 
                    {
                        Utf8JsonReaderManager namelessParameter{8};
                        JsonTokenType tokenType;
                        JsonOwnedLeaf instance;
                        string namelessParameter{9};
                        namelessParameter{8} = new Utf8JsonReaderManager(namelessParameter{6});
                        tokenType = namelessParameter{8}.CurrentReader.TokenType;
                        Loop(Break: done Continue: )
                        {
                            {
                                tokenType = namelessParameter{8}.MoveNext();
                                tokenType != EndObject ? switch (tokenType)
                                {
                                    case PropertyName: 
                                        namelessParameter{8}.CurrentReader.ValueTextEquals(SomethingSomething.EncodedUtf8Bytes) ? 
                                        {
                                            namelessParameter{8}.MoveNext();
                                            namelessParameter{9} = namelessParameter{8}.CurrentReader.TokenType == Null ? default(string) : (string)JsonStringReaderWriter.FromJson(namelessParameter{8});
                                        } : default(void)
                                    default: 
                                        {
                                            namelessParameter{8}.CurrentReader.TrySkip();
                                        }
                                }
                                 : Goto(break done)
                                ;
                            }}
                        namelessParameter{8}.CaptureState();
                        instance = new JsonOwnedLeaf();
                        instance.<SomethingSomething>k__BackingField = namelessParameter{9};
                        (instance is IInjectableService) ? ((IInjectableService)instance).Injected(
                            context: materializationContext1.Context, 
                            entity: instance, 
                            bindingInfo: ParameterBindingInfo) : default(void);
                        return instance;
                    };
                    return instance1;
                } : 
                {
                    object[] keyValues1;
                    keyValues1 = new object[]{ namelessParameter{5}[0] };
                    return EntityMaterializerInjectingExpressionVisitor.CreateNullKeyValueInNoTrackingQuery(
                        entityType: EntityType: JsonEntityBasic.OwnedReferenceRoot#JsonOwnedRoot.OwnedReferenceBranch#JsonOwnedBranch.OwnedReferenceLeaf#JsonOwnedLeaf CLR Type: JsonOwnedLeaf Owned, 
                        properties: List<RuntimeProperty> { RuntimeProperty }, 
                        keyValues: keyValues1);
                };
                return instance1;
            };
        });
    return namelessParameter{4};
}

[maumar]

nested scenario (still no-tracking)

ss.Set<JsonEntityBasic>().Select(x => x.OwnedReferenceRoot.OwnedCollectionBranch).AsNoTracking()

(queryContext, dataReader, resultContext, resultCoordinator) => 
{
    Stream namelessParameter{0};
    JsonReaderData namelessParameter{1};
    Utf8JsonReaderManager namelessParameter{2};
    object[] namelessParameter{3};
    List<JsonOwnedBranch> namelessParameter{4};
    namelessParameter{0} = dataReader.IsDBNull(0) ? default(MemoryStream) : new MemoryStream(Encoding.UTF8.GetBytes((string)dataReader.GetFieldValue<object>(0)));
    namelessParameter{1} = namelessParameter{0} == default(MemoryStream) ? default(JsonReaderData) : new JsonReaderData(namelessParameter{0});
    namelessParameter{1} != default(JsonReaderData) ? 
    {
        namelessParameter{2} = new Utf8JsonReaderManager(namelessParameter{1});
        namelessParameter{2}.MoveNext();
        namelessParameter{2}.CaptureState();
    } : default(void);
    namelessParameter{3} = new object[]{ (object)dataReader.GetInt32(1) };
    namelessParameter{4} = ShaperProcessingExpressionVisitor.MaterializeJsonEntityCollection<JsonOwnedBranch, List<JsonOwnedBranch>>(
        queryContext: queryContext, 
        keyPropertyValues: namelessParameter{3}, 
        jsonReaderData: namelessParameter{1}, 
        navigation: Navigation: JsonEntityBasic.OwnedReferenceRoot#JsonOwnedRoot.OwnedCollectionBranch (List<JsonOwnedBranch>) Collection ToDependent JsonEntityBasic.OwnedReferenceRoot#JsonOwnedRoot.OwnedCollectionBranch#JsonOwnedBranch, 
        innerShaper: (queryContext, namelessParameter{5}, namelessParameter{6}) => 
        {
            JsonOwnedBranch namelessParameter{7};
            return namelessParameter{7} = 
            {
                MaterializationContext materializationContext1;
                IEntityType entityType1;
                JsonOwnedBranch instance1;
                materializationContext1 = new MaterializationContext(
                    ValueBuffer, 
                    queryContext.Context
                );
                instance1 = null;
                namelessParameter{5}[0] != null && namelessParameter{5}[1] != null ? 
                {
                    ValueBuffer shadowValueBuffer1;
                    shadowValueBuffer1 = ValueBuffer;
                    entityType1 = EntityType: JsonEntityBasic.OwnedReferenceRoot#JsonOwnedRoot.OwnedCollectionBranch#JsonOwnedBranch CLR Type: JsonOwnedBranch Owned;
                    instance1 = 
                    {
                        Utf8JsonReaderManager namelessParameter{8};
                        JsonTokenType tokenType;
                        JsonOwnedBranch instance;
                        DateTime namelessParameter{9};
                        JsonEnum namelessParameter{10};
                        decimal namelessParameter{11};
                        JsonEnum? namelessParameter{12};
                        List<JsonOwnedLeaf> namelessParameter{13};
                        JsonOwnedLeaf namelessParameter{14};
                        namelessParameter{8} = new Utf8JsonReaderManager(namelessParameter{6});
                        tokenType = namelessParameter{8}.CurrentReader.TokenType;
                        Loop(Break: done Continue: )
                        {
                            {
                                tokenType = namelessParameter{8}.MoveNext();
                                tokenType != EndObject ? switch (tokenType)
                                {
                                    case PropertyName: 
                                        namelessParameter{8}.CurrentReader.ValueTextEquals(Date.EncodedUtf8Bytes) ? 
                                        {
                                            namelessParameter{8}.MoveNext();
                                            namelessParameter{9} = (DateTime)JsonDateTimeReaderWriter.FromJson(namelessParameter{8});
                                        } : namelessParameter{8}.CurrentReader.ValueTextEquals(Enum.EncodedUtf8Bytes) ? 
                                        {
                                            namelessParameter{8}.MoveNext();
                                            namelessParameter{10} = StringEnumConverter<JsonEnum, string, JsonEnum>.ConvertToEnum((string)JsonStringReaderWriter.FromJson(namelessParameter{8}));
                                        } : namelessParameter{8}.CurrentReader.ValueTextEquals(Fraction.EncodedUtf8Bytes) ? 
                                        {
                                            namelessParameter{8}.MoveNext();
                                            namelessParameter{11} = (decimal)JsonDecimalReaderWriter.FromJson(namelessParameter{8});
                                        } : namelessParameter{8}.CurrentReader.ValueTextEquals(NullableEnum.EncodedUtf8Bytes) ? 
                                        {
                                            namelessParameter{8}.MoveNext();
                                            namelessParameter{12} = namelessParameter{8}.CurrentReader.TokenType == Null ? default(JsonEnum?) : (JsonEnum?)StringEnumConverter<JsonEnum, string, JsonEnum>.ConvertToEnum((string)JsonStringReaderWriter.FromJson(namelessParameter{8}));
                                        } : namelessParameter{8}.CurrentReader.ValueTextEquals(OwnedCollectionLeaf.EncodedUtf8Bytes) ? 
                                        {
                                            namelessParameter{8}.MoveNext();
                                            namelessParameter{8}.CaptureState();
                                            namelessParameter{13} = ShaperProcessingExpressionVisitor.MaterializeJsonEntityCollection<JsonOwnedLeaf, List<JsonOwnedLeaf>>(
                                                queryContext: queryContext, 
                                                keyPropertyValues: namelessParameter{5}, 
                                                jsonReaderData: namelessParameter{6}, 
                                                navigation: Navigation: JsonEntityBasic.OwnedReferenceRoot#JsonOwnedRoot.OwnedCollectionBranch#JsonOwnedBranch.OwnedCollectionLeaf (List<JsonOwnedLeaf>) Collection ToDependent JsonEntityBasic.OwnedReferenceRoot#JsonOwnedRoot.OwnedCollectionBranch#JsonOwnedBranch.OwnedCollectionLeaf#JsonOwnedLeaf Inverse: Parent, 
                                                innerShaper: (queryContext, namelessParameter{15}, namelessParameter{16}) => 
                                                {
                                                    JsonOwnedLeaf namelessParameter{17};
                                                    return namelessParameter{17} = 
                                                    {
                                                        MaterializationContext materializationContext2;
                                                        IEntityType entityType2;
                                                        JsonOwnedLeaf instance2;
                                                        materializationContext2 = new MaterializationContext(
                                                            ValueBuffer, 
                                                            queryContext.Context
                                                        );
                                                        instance2 = null;
                                                        namelessParameter{15}[0] != null && namelessParameter{15}[1] != null && namelessParameter{15}[2] != null ? 
                                                        {
                                                            ValueBuffer shadowValueBuffer2;
                                                            shadowValueBuffer2 = ValueBuffer;
                                                            entityType2 = EntityType: JsonEntityBasic.OwnedReferenceRoot#JsonOwnedRoot.OwnedCollectionBranch#JsonOwnedBranch.OwnedCollectionLeaf#JsonOwnedLeaf CLR Type: JsonOwnedLeaf Owned;
                                                            instance2 = 
                                                            {
                                                                Utf8JsonReaderManager namelessParameter{18};
                                                                JsonTokenType tokenType;
                                                                JsonOwnedLeaf instance;
                                                                string namelessParameter{19};
                                                                namelessParameter{18} = new Utf8JsonReaderManager(namelessParameter{16});
                                                                tokenType = namelessParameter{18}.CurrentReader.TokenType;
                                                                Loop(Break: done Continue: )
                                                                {
                                                                    {
                                                                        tokenType = namelessParameter{18}.MoveNext();
                                                                        tokenType != EndObject ? switch (tokenType)
                                                                        {
                                                                            case PropertyName: 
                                                                                namelessParameter{18}.CurrentReader.ValueTextEquals(SomethingSomething.EncodedUtf8Bytes) ? 
                                                                                {
                                                                                    namelessParameter{18}.MoveNext();
                                                                                    namelessParameter{19} = namelessParameter{18}.CurrentReader.TokenType == Null ? default(string) : (string)JsonStringReaderWriter.FromJson(namelessParameter{18});
                                                                                } : default(void)
                                                                            default: 
                                                                                {
                                                                                    namelessParameter{18}.CurrentReader.TrySkip();
                                                                                }
                                                                        }
                                                                         : Goto(break done)
                                                                        ;
                                                                    }}
                                                                namelessParameter{18}.CaptureState();
                                                                instance = new JsonOwnedLeaf();
                                                                instance.<SomethingSomething>k__BackingField = namelessParameter{19};
                                                                (instance is IInjectableService) ? ((IInjectableService)instance).Injected(
                                                                    context: materializationContext2.Context, 
                                                                    entity: instance, 
                                                                    bindingInfo: ParameterBindingInfo) : default(void);
                                                                return instance;
                                                            };
                                                            return instance2;
                                                        } : default(void);
                                                        return instance2;
                                                    };
                                                });
                                            namelessParameter{8} = new Utf8JsonReaderManager(namelessParameter{6});
                                        } : namelessParameter{8}.CurrentReader.ValueTextEquals(OwnedReferenceLeaf.EncodedUtf8Bytes) ? 
                                        {
                                            namelessParameter{8}.MoveNext();
                                            namelessParameter{8}.CaptureState();
                                            namelessParameter{14} = ShaperProcessingExpressionVisitor.MaterializeJsonEntity<JsonOwnedLeaf>(
                                                queryContext: queryContext, 
                                                keyPropertyValues: namelessParameter{5}, 
                                                jsonReaderData: namelessParameter{6}, 
                                                nullable: True, 
                                                shaper: (queryContext, namelessParameter{20}, namelessParameter{21}) => 
                                                {
                                                    JsonOwnedLeaf namelessParameter{22};
                                                    return namelessParameter{22} = 
                                                    {
                                                        MaterializationContext materializationContext3;
                                                        IEntityType entityType3;
                                                        JsonOwnedLeaf instance3;
                                                        materializationContext3 = new MaterializationContext(
                                                            ValueBuffer, 
                                                            queryContext.Context
                                                        );
                                                        instance3 = null;
                                                        namelessParameter{20}[0] != null && namelessParameter{20}[1] != null ? 
                                                        {
                                                            ValueBuffer shadowValueBuffer3;
                                                            shadowValueBuffer3 = ValueBuffer;
                                                            entityType3 = EntityType: JsonEntityBasic.OwnedReferenceRoot#JsonOwnedRoot.OwnedCollectionBranch#JsonOwnedBranch.OwnedReferenceLeaf#JsonOwnedLeaf CLR Type: JsonOwnedLeaf Owned;
                                                            instance3 = 
                                                            {
                                                                Utf8JsonReaderManager namelessParameter{23};
                                                                JsonTokenType tokenType;
                                                                JsonOwnedLeaf instance;
                                                                string namelessParameter{24};
                                                                namelessParameter{23} = new Utf8JsonReaderManager(namelessParameter{21});
                                                                tokenType = namelessParameter{23}.CurrentReader.TokenType;
                                                                Loop(Break: done Continue: )
                                                                {
                                                                    {
                                                                        tokenType = namelessParameter{23}.MoveNext();
                                                                        tokenType != EndObject ? switch (tokenType)
                                                                        {
                                                                            case PropertyName: 
                                                                                namelessParameter{23}.CurrentReader.ValueTextEquals(SomethingSomething.EncodedUtf8Bytes) ? 
                                                                                {
                                                                                    namelessParameter{23}.MoveNext();
                                                                                    namelessParameter{24} = namelessParameter{23}.CurrentReader.TokenType == Null ? default(string) : (string)JsonStringReaderWriter.FromJson(namelessParameter{23});
                                                                                } : default(void)
                                                                            default: 
                                                                                {
                                                                                    namelessParameter{23}.CurrentReader.TrySkip();
                                                                                }
                                                                        }
                                                                         : Goto(break done)
                                                                        ;
                                                                    }}
                                                                namelessParameter{23}.CaptureState();
                                                                instance = new JsonOwnedLeaf();
                                                                instance.<SomethingSomething>k__BackingField = namelessParameter{24};
                                                                (instance is IInjectableService) ? ((IInjectableService)instance).Injected(
                                                                    context: materializationContext3.Context, 
                                                                    entity: instance, 
                                                                    bindingInfo: ParameterBindingInfo) : default(void);
                                                                return instance;
                                                            };
                                                            return instance3;
                                                        } : default(void);
                                                        return instance3;
                                                    };
                                                });
                                            namelessParameter{8} = new Utf8JsonReaderManager(namelessParameter{6});
                                        } : default(void)
                                    default: 
                                        {
                                            namelessParameter{8}.CurrentReader.TrySkip();
                                        }
                                }
                                 : Goto(break done)
                                ;
                            }}
                        namelessParameter{8}.CaptureState();
                        instance = new JsonOwnedBranch();
                        instance.<Date>k__BackingField = namelessParameter{9};
                        instance.<Enum>k__BackingField = namelessParameter{10};
                        instance.<Fraction>k__BackingField = namelessParameter{11};
                        instance.<NullableEnum>k__BackingField = namelessParameter{12};
                        (instance is IInjectableService) ? ((IInjectableService)instance).Injected(
                            context: materializationContext1.Context, 
                            entity: instance, 
                            bindingInfo: ParameterBindingInfo) : default(void);
                        instance != null && namelessParameter{13} != null ? Invoke((namelessParameter{25}, namelessParameter{26}) => 
                        {
                            namelessParameter{25}.<OwnedCollectionLeaf>k__BackingField = namelessParameter{26};
                            ShaperProcessingExpressionVisitor.InverseCollectionFixup<JsonOwnedLeaf, JsonOwnedBranch>(
                                collection: namelessParameter{26}, 
                                entity: namelessParameter{25}, 
                                elementFixup: (namelessParameter{27}, namelessParameter{28}) => 
                                {
                                    return namelessParameter{27}.<Parent>k__BackingField = namelessParameter{28};
                                });
                        }, instance, namelessParameter{13}) : default(void);
                        instance != null && namelessParameter{14} != null ? Invoke((namelessParameter{29}, namelessParameter{30}) => 
                        {
                            namelessParameter{29}.<OwnedReferenceLeaf>k__BackingField = namelessParameter{30};
                            return namelessParameter{30}.<Parent>k__BackingField = namelessParameter{29};
                        }, instance, namelessParameter{14}) : default(void);
                        return instance;
                    };
                    return instance1;
                } : default(void);
                return instance1;
            };
        });
    return namelessParameter{4};
}

[maumar]

tracking query with shadow properties, specifically note the differences between how we built materializer for tracked json entity vs non-json entity - there was a lot of rewrite there

(queryContext, dataReader, resultContext, resultCoordinator) => 
{
    MyEntityShadowProperties namelessParameter{0};
    Stream namelessParameter{1};
    JsonReaderData namelessParameter{2};
    Utf8JsonReaderManager namelessParameter{3};
    object[] namelessParameter{4};
    Stream namelessParameter{5};
    JsonReaderData namelessParameter{6};
    Utf8JsonReaderManager namelessParameter{7};
    object[] namelessParameter{8};
    Stream namelessParameter{9};
    JsonReaderData namelessParameter{10};
    Utf8JsonReaderManager namelessParameter{11};
    object[] namelessParameter{12};
    Stream namelessParameter{13};
    JsonReaderData namelessParameter{14};
    Utf8JsonReaderManager namelessParameter{15};
    object[] namelessParameter{16};
    namelessParameter{0} = 
    {
        MaterializationContext materializationContext1;
        IEntityType entityType1;
        MyEntityShadowProperties instance1;
        InternalEntityEntry entry1;
        bool hasNullKey1;
        materializationContext1 = new MaterializationContext(
            ValueBuffer, 
            queryContext.Context
        );
        instance1 = null;
        entry1 = queryContext.TryGetEntry(
            key: Key: MyEntityShadowProperties.Id PK, 
            keyValues: new object[]{ (object)dataReader.GetInt32(0) }, 
            throwOnNullKey: True, 
            hasNullKey: hasNullKey1);
        !(hasNullKey1) ? entry1 != default(InternalEntityEntry) ? 
        {
            entityType1 = entry1.EntityType;
            return instance1 = (MyEntityShadowProperties)entry1.Entity;
        } : 
        {
            ValueBuffer shadowValueBuffer1;
            shadowValueBuffer1 = ValueBuffer;
            entityType1 = EntityType: MyEntityShadowProperties;
            instance1 = switch (entityType1)
            {
                case EntityType: MyEntityShadowProperties: 
                    {
                        return 
                        {
                            MyEntityShadowProperties instance;
                            instance = new MyEntityShadowProperties();
                            instance.<Id>k__BackingField = dataReader.GetInt32(0);
                            instance.<Name>k__BackingField = dataReader.IsDBNull(1) ? default(string) : (string)dataReader.GetFieldValue<object>(1);
                            (instance is IInjectableService) ? ((IInjectableService)instance).Injected(
                                context: materializationContext1.Context, 
                                entity: instance, 
                                bindingInfo: ParameterBindingInfo) : default(void);
                            return instance;
                        }}
                default: 
                    null
            }
            ;
            entry1 = entityType1 == default(IEntityType) ? default(InternalEntityEntry) : queryContext.StartTracking(
                entityType: entityType1, 
                entity: instance1, 
                valueBuffer: shadowValueBuffer1);
            return instance1;
        } : default(void);
        return instance1;
    };
    namelessParameter{1} = dataReader.IsDBNull(2) ? default(MemoryStream) : new MemoryStream(Encoding.UTF8.GetBytes((string)dataReader.GetFieldValue<object>(2)));
    namelessParameter{2} = namelessParameter{1} == default(MemoryStream) ? default(JsonReaderData) : new JsonReaderData(namelessParameter{1});
    namelessParameter{2} != default(JsonReaderData) ? 
    {
        namelessParameter{3} = new Utf8JsonReaderManager(namelessParameter{2});
        namelessParameter{3}.MoveNext();
        namelessParameter{3}.CaptureState();
    } : default(void);
    namelessParameter{4} = new object[]{ (object)dataReader.GetInt32(0) };
    namelessParameter{5} = dataReader.IsDBNull(3) ? default(MemoryStream) : new MemoryStream(Encoding.UTF8.GetBytes((string)dataReader.GetFieldValue<object>(3)));
    namelessParameter{6} = namelessParameter{5} == default(MemoryStream) ? default(JsonReaderData) : new JsonReaderData(namelessParameter{5});
    namelessParameter{6} != default(JsonReaderData) ? 
    {
        namelessParameter{7} = new Utf8JsonReaderManager(namelessParameter{6});
        namelessParameter{7}.MoveNext();
        namelessParameter{7}.CaptureState();
    } : default(void);
    namelessParameter{8} = new object[]{ (object)dataReader.GetInt32(0) };
    namelessParameter{9} = dataReader.IsDBNull(4) ? default(MemoryStream) : new MemoryStream(Encoding.UTF8.GetBytes((string)dataReader.GetFieldValue<object>(4)));
    namelessParameter{10} = namelessParameter{9} == default(MemoryStream) ? default(JsonReaderData) : new JsonReaderData(namelessParameter{9});
    namelessParameter{10} != default(JsonReaderData) ? 
    {
        namelessParameter{11} = new Utf8JsonReaderManager(namelessParameter{10});
        namelessParameter{11}.MoveNext();
        namelessParameter{11}.CaptureState();
    } : default(void);
    namelessParameter{12} = new object[]{ (object)dataReader.GetInt32(0) };
    namelessParameter{13} = dataReader.IsDBNull(5) ? default(MemoryStream) : new MemoryStream(Encoding.UTF8.GetBytes((string)dataReader.GetFieldValue<object>(5)));
    namelessParameter{14} = namelessParameter{13} == default(MemoryStream) ? default(JsonReaderData) : new JsonReaderData(namelessParameter{13});
    namelessParameter{14} != default(JsonReaderData) ? 
    {
        namelessParameter{15} = new Utf8JsonReaderManager(namelessParameter{14});
        namelessParameter{15}.MoveNext();
        namelessParameter{15}.CaptureState();
    } : default(void);
    namelessParameter{16} = new object[]{ (object)dataReader.GetInt32(0) };
    ShaperProcessingExpressionVisitor.IncludeJsonEntityCollection<MyEntityShadowProperties, MyJsonEntityShadowProperties>(
        queryContext: queryContext, 
        keyPropertyValues: namelessParameter{4}, 
        jsonReaderData: namelessParameter{2}, 
        entity: namelessParameter{0}, 
        innerShaper: (queryContext, namelessParameter{17}, namelessParameter{18}) => 
        {
            MyJsonEntityShadowProperties namelessParameter{19};
            return namelessParameter{19} = 
            {
                MaterializationContext materializationContext2;
                IEntityType entityType2;
                MyJsonEntityShadowProperties instance2;
                InternalEntityEntry entry2;
                bool hasNullKey2;
                materializationContext2 = new MaterializationContext(
                    ValueBuffer, 
                    queryContext.Context
                );
                instance2 = null;
                entry2 = queryContext.TryGetEntry(
                    key: Key: MyEntityShadowProperties.Collection#MyJsonEntityShadowProperties.MyEntityShadowPropertiesId, MyEntityShadowProperties.Collection#MyJsonEntityShadowProperties.Id PK, 
                    keyValues: new object[]
                    { 
                        namelessParameter{17}[0], 
                        namelessParameter{17}[1] 
                    }, 
                    throwOnNullKey: False, 
                    hasNullKey: hasNullKey2);
                !(hasNullKey2) ? 
                {
                    bool namelessParameter{20};
                    ValueBuffer shadowValueBuffer2;
                    namelessParameter{20} = False;
                    shadowValueBuffer2 = ValueBuffer;
                    entityType2 = EntityType: MyEntityShadowProperties.Collection#MyJsonEntityShadowProperties CLR Type: MyJsonEntityShadowProperties Owned;
                    entry2 != default(InternalEntityEntry) ? 
                    {
                        entityType2 = entry2.EntityType;
                        instance2 = (MyJsonEntityShadowProperties)entry2.Entity;
                        namelessParameter{20} = True;
                    } : default(void);
                    instance2 = 
                    {
                        Utf8JsonReaderManager namelessParameter{21};
                        JsonTokenType tokenType;
                        MyJsonEntityShadowProperties instance;
                        object namelessParameter{22};
                        string namelessParameter{23};
                        namelessParameter{21} = new Utf8JsonReaderManager(namelessParameter{18});
                        tokenType = namelessParameter{21}.CurrentReader.TokenType;
                        Loop(Break: done Continue: )
                        {
                            {
                                tokenType = namelessParameter{21}.MoveNext();
                                tokenType != EndObject ? switch (tokenType)
                                {
                                    case PropertyName: 
                                        namelessParameter{21}.CurrentReader.ValueTextEquals(ShadowDouble.EncodedUtf8Bytes) ? 
                                        {
                                            namelessParameter{21}.MoveNext();
                                            namelessParameter{22} = (object)(double)JsonDoubleReaderWriter.FromJson(namelessParameter{21});
                                        } : namelessParameter{21}.CurrentReader.ValueTextEquals(Name.EncodedUtf8Bytes) ? 
                                        {
                                            namelessParameter{21}.MoveNext();
                                            namelessParameter{23} = namelessParameter{21}.CurrentReader.TokenType == Null ? default(string) : (string)JsonStringReaderWriter.FromJson(namelessParameter{21});
                                        } : default(void)
                                    default: 
                                        {
                                            namelessParameter{21}.CurrentReader.TrySkip();
                                        }
                                }
                                 : Goto(break done)
                                ;
                            }}
                        namelessParameter{21}.CaptureState();
                        !(namelessParameter{20}) ? shadowValueBuffer2 = new ValueBuffer(new object[]
                        { 
                            namelessParameter{17}[0], 
                            namelessParameter{17}[1], 
                            namelessParameter{22} 
                        }) : default(void);
                        namelessParameter{20} ? 
                        {
                            instance = instance2;
                        } : 
                        {
                            instance = new MyJsonEntityShadowProperties();
                            instance.<Name>k__BackingField = namelessParameter{23};
                            (instance is IInjectableService) ? ((IInjectableService)instance).Injected(
                                context: materializationContext2.Context, 
                                entity: instance, 
                                bindingInfo: ParameterBindingInfo) : default(void);
                        };
                        return instance;
                    };
                    !(namelessParameter{20} || entityType2 == default(IEntityType)) ? 
                    {
                        queryContext.StartTracking(
                            entityType: entityType2, 
                            entity: instance2, 
                            valueBuffer: shadowValueBuffer2);
                    } : default(void);
                } : default(void);
                return instance2;
            };
        }, 
        fixup: (namelessParameter{24}, namelessParameter{25}) => 
        {
            return ClrICollectionAccessor<MyEntityShadowProperties, List<MyJsonEntityShadowProperties>, MyJsonEntityShadowProperties>.Add(
                entity: namelessParameter{24}, 
                value: namelessParameter{25}, 
                forMaterialization: True);
        });
    ShaperProcessingExpressionVisitor.IncludeJsonEntityCollection<MyEntityShadowProperties, MyJsonEntityShadowPropertiesWithCtor>(
        queryContext: queryContext, 
        keyPropertyValues: namelessParameter{8}, 
        jsonReaderData: namelessParameter{6}, 
        entity: namelessParameter{0}, 
        innerShaper: (queryContext, namelessParameter{26}, namelessParameter{27}) => 
        {
            MyJsonEntityShadowPropertiesWithCtor namelessParameter{28};
            return namelessParameter{28} = 
            {
                MaterializationContext materializationContext3;
                IEntityType entityType3;
                MyJsonEntityShadowPropertiesWithCtor instance3;
                InternalEntityEntry entry3;
                bool hasNullKey3;
                materializationContext3 = new MaterializationContext(
                    ValueBuffer, 
                    queryContext.Context
                );
                instance3 = null;
                entry3 = queryContext.TryGetEntry(
                    key: Key: MyEntityShadowProperties.CollectionWithCtor#MyJsonEntityShadowPropertiesWithCtor.MyEntityShadowPropertiesId, MyEntityShadowProperties.CollectionWithCtor#MyJsonEntityShadowPropertiesWithCtor.Id PK, 
                    keyValues: new object[]
                    { 
                        namelessParameter{26}[0], 
                        namelessParameter{26}[1] 
                    }, 
                    throwOnNullKey: False, 
                    hasNullKey: hasNullKey3);
                !(hasNullKey3) ? 
                {
                    bool namelessParameter{29};
                    ValueBuffer shadowValueBuffer3;
                    namelessParameter{29} = False;
                    shadowValueBuffer3 = ValueBuffer;
                    entityType3 = EntityType: MyEntityShadowProperties.CollectionWithCtor#MyJsonEntityShadowPropertiesWithCtor CLR Type: MyJsonEntityShadowPropertiesWithCtor Owned;
                    entry3 != default(InternalEntityEntry) ? 
                    {
                        entityType3 = entry3.EntityType;
                        instance3 = (MyJsonEntityShadowPropertiesWithCtor)entry3.Entity;
                        namelessParameter{29} = True;
                    } : default(void);
                    instance3 = 
                    {
                        Utf8JsonReaderManager namelessParameter{30};
                        JsonTokenType tokenType;
                        MyJsonEntityShadowPropertiesWithCtor instance;
                        object namelessParameter{31};
                        string namelessParameter{32};
                        namelessParameter{30} = new Utf8JsonReaderManager(namelessParameter{27});
                        tokenType = namelessParameter{30}.CurrentReader.TokenType;
                        Loop(Break: done Continue: )
                        {
                            {
                                tokenType = namelessParameter{30}.MoveNext();
                                tokenType != EndObject ? switch (tokenType)
                                {
                                    case PropertyName: 
                                        namelessParameter{30}.CurrentReader.ValueTextEquals(ShadowNullableByte.EncodedUtf8Bytes) ? 
                                        {
                                            namelessParameter{30}.MoveNext();
                                            namelessParameter{31} = (object)namelessParameter{30}.CurrentReader.TokenType == Null ? default(byte?) : (byte?)(byte)JsonByteReaderWriter.FromJson(namelessParameter{30});
                                        } : namelessParameter{30}.CurrentReader.ValueTextEquals(Name.EncodedUtf8Bytes) ? 
                                        {
                                            namelessParameter{30}.MoveNext();
                                            namelessParameter{32} = namelessParameter{30}.CurrentReader.TokenType == Null ? default(string) : (string)JsonStringReaderWriter.FromJson(namelessParameter{30});
                                        } : default(void)
                                    default: 
                                        {
                                            namelessParameter{30}.CurrentReader.TrySkip();
                                        }
                                }
                                 : Goto(break done)
                                ;
                            }}
                        namelessParameter{30}.CaptureState();
                        !(namelessParameter{29}) ? shadowValueBuffer3 = new ValueBuffer(new object[]
                        { 
                            namelessParameter{26}[0], 
                            namelessParameter{26}[1], 
                            namelessParameter{31} 
                        }) : default(void);
                        namelessParameter{29} ? 
                        {
                            instance = instance3;
                        } : 
                        {
                            instance = new MyJsonEntityShadowPropertiesWithCtor(namelessParameter{32});
                        };
                        return instance;
                    };
                    !(namelessParameter{29} || entityType3 == default(IEntityType)) ? 
                    {
                        queryContext.StartTracking(
                            entityType: entityType3, 
                            entity: instance3, 
                            valueBuffer: shadowValueBuffer3);
                    } : default(void);
                } : default(void);
                return instance3;
            };
        }, 
        fixup: (namelessParameter{33}, namelessParameter{34}) => 
        {
            return ClrICollectionAccessor<MyEntityShadowProperties, List<MyJsonEntityShadowPropertiesWithCtor>, MyJsonEntityShadowPropertiesWithCtor>.Add(
                entity: namelessParameter{33}, 
                value: namelessParameter{34}, 
                forMaterialization: True);
        });
    ShaperProcessingExpressionVisitor.IncludeJsonEntityReference<MyEntityShadowProperties, MyJsonEntityShadowProperties>(
        queryContext: queryContext, 
        keyPropertyValues: namelessParameter{12}, 
        jsonReaderData: namelessParameter{10}, 
        entity: namelessParameter{0}, 
        innerShaper: (queryContext, namelessParameter{35}, namelessParameter{36}) => 
        {
            MyJsonEntityShadowProperties namelessParameter{37};
            return namelessParameter{37} = 
            {
                MaterializationContext materializationContext4;
                IEntityType entityType4;
                MyJsonEntityShadowProperties instance4;
                InternalEntityEntry entry4;
                bool hasNullKey4;
                materializationContext4 = new MaterializationContext(
                    ValueBuffer, 
                    queryContext.Context
                );
                instance4 = null;
                entry4 = queryContext.TryGetEntry(
                    key: Key: MyEntityShadowProperties.Reference#MyJsonEntityShadowProperties.MyEntityShadowPropertiesId PK, 
                    keyValues: new object[]{ namelessParameter{35}[0] }, 
                    throwOnNullKey: False, 
                    hasNullKey: hasNullKey4);
                !(hasNullKey4) ? 
                {
                    bool namelessParameter{38};
                    ValueBuffer shadowValueBuffer4;
                    namelessParameter{38} = False;
                    shadowValueBuffer4 = ValueBuffer;
                    entityType4 = EntityType: MyEntityShadowProperties.Reference#MyJsonEntityShadowProperties CLR Type: MyJsonEntityShadowProperties Owned;
                    entry4 != default(InternalEntityEntry) ? 
                    {
                        entityType4 = entry4.EntityType;
                        instance4 = (MyJsonEntityShadowProperties)entry4.Entity;
                        namelessParameter{38} = True;
                    } : default(void);
                    instance4 = 
                    {
                        Utf8JsonReaderManager namelessParameter{39};
                        JsonTokenType tokenType;
                        MyJsonEntityShadowProperties instance;
                        object namelessParameter{40};
                        string namelessParameter{41};
                        namelessParameter{39} = new Utf8JsonReaderManager(namelessParameter{36});
                        tokenType = namelessParameter{39}.CurrentReader.TokenType;
                        Loop(Break: done Continue: )
                        {
                            {
                                tokenType = namelessParameter{39}.MoveNext();
                                tokenType != EndObject ? switch (tokenType)
                                {
                                    case PropertyName: 
                                        namelessParameter{39}.CurrentReader.ValueTextEquals(ShadowString.EncodedUtf8Bytes) ? 
                                        {
                                            namelessParameter{39}.MoveNext();
                                            namelessParameter{40} = (object)namelessParameter{39}.CurrentReader.TokenType == Null ? default(string) : (string)JsonStringReaderWriter.FromJson(namelessParameter{39});
                                        } : namelessParameter{39}.CurrentReader.ValueTextEquals(Name.EncodedUtf8Bytes) ? 
                                        {
                                            namelessParameter{39}.MoveNext();
                                            namelessParameter{41} = namelessParameter{39}.CurrentReader.TokenType == Null ? default(string) : (string)JsonStringReaderWriter.FromJson(namelessParameter{39});
                                        } : default(void)
                                    default: 
                                        {
                                            namelessParameter{39}.CurrentReader.TrySkip();
                                        }
                                }
                                 : Goto(break done)
                                ;
                            }}
                        namelessParameter{39}.CaptureState();
                        !(namelessParameter{38}) ? shadowValueBuffer4 = new ValueBuffer(new object[]
                        { 
                            namelessParameter{35}[0], 
                            namelessParameter{40} 
                        }) : default(void);
                        namelessParameter{38} ? 
                        {
                            instance = instance4;
                        } : 
                        {
                            instance = new MyJsonEntityShadowProperties();
                            instance.<Name>k__BackingField = namelessParameter{41};
                            (instance is IInjectableService) ? ((IInjectableService)instance).Injected(
                                context: materializationContext4.Context, 
                                entity: instance, 
                                bindingInfo: ParameterBindingInfo) : default(void);
                        };
                        return instance;
                    };
                    !(namelessParameter{38} || entityType4 == default(IEntityType)) ? 
                    {
                        queryContext.StartTracking(
                            entityType: entityType4, 
                            entity: instance4, 
                            valueBuffer: shadowValueBuffer4);
                    } : default(void);
                } : default(void);
                return instance4;
            };
        }, 
        fixup: (namelessParameter{42}, namelessParameter{43}) => 
        {
            return namelessParameter{42}.<Reference>k__BackingField = namelessParameter{43};
        });
    ShaperProcessingExpressionVisitor.IncludeJsonEntityReference<MyEntityShadowProperties, MyJsonEntityShadowPropertiesWithCtor>(
        queryContext: queryContext, 
        keyPropertyValues: namelessParameter{16}, 
        jsonReaderData: namelessParameter{14}, 
        entity: namelessParameter{0}, 
        innerShaper: (queryContext, namelessParameter{44}, namelessParameter{45}) => 
        {
            MyJsonEntityShadowPropertiesWithCtor namelessParameter{46};
            return namelessParameter{46} = 
            {
                MaterializationContext materializationContext5;
                IEntityType entityType5;
                MyJsonEntityShadowPropertiesWithCtor instance5;
                InternalEntityEntry entry5;
                bool hasNullKey5;
                materializationContext5 = new MaterializationContext(
                    ValueBuffer, 
                    queryContext.Context
                );
                instance5 = null;
                entry5 = queryContext.TryGetEntry(
                    key: Key: MyEntityShadowProperties.ReferenceWithCtor#MyJsonEntityShadowPropertiesWithCtor.MyEntityShadowPropertiesId PK, 
                    keyValues: new object[]{ namelessParameter{44}[0] }, 
                    throwOnNullKey: False, 
                    hasNullKey: hasNullKey5);
                !(hasNullKey5) ? 
                {
                    bool namelessParameter{47};
                    ValueBuffer shadowValueBuffer5;
                    namelessParameter{47} = False;
                    shadowValueBuffer5 = ValueBuffer;
                    entityType5 = EntityType: MyEntityShadowProperties.ReferenceWithCtor#MyJsonEntityShadowPropertiesWithCtor CLR Type: MyJsonEntityShadowPropertiesWithCtor Owned;
                    entry5 != default(InternalEntityEntry) ? 
                    {
                        entityType5 = entry5.EntityType;
                        instance5 = (MyJsonEntityShadowPropertiesWithCtor)entry5.Entity;
                        namelessParameter{47} = True;
                    } : default(void);
                    instance5 = 
                    {
                        Utf8JsonReaderManager namelessParameter{48};
                        JsonTokenType tokenType;
                        MyJsonEntityShadowPropertiesWithCtor instance;
                        object namelessParameter{49};
                        string namelessParameter{50};
                        namelessParameter{48} = new Utf8JsonReaderManager(namelessParameter{45});
                        tokenType = namelessParameter{48}.CurrentReader.TokenType;
                        Loop(Break: done Continue: )
                        {
                            {
                                tokenType = namelessParameter{48}.MoveNext();
                                tokenType != EndObject ? switch (tokenType)
                                {
                                    case PropertyName: 
                                        namelessParameter{48}.CurrentReader.ValueTextEquals(ShadowInt.EncodedUtf8Bytes) ? 
                                        {
                                            namelessParameter{48}.MoveNext();
                                            namelessParameter{49} = (object)(int)JsonInt32ReaderWriter.FromJson(namelessParameter{48});
                                        } : namelessParameter{48}.CurrentReader.ValueTextEquals(Name.EncodedUtf8Bytes) ? 
                                        {
                                            namelessParameter{48}.MoveNext();
                                            namelessParameter{50} = namelessParameter{48}.CurrentReader.TokenType == Null ? default(string) : (string)JsonStringReaderWriter.FromJson(namelessParameter{48});
                                        } : default(void)
                                    default: 
                                        {
                                            namelessParameter{48}.CurrentReader.TrySkip();
                                        }
                                }
                                 : Goto(break done)
                                ;
                            }}
                        namelessParameter{48}.CaptureState();
                        !(namelessParameter{47}) ? shadowValueBuffer5 = new ValueBuffer(new object[]
                        { 
                            namelessParameter{44}[0], 
                            namelessParameter{49} 
                        }) : default(void);
                        namelessParameter{47} ? 
                        {
                            instance = instance5;
                        } : 
                        {
                            instance = new MyJsonEntityShadowPropertiesWithCtor(namelessParameter{50});
                        };
                        return instance;
                    };
                    !(namelessParameter{47} || entityType5 == default(IEntityType)) ? 
                    {
                        queryContext.StartTracking(
                            entityType: entityType5, 
                            entity: instance5, 
                            valueBuffer: shadowValueBuffer5);
                    } : default(void);
                } : default(void);
                return instance5;
            };
        }, 
        fixup: (namelessParameter{51}, namelessParameter{52}) => 
        {
            return namelessParameter{51}.<ReferenceWithCtor>k__BackingField = namelessParameter{52};
        });
    return namelessParameter{0};
}

[roji]

Spent some time reviewing this; I'm definitely far from grokking everything that goes on :) But overall it looks good, a a good basis to build on for better perf in the future.

See comments which are almost only nits. I do think we should start systematically assigning names to ParameterExpressions we create, Having to figure out what namelessParameter{7} means in the generated code isn't easy...

[roji]

@ajcvickers naming nit: should this simply be called a converter?

[AndriySvyryd]

Is this line still needed?

[AndriySvyryd]

This can be refactored to share implementation with MaterializeJsonEntityCollection