Materialization uses incorrect column value with multiple derived types and shadow properties

[tiefling]

Steps to reproduce

Load some data from your DbContext as follows...

            var contactInfo = (from contact in Context.Contacts
                                where contact.UserName == user.UserName
                                select contact).FirstOrDefault();

Here's the definition for Contact, it's parent and it's children...

public class Contact : Person
    {
        public override string UserName { get; set; }
        public bool IsPrimary { get; set; }
    }

    public abstract class Person
    {
        public int ID { get; set; }

        // ReSharper disable once UnusedMemberInSuper.Global
        public virtual string UserName { get; set; }

        [Required]
        [MaxLength(50)]
        public string FirstName { get; set; }

        [Required]
        [MaxLength(50)]
        public string LastName { get; set; }

        [Required]
        [EmailAddress]
        [DataType(DataType.EmailAddress)]
        public string Email { get; set; }

        [DataType(DataType.Date)]
        public DateTime CreatedDate { get; set; }

        [NotMapped]
        public string FullName => FirstName + " " + LastName;

        public override string ToString() => FullName;
    }

    public class EmployerContact : Contact
    {
        [Required]
        public Employer Employer { get; set; }
    }

    public class ServiceOperatorContact : Contact
    {
        [Required]
        public ServiceOperator ServiceOperator { get; set; }
    }

    public class SystemProviderContact : Contact
    {
        [Required]
        public SystemProvider SystemProvider { get; set; }
    }
    public class VehicleOperatorContact : Contact
    {
        [Required]
        public VehicleOperator VehicleOperator { get; set; }
    }

And my DbContext (editied)...

   public class ApplicationDbContext : IdentityDbContext<ApplicationUser>
    {
        public ApplicationDbContext(DbContextOptions<ApplicationDbContext> options)
            : base(options)
        {   }

        public DbSet<Contact> Contacts { get; set; }
        public DbSet<EmployerContact> EmployerContacts { get; set; }
        public DbSet<ServiceOperatorContact> ServiceOperatorContacts { get; set; }
        public DbSet<SystemProviderContact> SystemProviderContacts { get; set; }
        public DbSet<VehicleOperatorContact> VehicleOperatorContacts { get; set; }
    }

The issue

I am unable to load any data from my Contacts table in the database. Contacts is a base class with 4 inheriting types (all of which should be accessible). The only null fields in the database are the ID fields that are specific to each child class.

I CAN load data using the DbSet properties for each of the individual child classes with no problems whatsoever.

Here is the stack trace pulled from the Debug window by calling
Context.Contacts.First()
before allowing the line above to execute.

Microsoft.EntityFrameworkCore.Storage.IRelationalCommandBuilderFactory:Information: Executed DbCommand (0ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT TOP(1) [c].[ID], [c].[CreatedDate], [c].[Discriminator], [c].[Email], [c].[FirstName], [c].[IsPrimary], [c].[LastName], [c].[UserName], [c].[EmployerID], [c].[ServiceOperatorID], [c].[SystemProviderID], [c].[VehicleOperatorID]
FROM [Contacts] AS [c]
WHERE [c].[Discriminator] IN (N'VehicleOperatorContact', N'SystemProviderContact', N'ServiceOperatorContact', N'EmployerContact', N'Contact')
Microsoft.EntityFrameworkCore.Query.Internal.SqlServerQueryCompilationContextFactory:Error: An exception occurred in the database while iterating the results of a query.
System.NullReferenceException: Object reference not set to an instance of an object.
   at lambda_method(Closure , ValueBuffer )
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalMixedEntityEntry..ctor(IStateManager stateManager, IEntityType entityType, Object entity, ValueBuffer valueBuffer)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntityEntryFactory.NewInternalEntityEntry(IStateManager stateManager, IEntityType entityType, Object entity, ValueBuffer valueBuffer)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntityEntryFactory.Create(IStateManager stateManager, IEntityType entityType, Object entity, ValueBuffer valueBuffer)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.StartTrackingFromQuery(IEntityType baseEntityType, Object entity, ValueBuffer valueBuffer, ISet`1 handledForeignKeys)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityTrackingInfo.StartTracking(IStateManager stateManager, Object entity, ValueBuffer valueBuffer)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryBuffer.StartTracking(Object entity, EntityTrackingInfo entityTrackingInfo)
   at Microsoft.EntityFrameworkCore.Query.QueryContext.StartTracking(Object entity, EntityTrackingInfo entityTrackingInfo)
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.<_TrackEntities>d__15`2.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()

System.NullReferenceException: Object reference not set to an instance of an object.
   at lambda_method(Closure , ValueBuffer )
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalMixedEntityEntry..ctor(IStateManager stateManager, IEntityType entityType, Object entity, ValueBuffer valueBuffer)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntityEntryFactory.NewInternalEntityEntry(IStateManager stateManager, IEntityType entityType, Object entity, ValueBuffer valueBuffer)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntityEntryFactory.Create(IStateManager stateManager, IEntityType entityType, Object entity, ValueBuffer valueBuffer)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.StartTrackingFromQuery(IEntityType baseEntityType, Object entity, ValueBuffer valueBuffer, ISet`1 handledForeignKeys)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityTrackingInfo.StartTracking(IStateManager stateManager, Object entity, ValueBuffer valueBuffer)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryBuffer.StartTracking(Object entity, EntityTrackingInfo entityTrackingInfo)
   at Microsoft.EntityFrameworkCore.Query.QueryContext.StartTracking(Object entity, EntityTrackingInfo entityTrackingInfo)
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.<_TrackEntities>d__15`2.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()
Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler:Error: An exception occurred in the database while iterating the results of a query.
System.NullReferenceException: Object reference not set to an instance of an object.
   at lambda_method(Closure , ValueBuffer )
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalMixedEntityEntry..ctor(IStateManager stateManager, IEntityType entityType, Object entity, ValueBuffer valueBuffer)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntityEntryFactory.NewInternalEntityEntry(IStateManager stateManager, IEntityType entityType, Object entity, ValueBuffer valueBuffer)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntityEntryFactory.Create(IStateManager stateManager, IEntityType entityType, Object entity, ValueBuffer valueBuffer)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.StartTrackingFromQuery(IEntityType baseEntityType, Object entity, ValueBuffer valueBuffer, ISet`1 handledForeignKeys)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityTrackingInfo.StartTracking(IStateManager stateManager, Object entity, ValueBuffer valueBuffer)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryBuffer.StartTracking(Object entity, EntityTrackingInfo entityTrackingInfo)
   at Microsoft.EntityFrameworkCore.Query.QueryContext.StartTracking(Object entity, EntityTrackingInfo entityTrackingInfo)
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.<_TrackEntities>d__15`2.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()
   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass19_1`1.<CompileQuery>b__1(QueryContext qc)

System.NullReferenceException: Object reference not set to an instance of an object.
   at lambda_method(Closure , ValueBuffer )
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalMixedEntityEntry..ctor(IStateManager stateManager, IEntityType entityType, Object entity, ValueBuffer valueBuffer)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntityEntryFactory.NewInternalEntityEntry(IStateManager stateManager, IEntityType entityType, Object entity, ValueBuffer valueBuffer)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntityEntryFactory.Create(IStateManager stateManager, IEntityType entityType, Object entity, ValueBuffer valueBuffer)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.StartTrackingFromQuery(IEntityType baseEntityType, Object entity, ValueBuffer valueBuffer, ISet`1 handledForeignKeys)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityTrackingInfo.StartTracking(IStateManager stateManager, Object entity, ValueBuffer valueBuffer)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryBuffer.StartTracking(Object entity, EntityTrackingInfo entityTrackingInfo)
   at Microsoft.EntityFrameworkCore.Query.QueryContext.StartTracking(Object entity, EntityTrackingInfo entityTrackingInfo)
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.<_TrackEntities>d__15`2.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()
   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass19_1`1.<CompileQuery>b__1(QueryContext qc)
'Context.Contacts.First()' threw an exception of type 'System.NullReferenceException'
    Data: {System.Collections.ListDictionaryInternal}
    HResult: -2147467261
    HelpLink: null
    InnerException: null
    Message: "Object reference not set to an instance of an object."
    Source: "Anonymously Hosted DynamicMethods Assembly"
    StackTrace: "   at lambda_method(Closure , ValueBuffer )\r\n   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalMixedEntityEntry..ctor(IStateManager stateManager, IEntityType entityType, Object entity, ValueBuffer valueBuffer)\r\n   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntityEntryFactory.NewInternalEntityEntry(IStateManager stateManager, IEntityType entityType, Object entity, ValueBuffer valueBuffer)\r\n   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntityEntryFactory.Create(IStateManager stateManager, IEntityType entityType, Object entity, ValueBuffer valueBuffer)\r\n   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.StartTrackingFromQuery(IEntityType baseEntityType, Object entity, ValueBuffer valueBuffer, ISet`1 handledForeignKeys)\r\n   at Microsoft.EntityFrameworkCore.Query.Internal.EntityTrackingInfo.StartTracking(IStateManager stateManager, Object entity, ValueBuffer valueBuffer)\r\n   at Microsoft.EntityFrameworkC
ore.Query.Internal.QueryBuffer.StartTracking(Object entity, EntityTrackingInfo entityTrackingInfo)\r\n   at Microsoft.EntityFrameworkCore.Query.QueryContext.StartTracking(Object entity, EntityTrackingInfo entityTrackingInfo)\r\n   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.<_TrackEntities>d__15`2.MoveNext()\r\n   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()\r\n   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)\r\n   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass19_1`1.<CompileQuery>b__1(QueryContext qc)\r\n   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.Execute[TResult](Expression query)\r\n   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.Execute[TResult](Expression expression)\r\n   at System.Linq.Queryable.First[TSource](IQueryable`1 source)"
    TargetSite: {Microsoft.EntityFrameworkCore.ChangeTracking.Internal.ISnapshot lambda_method(System.Runtime.CompilerServices.Closure, Microsoft.EntityFrameworkCore.Storage.ValueBuffer)}

Further technical details

EF Core version: 1.1.0-preview1-final
Operating system: Windows 10
Visual Studio version: VS2015

Other details about my project setup: See above or on request

[smitpatel]

I am not able to repro this based on data provided above.
Based on exception message query is returning null value where EF does not expect a null.
Since this could be based on data can you post results of this query:

SELECT TOP(1) [c].[ID], [c].[CreatedDate], [c].[Discriminator], [c].[Email], [c].[FirstName], [c].[IsPrimary], [c].[LastName], [c].[UserName], [c].[EmployerID], [c].[ServiceOperatorID], [c].[SystemProviderID], [c].[VehicleOperatorID]
FROM [Contacts] AS [c]
WHERE [c].[Discriminator] IN (N'VehicleOperatorContact', N'SystemProviderContact', N'ServiceOperatorContact', N'EmployerContact', N'Contact')

[tiefling]

I'll get that to you ASAP. The only nulls in the data are with the Ids that are specific to each child class of contact. I think that is where the bug lies.

[tiefling]

Output from the query and also a list of all of the data in the table in the same format is attached...

I can create the data fine, whether I create a Contact or any of it's children (an EmployerContact for example). The data is being created in my DbInitializer as follows...

        private static void SeedContacts(ApplicationDbContext context, IHostingEnvironment env)
        {
            var systemProvider = context.SystemProviders.First();

            context.Contacts.AddRange(
                new SystemProviderContact
                {
                    UserName = "keith.jackson@esoterix.co.uk",
                    FirstName = "Keith",
                    LastName = "Jackson",
                    Email = "keith.jackson@esoterix.co.uk",
                    CreatedDate = DateTime.Now,
                    SystemProvider = systemProvider
                },
                new SystemProviderContact
                {
                    UserName = "david.stewart@esoterix.co.uk",
                    FirstName = "David",
                    LastName = "Stewart",
                    Email = "david.stewart@esoterix.co.uk",
                    CreatedDate = DateTime.Now,
                    SystemProvider = systemProvider
                });

            if (env.IsDevelopment())
                context.Contacts.AddRange(
                    new ServiceOperatorContact
                    {
                        UserName = "service.operator@esoterix.co.uk",
                        FirstName = "Service",
                        LastName = "Operator",
                        Email = "service.operator@esoterix.co.uk",
                        CreatedDate = DateTime.Now,
                        ServiceOperator = context.ServiceOperators.First()
                    },
                    new VehicleOperatorContact
                    {
                        UserName = "vehicle.operator@esoterix.co.uk",
                        FirstName = "Vehicle",
                        LastName = "Operator",
                        Email = "vehicle.operator@esoterix.co.uk",
                        CreatedDate = DateTime.Now,
                        VehicleOperator = context.VehicleOperators.First()
                    },
                    new EmployerContact
                    {
                        UserName = "uwe@esoterix.co.uk",
                        FirstName = "John",
                        LastName = "Parkin",
                        Email = "uwe@esoterix.co.uk",
                        CreatedDate = DateTime.Now,
                        Employer = context.Employers.Where(e => e.Name == "UWE").First()
                    },
                    new EmployerContact
                    {
                        UserName = "hp@esoterix.co.uk",
                        FirstName = "HP",
                        LastName = "Sauce",
                        Email = "hp@esoterix.co.uk",
                        CreatedDate = DateTime.Now,
                        Employer = context.Employers.Where(e => e.Name == "Hewlett Packard").First()
                    },
                    new EmployerContact
                    {
                        UserName = "aviva@esoterix.co.uk",
                        FirstName = "Parkway",
                        LastName = "Livings",
                        Email = "aviva@esoterix.co.uk",
                        CreatedDate = DateTime.Now,
                        Employer = context.Employers.Where(e => e.Name == "Aviva").First()
                    },
                    new Contact
                    {
                        UserName = "noroles@esoterix.co.uk",
                        FirstName = "Billy",
                        LastName = "Nomates",
                        Email = "noroles@esoterix.co.uk",
                        CreatedDate = DateTime.Now
                    });
            context.SaveChanges();
        }

[smitpatel]

Data-corruption issue with shadow properties. It throws in this case due to presence of Required attribute.

[tiefling]

I've refactored my design to unblock myself for now by using a composition rather than inheritance approach to the different contact types. I've kept a tag and branch live of the original code state though, so if I can help with anything more don't hesitate to give me a shout.

[xrkolovos]

Is there any workaround for this issue? (without changing inheritance to composition)

[divega]

Clearing up milestone to discuss bringing this into 1.1.1.

[smitpatel]

@xrkolovos - Another workaround would be to use non-shadow properties which does not encounter this issue.

[rowanmiller]

@smitpatel let us know if you don't think this is critical enough for 1.1.1

[nh43de]

Having the same problem, and have not been able to find a suitable workaround (that does not involve removing inheritance).

[smitpatel]

@nh43de - You can use non-shadow property which will not encounter this issue.

[smitpatel]

I have tackled this issue in various ways but yet to arrive at fully working fix. But based on all the attempts so far, the fix for this issue would involve breaking changes & may be risky for patch release.
Since this is data corruption issue, for patch we can just throw exception and block the scenario. Removing milestone for the discussion if we should split the issue in 2 parts.

[rowanmiller]

@smitpatel to code the fix and then we will make the decision

[smitpatel]

Justification: In the scenario where there are shadow properties in derived type, doing any query operation on the base type would give incorrect data for derived type. In certain cases where user model is more restricted like this it would throw exception.

Risk: It is low risk. The fix changes the pipeline in case of the particularly faulty scenario and doesn't affect the path for any other cases.

[Eilon]

This patch bug is approved. Please use the normal code review process w/ a PR and make sure the fix is in the correct branch, then close the bug and mark it as done.

[smitpatel]

Merged in 1.1.1 with 0485df0
in dev with d3fb6e9