Migration to .net5 issue

[taconaut]

Hi,

For testing purposes, I've tried to update our project to .net5. This only required to change the target framework to net5.0 and updating the referenced nuget packages. After this, the project could be built.

When running it, I get an exception when adding the DbContext and enabling temporal table queries:

Exception details:

System.TypeLoadException
  HResult=0x80131522
  Message=Method 'Create' in type 'EntityFrameworkCore.TemporalTables.Query.AsOfQueryableMethodTranslatingExpressionVisitorFactory' from assembly 'Dabble.EntityFrameworkCore.Temporal.Query, Version=1.0.3.0, Culture=neutral, PublicKeyToken=null' does not have an implementation.
  Source=Dabble.EntityFrameworkCore.Temporal.Query
  StackTrace:
   at Microsoft.EntityFrameworkCore.SqlServerAsOfEntityTypeBuilderExtensions.EnableTemporalTableQueries(DbContextOptionsBuilder optionsBuilder)
   at Refdata.SAI.Data.Extensions.ServiceCollectionExtensions.<>c__DisplayClass0_0.<AddEntityFrameworkRepositories>b__0(DbContextOptionsBuilder builder) in D:\Dev\Refdata.SAI\Source\Refdata.SAI.Data\Extensions\ServiceCollectionExtensions.cs:line 38
   at Microsoft.Extensions.DependencyInjection.EntityFrameworkServiceCollectionExtensions.<>c__DisplayClass1_0`2.<AddDbContext>b__0(IServiceProvider p, DbContextOptionsBuilder b)
   at Microsoft.Extensions.DependencyInjection.EntityFrameworkServiceCollectionExtensions.CreateDbContextOptions[TContext](IServiceProvider applicationServiceProvider, Action`2 optionsAction)
   at Microsoft.Extensions.DependencyInjection.EntityFrameworkServiceCollectionExtensions.<>c__DisplayClass17_0`1.<AddCoreServices>b__0(IServiceProvider p)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite factoryCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass1_0.<RealizeService>b__0(ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.EntityFrameworkServiceCollectionExtensions.<>c__17`1.<AddCoreServices>b__17_1(IServiceProvider p)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite factoryCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass1_0.<RealizeService>b__0(ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.GetService(IServiceProvider sp, Type type, Type requiredBy, Boolean isDefaultParameterRequired)
   at Microsoft.AspNetCore.Mvc.Controllers.ControllerActivatorProvider.<>c__DisplayClass4_0.<CreateActivator>b__0(ControllerContext controllerContext)
   at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass5_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()

Do you plan on supporting .net 5?

[Adam-Langley]

Hi @taconaut - are you running this from code? Have you tried running it from code, rather than the nuget package?
An error like this I would expect might only require recompilation.

I don't have any short-term plans per-se, other than to fix any issues that come up.

[stzoran1]

Great library. I will try making it compatible with EF Core 5.0 but can't promise I will succeed. There are some breaking changes that might be a deal breaker for making this library compatible without investing much time in it.

[taconaut]

@Adam-Langley I've tdone following:

• Checkout this project
• Remove nuget reference to this project in our project
• Add checked out project in our solution
• Reference the project (instead of the nuget package) in out project

-> Result is the same
System.TypeLoadException: 'Method 'Create' in type 'EntityFrameworkCore.TemporalTables.Query.AsOfQueryableMethodTranslatingExpressionVisitorFactory' from assembly 'Dabble.EntityFrameworkCore.Temporal.Query, Version=1.0.3.0, Culture=neutral, PublicKeyToken=null' does not have an implementation.'

@stzoran1 Hope you succeed :)

[taconaut]

Found the reason, the IQueryableMethodTranslatingExpressionVisitorFactory definition has changed.

Currently implemented:
public QueryableMethodTranslatingExpressionVisitor Create(IModel model)

Newly required:
public QueryableMethodTranslatingExpressionVisitor Create(QueryCompilationContext queryCompilationContext)

[stzoran1]

Yes, that is one of the reasons. Biggest issue is Print method which is now protected and can't be overridden.

[taconaut]

@stzoran1 Print can be overridden as protected. The issue I've mentioned in the PR worries me more. Non-public constructors and methods are being accessed by reflection and the signature of SqlExpressionFactory.AddConditions has changed. When removing the last null to respect the new method signature privateInitializer.Invoke(this, new object[] {select, entityType, null}); I'm getting below exception.

System.InvalidOperationException: 'Unhandled expression '[EntityFrameworkCore.TemporalTables.Query.AsOfTableExpression]' of type 'EntityFrameworkCore.TemporalTables.Query.AsOfTableExpression' encountered in 'SqlNullabilityProcessor'.'

@Adam-Langley Isn't there a way to access wanted functionality through public methods instead of using reflection? Or should a request be submitted that these constructor/method be made public?

[stzoran1]

@taconaut you are right. I overlooked that related to protected Print.

[stzoran1]

@taconaut I invested some time to resolve this but got stuck at very same point as you. Any progress on your side?

[taconaut]

@stzoran1 I haven't tried any further as we don't want to migrate our application just yet. I guess the next step would be to understand the changes in the signature of the internal method SqlExpressionFactory.AddConditions and possibly what is going on further down the chain.
I have never dug this deep into EF before and am not sure what the best/intended approach is, to do what we want. As mentioned before, accessing internal methods through reflection seems risky to me as it can break with every library update.

[stzoran1]

@taconaut you are happy you are waiting with upgrading. My project owner insisted to switch to 5.0 immediately after release. Now we are struggling with bugs. I fixed one library related to temporal tables and made contribution. But this will be more challenging. I plan to download EF Core repository and to use it inside my project instead of NUGET package in order to get better overview what is going on under the hood.

[Adam-Langley]

Hi guys,

Sorry to see you're having trouble with this. Please note that even the original code needed to use private APIs - MS hasn't yet made it possible to support temporal query code generation without doing so. It is on their road map - but until then, you're going to get these kinds of breakages when upgrading.
I also dread the task of having a flavour of this library for each major framework version.

[stzoran1]

I setup test project using Rider this morning in order to get better debugging options. This revealed what is causing issue:

So this library is creating AsOfTableExpression which is not supported in SqlNullabilityProcessor's Visit method.

I will try resolving this. If anyone have some ideas please share.

[stzoran1]

SqlNullabilityProcessor is a new class in EF Core 5.0.0. It was not used in 3.1.x

[stzoran1]

I failed to trace it. So many changes in EF Core 5 which are affecting this library.

[taconaut]

@stzoran1 I haven't used Rider, but in VS you can disable the debug option 'just my code' and enable braking on exception. When the exception is being thrown (and the debugger halts) I'd expect to figure something out from the call stack your in. Is this what you tried?

[stzoran1]

@taconaut yes, I know for this option, but on my side this is sometimes working and sometimes not. Never mind, this is not too important. Important thing is that EF Core team added new class to check Expression type and all unknown expression types are rejected. I am stuck there and have no idea at the moment how to resolve it.

[taconaut]

@stzoran1 I'd try checking the ef-core issues here on github to see if a relevant issue exists and create a new one if it doesn't. Sorry I can't be of more help but for these tasks you know when you start but not when you're going to get them fixed.

[stzoran1]

@taconaut you are right. Thank you for your help. Please let me know if you find something in ef-core issues.

[tonithenhausen]

I found a workaround to intercept the SqlNullabilityProcessor. Normally it would throw in case of unkown TableExpressions.

public class MyRelationalParameterBasedSqlProcessorFactory : IRelationalParameterBasedSqlProcessorFactory
    {
        private readonly RelationalParameterBasedSqlProcessorDependencies _dependencies;

        /// <summary>
        ///     This is an internal API that supports the Entity Framework Core infrastructure and not subject to
        ///     the same compatibility standards as public APIs. It may be changed or removed without notice in
        ///     any release. You should only use it directly in your code with extreme caution and knowing that
        ///     doing so can result in application failures when updating to a new Entity Framework Core release.
        /// </summary>
        public MyRelationalParameterBasedSqlProcessorFactory(RelationalParameterBasedSqlProcessorDependencies dependencies)
        {
            _dependencies = dependencies;
        }

        public RelationalParameterBasedSqlProcessor Create(bool useRelationalNulls)
        {
            return new MySqlServerParameterBasedSqlProcessor(_dependencies, useRelationalNulls);
        }
    }

    public class MySqlServerParameterBasedSqlProcessor : SqlServerParameterBasedSqlProcessor
    {
        public MySqlServerParameterBasedSqlProcessor(RelationalParameterBasedSqlProcessorDependencies dependencies, bool useRelationalNulls)
            : base(dependencies, useRelationalNulls)
        {
        }

        protected override SelectExpression ProcessSqlNullability(SelectExpression selectExpression, IReadOnlyDictionary<string, object> parametersValues, out bool canCache)
        {
            return new MySqlNullabilityProcessor(Dependencies, UseRelationalNulls).Process(selectExpression, parametersValues, out canCache);
        }
    }

    public class MySqlNullabilityProcessor : SqlNullabilityProcessor
    {
        public MySqlNullabilityProcessor(RelationalParameterBasedSqlProcessorDependencies dependencies, bool useRelationalNulls)
            : base(dependencies, useRelationalNulls)
        {
        }

        protected override TableExpressionBase Visit(TableExpressionBase tableExpressionBase)
        {
            if (tableExpressionBase is AsOfTableExpression)
                return tableExpressionBase;
            return base.Visit(tableExpressionBase);
        }
    }

You need to register it as follows:

services.AddSingleton<IRelationalParameterBasedSqlProcessorFactory, MyRelationalParameterBasedSqlProcessorFactory>();

[stzoran1]

@tonithenhausen it is looking as a good patch for SqlNullabilityProcessor issue. I will try it later this week. Thank you.

[tonithenhausen]

Also I found out that the AsOfQueryableMethodTranslatingExpressionVisitor.VisitConstant method that captures the asOfDateParameter from the query is not being called.
Instead I changed the VisitMethodCall method like so:

     protected override Expression VisitMethodCall(MethodCallExpression methodCallExpression)
        {
            if (methodCallExpression.Method.DeclaringType == typeof(SqlServerAsOfQueryableExtensions))
            {
                switch (methodCallExpression.Method.Name)
                {
                    case nameof(SqlServerAsOfQueryableExtensions.AsOf):
                        // capture the date parameter for use by all AsOfTableExpression instances
                        _asOfDateParameter = Visit(methodCallExpression.Arguments[1]) as ParameterExpression;
                        var visitMethodCall = Visit(methodCallExpression.Arguments[0]);
                        
                        if (null != _asOfDateParameter && visitMethodCall is ShapedQueryExpression shapedExpression)
                        {
                            // attempt to apply the captured date parameter to any select-from-table expressions
                            shapedExpression.TrySetDateParameter(_asOfDateParameter);
                        }

                        return visitMethodCall;
                }
            }

            return base.VisitMethodCall(methodCallExpression);
        }

Also in the AsOfQuerySqlGenerator.VisitAsOfTable I made the following change to support dbContext.User.Include(x=>x.Roles) EF Core queries. If an AsOf date is specified for the main query, it will be used for the .Include(...) part as well.

        public ParameterExpression FirstAsOfDate { get; set; }

protected virtual Expression VisitAsOfTable(AsOfTableExpression tableExpression)
        {
            // This method was modeled on "SqlServerQuerySqlGenerator.VisitTable".
            // Where we deviate, is after printing the table name, we check if temporal constraints
            // need to be applied.

            Sql.Append(_sqlGenerationHelper.DelimitIdentifier(tableExpression.Name, tableExpression.Schema));

            if (tableExpression.AsOfDate != null || FirstAsOfDate != null)
            {
                var asOffDate = tableExpression?.AsOfDate ?? FirstAsOfDate;
                
                var name = TEMPORAL_PARAMETER_PREFIX + asOffDate.Name;
                Sql.Append($" FOR SYSTEM_TIME AS OF @{name}"); //2020-02-28T11:00:00

                if (!_commandbuilder.Parameters.Any(x => x.InvariantName == asOffDate.Name))
                    _commandbuilder.AddParameter(asOffDate.Name, name);

                if (tableExpression.AsOfDate != null && FirstAsOfDate == null)
                    FirstAsOfDate = tableExpression.AsOfDate; }
            
            Sql
                .Append(AliasSeparator)
                .Append(_sqlGenerationHelper.DelimitIdentifier(tableExpression.Alias));

            return tableExpression;
        }

[stzoran1]

I found a workaround to intercept the SqlNullabilityProcessor. Normally it would throw in case of unkown TableExpressions.

public class MyRelationalParameterBasedSqlProcessorFactory : IRelationalParameterBasedSqlProcessorFactory
    {
        private readonly RelationalParameterBasedSqlProcessorDependencies _dependencies;

        /// <summary>
        ///     This is an internal API that supports the Entity Framework Core infrastructure and not subject to
        ///     the same compatibility standards as public APIs. It may be changed or removed without notice in
        ///     any release. You should only use it directly in your code with extreme caution and knowing that
        ///     doing so can result in application failures when updating to a new Entity Framework Core release.
        /// </summary>
        public MyRelationalParameterBasedSqlProcessorFactory(RelationalParameterBasedSqlProcessorDependencies dependencies)
        {
            _dependencies = dependencies;
        }

        public RelationalParameterBasedSqlProcessor Create(bool useRelationalNulls)
        {
            return new MySqlServerParameterBasedSqlProcessor(_dependencies, useRelationalNulls);
        }
    }

    public class MySqlServerParameterBasedSqlProcessor : SqlServerParameterBasedSqlProcessor
    {
        public MySqlServerParameterBasedSqlProcessor(RelationalParameterBasedSqlProcessorDependencies dependencies, bool useRelationalNulls)
            : base(dependencies, useRelationalNulls)
        {
        }

        protected override SelectExpression ProcessSqlNullability(SelectExpression selectExpression, IReadOnlyDictionary<string, object> parametersValues, out bool canCache)
        {
            return new MySqlNullabilityProcessor(Dependencies, UseRelationalNulls).Process(selectExpression, parametersValues, out canCache);
        }
    }

    public class MySqlNullabilityProcessor : SqlNullabilityProcessor
    {
        public MySqlNullabilityProcessor(RelationalParameterBasedSqlProcessorDependencies dependencies, bool useRelationalNulls)
            : base(dependencies, useRelationalNulls)
        {
        }

        protected override TableExpressionBase Visit(TableExpressionBase tableExpressionBase)
        {
            if (tableExpressionBase is AsOfTableExpression)
                return tableExpressionBase;
            return base.Visit(tableExpressionBase);
        }
    }

You need to register it as follows:

services.AddSingleton<IRelationalParameterBasedSqlProcessorFactory, MyRelationalParameterBasedSqlProcessorFactory>();

@tonithenhausen I tried approach you proposed but with no success. May you please specify where to register singleton? Is it important to register it before or after some other service? Do I need to make other changes in library to implement your solution?

[tonithenhausen]

@stzoran1 I am using efcore-temporal-query with https://github.com/findulov/EntityFrameworkCore.TemporalTables.
For the EntityFrameworkCore.TemporalTables I needed to replace the EFCore internal service provider. Maybe this is a requirement to allow to register custom EFCore services.

My code is as follows:

      services.AddDbContext<EssDbContext>((provider, options) =>
            {
                options.UseSqlServer(connectionString));
                options.UseInternalServiceProvider(provider);
            });
            services.AddEntityFrameworkSqlServer();
            services.AddSingleton<IRelationalParameterBasedSqlProcessorFactory, MyRelationalParameterBasedSqlProcessorFactory>();

[tonithenhausen]

As I have modified the source code of your package and the https://github.com/findulov/EntityFrameworkCore.TemporalTables as well, I could upload my modified source code to Github if you like.

[stzoran1]

As I have modified the source code of your package and the https://github.com/findulov/EntityFrameworkCore.TemporalTables as well, I could upload my modified source code to Github if you like.

@tonithenhausen thank you for quick reply. I am using the same. I managed to fix Findulov's library but still struggling with this one. It would be great if you can upload the code or even make fork with your version. Thank you in advance.

[tonithenhausen]

@stzoran1 I have uploaded the code to: https://github.com/tonithenhausen/EntityFrameworkCore.TemporalTables

[stzoran1]

@tonithenhausen Thank you, I will check it on Monday and get back to you.

[rdeprins]

@tonithenhausen Thanks for sharing your findings. It's not clear to me from the comments if your fixes restore 100% of the functionality. Is there more work to be done or could this be upstreamed soon?

[tonithenhausen]

@rdeprins I am currently using the code from https://github.com/tonithenhausen/EntityFrameworkCore.TemporalTables in a production EF Core 5 application. However, I made modifications to apply As OF filtering to .Include(...) queries. Not sure if this was working before with EF Core 3.