JIT Compiler encountered an internal limitation. EF Core Migrations.

[mdmoura]

On an ASP.NET Core with EntityFramework Core I am creating an EF Migration:

 dotnet ef migrations add InitialCommit

The migration includes seeding 280 objects in one of the entities using HasData.

I am able to seed 1 and 20 objects but I get an exception when seeding 280 objects.

Exception

Exception message: JIT Compiler encountered an internal limitation.
Stack trace:
System.InvalidProgramException: JIT Compiler encountered an internal limitation.
   at Proj.Configurations.CompositionConfiguration.Configure(EntityTypeBuilder`1 builder)
   at Microsoft.EntityFrameworkCore.ModelBuilder.ApplyConfiguration[TEntity](IEntityTypeConfiguration`1 configuration)
   at Proj.Context.<>c.<OnModelCreating>b__209_0(ModelBuilder x) in Proj/Context.cs:line 80
   at Proj.Context.OnModelCreating(ModelBuilder builder) in Proj/Context.cs:line 72
   at Microsoft.EntityFrameworkCore.Infrastructure.ModelCustomizer.Customize(ModelBuilder modelBuilder, DbContext context)
   at Microsoft.EntityFrameworkCore.Infrastructure.RelationalModelCustomizer.Customize(ModelBuilder modelBuilder, DbContext context)
   at Microsoft.EntityFrameworkCore.Infrastructure.ModelSource.CreateModel(DbContext context, IConventionSetBuilder conventionSetBuilder, IModelValidator validator)
   at Microsoft.EntityFrameworkCore.Infrastructure.ModelSource.<>c__DisplayClass5_0.<GetModel>b__1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at Microsoft.EntityFrameworkCore.Infrastructure.ModelSource.GetModel(DbContext context, IConventionSetBuilder conventionSetBuilder, IModelValidator validator)
   at Microsoft.EntityFrameworkCore.Internal.DbContextServices.CreateModel()
   at Microsoft.EntityFrameworkCore.Internal.DbContextServices.get_Model()
   at Microsoft.EntityFrameworkCore.Infrastructure.EntityFrameworkServicesBuilder.<>c.<TryAddCoreServices>b__7_1(IServiceProvider p)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite factoryCallSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass1_0.<RealizeService>b__0(ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.EntityFrameworkCore.DbContext.get_DbContextDependencies()
   at Microsoft.EntityFrameworkCore.DbContext.get_InternalServiceProvider()
   at Microsoft.EntityFrameworkCore.DbContext.Microsoft.EntityFrameworkCore.Infrastructure.IInfrastructure<System.IServiceProvider>.get_Instance()
   at Microsoft.EntityFrameworkCore.Internal.InternalAccessorExtensions.GetService[TService](IInfrastructure`1 accessor)
   at Microsoft.EntityFrameworkCore.Infrastructure.AccessorExtensions.GetService[TService](IInfrastructure`1 accessor)
   at Microsoft.EntityFrameworkCore.Design.Internal.DbContextOperations.CreateContext(Func`1 factory)
   at Microsoft.EntityFrameworkCore.Design.Internal.DbContextOperations.CreateContext(String contextType)
   at Microsoft.EntityFrameworkCore.Design.Internal.MigrationsOperations.AddMigration(String name, String outputDir, String contextType)
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.AddMigrationImpl(String name, String outputDir, String contextType)
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.AddMigration.<>c__DisplayClass0_1.<.ctor>b__0()
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OperationBase.<>c__DisplayClass3_0`1.<Execute>b__0()
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OperationBase.Execute(Action action)
JIT Compiler encountered an internal limitation.

Steps to reproduce

The entity Composition is as follows:

  public class Composition {

    public Int32 ProfuctId { get; set; }
    public Decimal? P01 { get; set; }
    public Decimal P02 { get; set; }
    public Decimal? P03 { get; set; }
    public Decimal P04 { get; set; } 
    public Decimal? P05 { get; set; }    
    public Decimal? P06 { get; set; }
    public Decimal? P07 { get; set; }
    public Decimal? P08 { get; set; }
    public Decimal? P09 { get; set; }
    public Decimal? P10 { get; set; }
    public Decimal? P11 { get; set; }
    public Decimal? P12 { get; set; }
    public Decimal? P13 { get; set; }
    public Decimal? P14 { get; set; }
    public Decimal P15 { get; set; }
    public Decimal? P16 { get; set; }
    public Decimal? P17 { get; set; }
    public Decimal? P18 { get; set; }
    public Decimal? P19 { get; set; }
    public Decimal? P20 { get; set; }
    public Decimal P21 { get; set; }
    public Decimal? P22 { get; set; }
    public Decimal? P23 { get; set; }    
    public Decimal? P24 { get; set; }    
    public Decimal? P25 { get; set; }      
    public Decimal? P26 { get; set; }    
    public Decimal? P27 { get; set; }
    public Decimal? P28 { get; set; }
    public Decimal? P29 { get; set; }
    public Decimal? P30 { get; set; }
    public Decimal? P31 { get; set; }
    public Decimal? P32 { get; set; }    

    public virtual Product { get; set; }

  }

And this is the Composition configuration:

    builder.ToTable("Compositions");

    builder.HasKey(y => y.ProductId);

    builder.Property(x => x.P01).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P02).IsRequired(true).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P03).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P04).IsRequired(true).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P05).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));        
    builder.Property(x => x.P06).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P07).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P08).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P09).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P10).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P11).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P12).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P13).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P14).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P15).IsRequired(true).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P16).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P17).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P18).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P19).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P20).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P21).IsRequired(true).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P22).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P23).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P24).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));      
    builder.Property(x => x.P25).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P26).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P27).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P28).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P29).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P30).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P31).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));
    builder.Property(x => x.P32).IsRequired(false).HasColumnType(Sql.Decimal(8, 2));        	      

    builder.HasOne(y => y.Product).WithOne(y => y.Composition).HasForeignKey<Composition>(y => y.ProductId).IsRequired(true).OnDelete(DeleteBehavior.Cascade);

    builder.HasData(
      new { 
        ProductId = 1,
        P01 = (Decimal?) 110.0, 
        P02 = (Decimal) 60030.0, 
        P03 = (Decimal?) 0.0, 
        P04 = (Decimal) 336.0, 
        P05 = (Decimal?) 10600.0, 
        P06 = (Decimal?) 0.0, 
        P07 = (Decimal?) 0.633, 
        P08 = (Decimal?) 8.27, 
        P09 = (Decimal?) 184.0, 
        P10 = (Decimal?) 106.0, 
        P11 = (Decimal?) 2.075, 
        P12 = (Decimal?) 424.0, 
        P13 = (Decimal?) 542.0, 
        P14 = (Decimal?) 1112.0, 
        P15 = (Decimal) 23520.0, 
        P16 = (Decimal?) 0.226, 
        P17 = (Decimal?) 331.0, 
        P18 = (Decimal?) 16.0, 
        P19 = (Decimal?) 6900.0, 
        P20 = (Decimal?) 0.853, 
        P21 = (Decimal) 1260.0, 
        P22 = (Decimal?) 0.633, 
        P23 = (Decimal?) 0.0, 
        P24 = (Decimal?) 0.003, 
        P25 = (Decimal?) 0.0, 
        P26 = (Decimal?) 0.357, 
        P27 = (Decimal?) 1.5, 
        P28 = (Decimal?) 0.0, 
        P29 = (Decimal?) 0.39, 
        P30 = (Decimal?) 0.005, 
        P31 = (Decimal?) 11950.0, 
        P32 = (Decimal?) 3.37 
    });

In the code example I just posted HasData is seeding only one Composition ...
In reality I am seeding 280 compositions and I also tested seeding only 20 ...

Further technical details

EF Core version: 2.1.3
Database Provider: Microsoft.EntityFrameworkCore.SqlServer
Operating system: macOS Mojave
IDE: Visual Studio Code 1.29.1
NET Core SDK: 2.1.403

[mikedn]

Any chance of a self contained repro? I've been trying to put together something using the bits and pieces of code you provided but so far I haven't been able to reproduce the error.

In reality I am seeding 280 compositions and I also tested seeding only 20 …

Does that mean that you have a HasData call with 280 parameters? Could you try building the params array manually (allocate an array, initialize its elements one by one, pass it to HasData) or is this some kind of autogenerated code?

[mdmoura]

Does that mean that you have a HasData call with 280 parameters?

No, HasData has 280 instances each one with 32 parameters ...

@mikedn I just created a repository with it: https://github.com/mdmoura/JITCompilerEFCore

You just need to run the command:

dotnet ef migrations add InitialCommit

With that command, and with the code as it is, you will get the error:

JIT Compiler encountered an internal limitation.

Note: I am inserting 262 records to get the error ... But you can get the error with much less.

If you comment all lines, in HasData, leaving only one then the command dotnet ef migrations add InitialCommit will create the migration without any error ...

If you then want to create the database for that migration just run:

 dotnet ef database update

The SQL Database configuration is in ContextFactory. I am on a Mac so using a Docker container.

But the error happens when adding the migration. Any idea what might be wrong?

[mikedn]

Thank you for the sample!

I can reproduce the error using netcoreapp2.1 but I can't seem to reproduce it when using netcoreapp3.0.

@AndyAyersMS Was there any improvement done for cases where a large number of temporary variables are created due to newobj? AFAIR there was an issue like this and you made some improvements in the importer? Method Context.OnModelCreating generates tens of thousands of lclvars...

[mikedn]

Hmm, this might have been fixed by @briansull in dotnet/coreclr#17793

[mdmoura]

@mikedn Is the fix included in Net Core 2.2 or only in Net Core 3.0?

[mikedn]

@mdmoura As far as I can tell no, the fix is not in 2.2.

[briansull]

I also checked and dotnet/coreclr#17793 is NOT in .Core 2.2 sources

[mdmoura]

@briansull Is there any way to solve this problem in the meanwhile?

[mikedn]

Do you really need to use nullable decimal in the initialization data (apart from those couple of values that are really null)? I would guess that even if the column is nullable EF should be able to convert from decimal to nullable decimal when inserting the data. Or perhaps use double instead of decimal, assuming that you don't end up with precision issues and that EF can convert from double to decimal.

Other options:

• split the data into multiple methods (each returns an array containing a part of the data, concatenate all arrays and pass the result to HasData)
• deserialize the array from some kind of file (JSON, XML, CSV etc.) instead of hardcoding it

I'd do the later, it seems preferable to keep this kind of data in a CSV file rather than in code. Even VS seems to having some problems dealing with this amount of code, it's rather sluggish when I open the file. And I wouldn't be surprised to discover that it is faster to read the file rather than to JIT-compile such code. The JIT has to generate around one megabyte of code!

To try to explain a bit the tehnical issue: this code, while perhaps it doesn't look like much - a bunch of numbers, generates a very large number of local variables in the JIT. Both decimal and Nullable<T> are structs and they need memory locations, unlike primitive types that can be hold in registers. So the JIT keeps creating variables - one for a decimal, another one for a Nullable<decimal>, another one for the next decimal and so on, until it hits an internal limit.

[mdmoura]

Do you really need to use nullable decimal in the initialization data (apart from those couple of values that are really null)? I would guess that even if the column is nullable EF should be able to convert from decimal to nullable decimal when inserting the data.

I am casting to Decimal? because I got an error when not using it ...

Deserialize the array from some kind of file (JSON, XML, CSV etc.) instead of hardcoding it

Yes, I am going for this one ...

I was hardcoding because that would avoid to have files embedded on the class library.

[mikedn]

Yes, I am going for this one …

I wonder if that actually helps. I'm not exactly familiar with EF but I was looking around and I see that the migrations command generated some .cs files in a directory. And those .cs files also contain similar code. I wonder if it won't do the same thing even if the data is loaded from a file.

[AndyAyersMS]

If this is an ingrained pattern produced by EF tooling, we can certainly argue that Brian's fix should be ported to 2.2. So please keep us updated.

I have some prototype changes lying around where the jit will reuse temps in cases like this and so avoid hitting these limitations and also emit much cleaner and smaller code, but don't recall offhand how close they were to being usable.

[mikedn]

the jit will reuse temps

Now I'm curious, how do you know when it's safe to reuse such a temp?

[briansull]

However this is Fixed in .Net 4.8
VSO 604943

[AndyAyersMS]

Idea was to do something similar to what we do for the box temp: https://github.com/AndyAyersMS/coreclr/tree/OpportunisticReuseNewObjTemp

[mikedn]

https://github.com/AndyAyersMS/coreclr/tree/OpportunisticReuseNewObjTemp

Hmm, I don't think that handles the (hopefully very rare) corner case of a struct constructor that escapes this somehow, does it? Come to think of it, that would be extremely bizarre, especially in the context of newobj.

[ajcvickers]

@AndyAyersMS @briansull This is a pretty common pattern in EF. Can we bring this fix to shiproom for a 2.2.x patch?
/cc @divega @Eilon @DamianEdwards

[AndyAyersMS]

Seems reasonable -- @briansull do you want to set this up, it since it's your change?

cc @RussKeldorph

[mdmoura]

It would be great to have a patch, not in a far future, to solve this problem.

[briansull]

Fix is in PR dotnet/coreclr#17793 and PR dotnet/coreclr#19065

[RussKeldorph]

Should have been closed by dotnet/coreclr#21493?

[mdmoura]

Any ETA for this fix? It seems it wasn't released with Asp.Net Core 2.2.1 patch.

[briansull]

It is only shipping in 2.2.2

@vivmishra any ETA for 2.2.2 ?

[Eilon]

@briansull - should be in early-to-mid Feb, so very soon.