Skip to content
This repository has been archived by the owner on Nov 1, 2020. It is now read-only.

MissingMetadataException when using EntityFrameworkCore with SQLite #6477

Open
rubin55 opened this issue Oct 22, 2018 · 20 comments
Open

MissingMetadataException when using EntityFrameworkCore with SQLite #6477

rubin55 opened this issue Oct 22, 2018 · 20 comments

Comments

@rubin55
Copy link

rubin55 commented Oct 22, 2018

I've got a dotnet core 2.1 project where I'm experimenting with creating native static binaries. I'm getting a MissingMetadataException after compilation (using dotnet publish -r osx-x64 -c debug):

[rubin@KURO dot-hello (master)]$ bin/Debug/netcoreapp2.1/osx-x64/publish/hello
Let's do some SQlite!
Unhandled Exception: System.Reflection.MissingMetadataException: 'Microsoft.EntityFrameworkCore.Internal.DiagnosticsLogger<Microsoft.EntityFrameworkCore.DbLoggerCategory.Infrastructure>' is missing metadata. For more information, please visit http://go.microsoft.com/fwlink/?LinkID=392859
   at hello!<BaseAddress>+0x56da94
   at hello!<BaseAddress>+0xa2f123
   at hello!<BaseAddress>+0x33477c
   at hello!<BaseAddress>+0x333f60
   at hello!<BaseAddress>+0x4c5e68
   at hello!<BaseAddress>+0x2d1bdf
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.TryCreateOpenGeneric(ServiceDescriptor, Type, CallSiteChain) + 0x100
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.TryCreateOpenGeneric(Type, CallSiteChain) + 0xc6
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.CreateCallSite(Type, CallSiteChain) + 0xbf
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.CreateServiceAccessor(Type) + 0x5b
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey, Func`2) + 0xff
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type, ServiceProviderEngineScope) + 0x5f
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type) + 0x54
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type) + 0x32
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider, Type) + 0xdb
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider) + 0x86
   at Microsoft.EntityFrameworkCore.Internal.ServiceProviderCache.<>c__DisplayClass4_0.<GetOrAdd>b__2(Int64) + 0x330
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey, Func`2) + 0xfc
   at Microsoft.EntityFrameworkCore.Internal.ServiceProviderCache.GetOrAdd(IDbContextOptions, Boolean) + 0x43b
   at Microsoft.EntityFrameworkCore.DbContext..ctor(DbContextOptions) + 0x179
   at Microsoft.EntityFrameworkCore.DbContext..ctor() + 0x32
   at hello.BloggingContext..ctor() + 0x16
   at hello.Program.Main(String[]) + 0x4b
   at hello!<BaseAddress>+0xcb2a76
   at hello!<BaseAddress>+0xcb2b03

I've read the information on the page referenced in the backtrace, but I can't figure out what to add to my rd.xml. My project is here: https://github.com/rubin55/dot-hello . To reproduce (on macos, I suspect it is the same on other platforms):

  1. git clone git@github.com:rubin55/dot-hello.git
  2. cd dot-hello
  3. dotnet ef database update
  4. dotnet publish -r osx-x64 -c debug
  5. bin/Debug/netcoreapp2.1/osx-x64/publish/hello

I'm wondering if the fact that EntityFramework in this case, having an external dependency on the SQLite .so/.dll/.dylib is the root cause of this, which raises the question: how does one deal with external dependencies like that when using CoreRT?

@MichalStrehovsky
Copy link
Member

There is a piece of RD.XML that will likely fix the issue you're seeing over in issue #6459 (the line with DiagnosticsLogger).

See the WebApi sample CSPROJ file for how to reference the RD.XML from your project: https://github.com/dotnet/corert/tree/master/samples/WebApi

@rubin55
Copy link
Author

rubin55 commented Oct 22, 2018

Alright, I've managed to iteratively add various missing metadata things. My rd.xml now looks like:

<Directives>
    <Application>
        <Assembly Name="hello" Dynamic="Required All" />
        <!-- 'Microsoft.EntityFrameworkCore.Internal.DiagnosticsLogger<Microsoft.EntityFrameworkCore.DbLoggerCategory.Infrastructure>' -->
        <!-- 'Microsoft.EntityFrameworkCore.Internal.DbSetInitializer' -->
        <!-- 'Microsoft.EntityFrameworkCore.DbSet<hello.Blog>'-->
        <Assembly Name="Microsoft.EntityFrameworkCore">
            <Type Name="Microsoft.EntityFrameworkCore.Internal.DiagnosticsLogger`1[[Microsoft.EntityFrameworkCore.DbLoggerCategory+Infrastructure,Microsoft.EntityFrameworkCore]]" Dynamic="Required All" />
            <Type Name="Microsoft.EntityFrameworkCore.Internal.DbSetInitializer" Dynamic="Required All" />
            <Type Name="Microsoft.EntityFrameworkCore.DbSet`1[[hello.Blog,hello]]" Dynamic="Required All" />
        </Assembly>
        <!-- 'Microsoft.Extensions.Options.OptionsMonitor<Microsoft.Extensions.Logging.LoggerFilterOptions>' -->
        <!-- 'Microsoft.Extensions.Options.OptionsFactory<Microsoft.Extensions.Logging.LoggerFilterOptions>' -->
        <!-- 'Microsoft.Extensions.Options.OptionsCache<Microsoft.Extensions.Logging.LoggerFilterOptions>' -->
        <Assembly Name="Microsoft.Extensions.Options">
            <Type Name="Microsoft.Extensions.Options.OptionsMonitor`1[[Microsoft.Extensions.Logging.LoggerFilterOptions,Microsoft.Extensions.Logging]]" Dynamic="Required All" />
            <Type Name="Microsoft.Extensions.Options.OptionsFactory`1[[Microsoft.Extensions.Logging.LoggerFilterOptions,Microsoft.Extensions.Logging]]" Dynamic="Required All" />
            <Type Name="Microsoft.Extensions.Options.OptionsCache`1[[Microsoft.Extensions.Logging.LoggerFilterOptions,Microsoft.Extensions.Logging]]" Dynamic="Required All" />
        </Assembly>

        <Assembly Name="Microsoft.Extensions.Logging">
            <Type Name="Microsoft.Extensions.Logging.LoggerFilterOptions" Dynamic="Required All" />
        </Assembly>
    </Application>
</Directives>

However, I'm now getting a TypeInitializationException from Linq:

[rubin@KURO dot-hello (master)]$ /Users/rubin/Syncthing/Source/Rubin/dot-hello/bin/debug/netcoreapp2.1/osx-x64/publish/hello 
Let's do some SQlite!
Unhandled Exception: System.TypeInitializationException: A type initializer threw an exception. To determine which type, inspect the InnerException's StackTrace property. ---> System.InvalidOperationException: Sequence contains no elements
   at System.Linq.Enumerable.Single[TSource](IEnumerable`1) + 0x18a
   at hello!<BaseAddress>+0x5f25af
   at hello!<BaseAddress>+0x4c9f8e
   at hello!<BaseAddress>+0x4c9ad2

   --- End of inner exception stack trace ---
   at hello!<BaseAddress>+0x4c9bbe
   at hello!<BaseAddress>+0x4c98ea
   at Microsoft.EntityFrameworkCore.Metadata.Internal.ClrAccessorFactory`1.Create(PropertyInfo, IPropertyBase) + 0x4a
   at Microsoft.EntityFrameworkCore.Metadata.Internal.ClrAccessorFactory`1.Create(PropertyInfo) + 0x26
   at Microsoft.EntityFrameworkCore.Internal.DbSetFinder.<>c__DisplayClass2_0.<FindSets>b__2(PropertyInfo) + 0x110
   at System.Linq.Enumerable.SelectIPartitionIterator`2.LazyToArray() + 0xdd
   at System.Linq.Enumerable.SelectIPartitionIterator`2.ToArray() + 0x74
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1) + 0x8a
   at Microsoft.EntityFrameworkCore.Internal.DbSetFinder.FindSets(Type) + 0x1c2
   at hello!<BaseAddress>+0x5a6c18
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey, Func`2) + 0xff
   at Microsoft.EntityFrameworkCore.Internal.DbSetFinder.FindSets(DbContext) + 0x68
   at Microsoft.EntityFrameworkCore.Internal.DbSetInitializer.InitializeSets(DbContext) + 0x4a
   at Microsoft.EntityFrameworkCore.DbContext..ctor(DbContextOptions) + 0x1a8
   at Microsoft.EntityFrameworkCore.DbContext..ctor() + 0x32
   at hello.BloggingContext..ctor() + 0x16
   at hello.Program.Main(String[]) + 0x4b
   at hello!<BaseAddress>+0x691a1a
   at hello!<BaseAddress>+0x691aa7

How would one normally proceed from the backtrace above? I also note that creating the rd.xml was rather a painful process; will this always be required for native compilations?

@MichalStrehovsky
Copy link
Member

Add this to your project file:

  <ItemGroup>
    <!-- Do not generate partial metadata for types - always include all members even if they're unused -->
    <IlcArg Include="--completetypemetadata" />

    <!-- Generate extra data to make stack traces nicer -->
    <IlcArg Include="--stacktracedata" />
  </ItemGroup>

(The completetypemetadata option might make the exception go away and get you to the next one.)

I also note that creating the rd.xml was rather a painful process; will this always be required for native compilations?

You can always do <Assembly Name="foo" Dynamic="Required All" /> for all assemblies that your app has - that should cover all the cases of non-generic things. For generic things, there's a feature that the .NET Native for UWP apps compiler has ("universal shared code" or "universal shared generics") that the open source compiler doesn't have yet, so there's known pain in that. Generating code for everything comes with certain costs though (compilation speed and the size of the resulting executable).

Entity framework is pretty much the reason why we added the universal shared code feature to the ".NET Native for UWP apps" compiler. EF does a lot of generic reflection.

@rubin55
Copy link
Author

rubin55 commented Oct 22, 2018

Alright, thanks for the pointers + options for completetypemetadata and stacktracedata. I've added them to my project and get a more informational backtrace:

[rubin@KURO dot-hello (master)]$ /Users/rubin/Syncthing/Source/Rubin/dot-hello/bin/debug/netcoreapp2.1/osx-x64/publish/hello 
Let's do some SQlite!
Unhandled Exception: EETypeRva:0x0122CF10(System.Reflection.MissingRuntimeArtifactException): MakeGenericMethod() cannot create this generic method instantiation because the instantiation was not metadata-enabled: 'Microsoft.EntityFrameworkCore.Metadata.Internal.ClrAccessorFactory<Microsoft.EntityFrameworkCore.Metadata.Internal.IClrPropertySetter>.CreateGeneric<hello.BloggingContext,Microsoft.EntityFrameworkCore.DbSet<hello.Blog>,Microsoft.EntityFrameworkCore.DbSet<hello.Blog>>(System.Reflection.PropertyInfo,Microsoft.EntityFrameworkCore.Metadata.IPropertyBase)' For more information, please visit http://go.microsoft.com/fwlink/?LinkID=616868
   at Internal.Reflection.Core.Execution.ExecutionEnvironment.GetMethodInvoker(RuntimeTypeInfo, QMethodDefinition, RuntimeTypeInfo[], MemberInfo) + 0x26a
   at System.Reflection.Runtime.MethodInfos.NativeFormat.NativeFormatMethodCommon.GetUncachedMethodInvoker(RuntimeTypeInfo[], MemberInfo) + 0xa4
   at System.Reflection.Runtime.MethodInfos.RuntimeNamedMethodInfo`1.GetUncachedMethodInvoker(RuntimeTypeInfo[], MemberInfo) + 0x36
   at System.Reflection.Runtime.MethodInfos.RuntimeConstructedGenericMethodInfo.get_UncachedMethodInvoker() + 0x3a
   at System.Reflection.Runtime.MethodInfos.RuntimeMethodInfo.get_MethodInvoker() + 0x197
   at System.Reflection.Runtime.MethodInfos.RuntimeNamedMethodInfo`1.MakeGenericMethod(Type[]) + 0x3b5
   at Microsoft.EntityFrameworkCore.Metadata.Internal.ClrAccessorFactory`1.Create(PropertyInfo, IPropertyBase) + 0x173
   at Microsoft.EntityFrameworkCore.Metadata.Internal.ClrAccessorFactory`1.Create(PropertyInfo) + 0x26
   at Microsoft.EntityFrameworkCore.Internal.DbSetFinder.<>c__DisplayClass2_0.<FindSets>b__2(PropertyInfo) + 0x110
   at System.Linq.Enumerable.SelectIPartitionIterator`2.LazyToArray() + 0xdd
   at System.Linq.Enumerable.SelectIPartitionIterator`2.ToArray() + 0x74
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1) + 0x8a
   at Microsoft.EntityFrameworkCore.Internal.DbSetFinder.FindSets(Type) + 0x1c2
   at hello!<BaseAddress>+0x609980
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey, Func`2) + 0xff
   at Microsoft.EntityFrameworkCore.Internal.DbSetFinder.FindSets(DbContext) + 0x68
   at Microsoft.EntityFrameworkCore.Internal.DbSetInitializer.InitializeSets(DbContext) + 0x4a
   at Microsoft.EntityFrameworkCore.DbContext..ctor(DbContextOptions) + 0x1a8
   at Microsoft.EntityFrameworkCore.DbContext..ctor() + 0x32
   at hello.BloggingContext..ctor() + 0x16
   at hello.Program.Main(String[]) + 0x4b
   at hello!<BaseAddress>+0x70379a
   at hello!<BaseAddress>+0x703827

Thanks for the help + pointers btw, super insightful stuff too!

@MichalStrehovsky
Copy link
Member

So now it's a generic method - the RD.XML syntax for that is here: #4775.

@rubin55
Copy link
Author

rubin55 commented Oct 22, 2018

Alright, I've added a generic method element to my rd.xml which looks like the following:

<Type Name="Microsoft.EntityFrameworkCore.Metadata.Internal.ClrAccessorFactory`1[[Microsoft.EntityFrameworkCore.Metadata.Internal.IClrPropertySetter,Microsoft.EntityFrameworkCore]]">
    <Method Name="CreateGeneric" Dynamic="Required">
        <GenericArgument Name="hello.BloggingContext,hello" />
        <GenericArgument Name="Microsoft.EntityFrameworkCore.DbSet`1[[hello.Blog,hello]], Microsoft.EntityFrameworkCore" />
        <GenericArgument Name="Microsoft.EntityFrameworkCore.DbSet`1[[hello.Blog,hello]], Microsoft.EntityFrameworkCore" />
    </Method>
</Type>

After compilation, I now get:

[rubin@KURO dot-hello (master)]$ bin/debug/netcoreapp2.1/osx-x64/publish/hello 
Let's do some SQlite!
GVM method pointer lookup failure

@MichalStrehovsky
Copy link
Member

GVM method pointer lookup failure

Can you try setting the version of the ILCompiler package you reference to 1.0.0-alpha-27012-01 (you probably have 1.0.0-alpha-* which defaults to latest)? I have a suspicion there might have been a recent regression.

@rubin55
Copy link
Author

rubin55 commented Oct 22, 2018

git diff hello.csproj:

...
-    <PackageReference Include="Microsoft.DotNet.ILCompiler" Version="1.0.0-alpha-*" />
+    <PackageReference Include="Microsoft.DotNet.ILCompiler" Version="1.0.0-alpha-27012-01" />

rm -rf obj/ bin/ && dotnet publish -r osx-x64 -c debug && bin/debug/netcoreapp2.1/osx-x64/publish/hello:

Let's do some SQlite!
GVM method pointer lookup failure

@MichalStrehovsky
Copy link
Member

Oh, okay. Kind of glad it's not a regression. In any case, that's a CoreRT bug. I've opened #6478 to track.

@MichalStrehovsky
Copy link
Member

@rubin55 @frankzye The fix for the GVM method pointer lookup failure has been merged and a new ILCompiler package with the fix is now out.

@rubin55
Copy link
Author

rubin55 commented Oct 24, 2018

I retried a build and I can pass the GVM lookup failure now. After another few missed types and generic methods, I end up with something new (for me, at least):

Unhandled Exception: EETypeRva:0x01358468(System.Reflection.MissingRuntimeArtifactException): This object cannot be invoked because it was metadata-enabled for browsing only: 'Microsoft.EntityFrameworkCore.Storage.DatabaseProvider<Microsoft.EntityFrameworkCore.Sqlite.Infrastructure.Internal.SqliteOptionsExtension>..ctor(Microsoft.EntityFrameworkCore.Storage.DatabaseProviderDependencies)' For more information, please visit  http://go.microsoft.com/fwlink/?LinkID=616867
   at Internal.Reflection.Core.Execution.ExecutionEnvironment.GetMethodInvoker(RuntimeTypeInfo, QMethodDefinition, RuntimeTypeInfo[], MemberInfo) + 0x26a
   at System.Reflection.Runtime.MethodInfos.NativeFormat.NativeFormatMethodCommon.GetUncachedMethodInvoker(RuntimeTypeInfo[], MemberInfo) + 0xa4
   at System.Reflection.Runtime.MethodInfos.RuntimePlainConstructorInfo`1.get_UncachedMethodInvoker() + 0x167
   at System.Reflection.Runtime.MethodInfos.RuntimeConstructorInfo.get_MethodInvoker() + 0x41
   at System.Reflection.Runtime.MethodInfos.RuntimePlainConstructorInfo`1.Invoke(BindingFlags, Binder, Object[], CultureInfo) + 0xd6
   at System.Reflection.ConstructorInfo.Invoke(Object[]) + 0x39
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, ServiceProviderEngineScope) + 0x110
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite, TArgument) + 0x152
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite, ServiceProviderEngineScope) + 0xd4
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitSingleton(SingletonCallSite, ServiceProviderEngineScope) + 0x77
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite, TArgument) + 0x1ea
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitIEnumerable(IEnumerableCallSite, ServiceProviderEngineScope) + 0xbb
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite, TArgument) + 0x100
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(IServiceCallSite, ServiceProviderEngineScope) + 0x3a
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass1_0.<RealizeService>b__0(ServiceProviderEngineScope) + 0xd1
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type, ServiceProviderEngineScope) + 0xce
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type) + 0x4c
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider) + 0x90
   at Microsoft.EntityFrameworkCore.Internal.DbContextServices.Initialize(IServiceProvider, IDbContextOptions, DbContext) + 0x9c
   at Microsoft.EntityFrameworkCore.DbContext.get_InternalServiceProvider() + 0x2a1
   at Microsoft.EntityFrameworkCore.DbContext.get_DbContextDependencies() + 0x57
   at Microsoft.EntityFrameworkCore.DbContext.EntryWithoutDetectChanges[TEntity](TEntity) + 0x33
   at Microsoft.EntityFrameworkCore.DbContext.SetEntityState[TEntity](TEntity, EntityState) + 0x53
   at Microsoft.EntityFrameworkCore.DbContext.Add[TEntity](TEntity) + 0x83
   at Microsoft.EntityFrameworkCore.Internal.InternalDbSet`1.Add(TEntity) + 0x85
   at hello.Program.Main(String[]) + 0xa3
   at hello!<BaseAddress>+0x7684a6
   at hello!<BaseAddress>+0x768533

What does the runtime mean by "This object cannot be invoked because it was metadata-enabled for browsing only"?

@MichalStrehovsky
Copy link
Member

What does the runtime mean by "This object cannot be invoked because it was metadata-enabled for browsing only"?

This means that we didn't compile the code for the non-generic method on the generic type.

Adding a Type directive for the instantiation with `Dynamic="Required All" should fix this. Similar to e.g. this line:

<Type Name="System.Linq.Expressions.ExpressionCreator`1[[System.Func`2[[System.Object,System.Private.CoreLib],[System.Object,System.Private.CoreLib]],System.Private.CoreLib]]" Dynamic="Required All" />

@rubin55
Copy link
Author

rubin55 commented Oct 24, 2018

I've stubbornly soldiered on to add whichever type the runtime comes up with (see latest commit here: https://github.com/rubin55/dot-hello)`. I get a by-now-familiar backtrace:

[rubin@KURO dot-hello (master)]$ bin/debug/netcoreapp2.1/osx-x64/publish/hello 
Let's do some SQlite!
Unhandled Exception: EETypeRva:0x0136C738(System.Reflection.MissingRuntimeArtifactException): This object cannot be invoked because it was metadata-enabled for browsing only: 'Microsoft.EntityFrameworkCore.Infrastructure.RelationalModelCustomizer..ctor(Microsoft.EntityFrameworkCore.Infrastructure.ModelCustomizerDependencies)' For more information, please visit  http://go.microsoft.com/fwlink/?LinkID=616867
   at Internal.Reflection.Core.Execution.ExecutionEnvironment.GetMethodInvoker(RuntimeTypeInfo, QMethodDefinition, RuntimeTypeInfo[], MemberInfo) + 0x26a
   at System.Reflection.Runtime.MethodInfos.NativeFormat.NativeFormatMethodCommon.GetUncachedMethodInvoker(RuntimeTypeInfo[], MemberInfo) + 0xa4
   at System.Reflection.Runtime.MethodInfos.RuntimePlainConstructorInfo`1.get_UncachedMethodInvoker() + 0x167
   at System.Reflection.Runtime.MethodInfos.RuntimeConstructorInfo.get_MethodInvoker() + 0x41
   at System.Reflection.Runtime.MethodInfos.RuntimePlainConstructorInfo`1.Invoke(BindingFlags, Binder, Object[], CultureInfo) + 0xd6
   at System.Reflection.ConstructorInfo.Invoke(Object[]) + 0x39
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, ServiceProviderEngineScope) + 0x110
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite, TArgument) + 0x152
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite, ServiceProviderEngineScope) + 0xd4
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitSingleton(SingletonCallSite, ServiceProviderEngineScope) + 0x77
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite, TArgument) + 0x1ea
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, ServiceProviderEngineScope) + 0xbf
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite, TArgument) + 0x152
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite, ServiceProviderEngineScope) + 0xd4
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitSingleton(SingletonCallSite, ServiceProviderEngineScope) + 0x77
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite, TArgument) + 0x1ea
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, ServiceProviderEngineScope) + 0xbf
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite, TArgument) + 0x152
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite, ServiceProviderEngineScope) + 0xd4
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitSingleton(SingletonCallSite, ServiceProviderEngineScope) + 0x77
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite, TArgument) + 0x1ea
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(IServiceCallSite, ServiceProviderEngineScope) + 0x3a
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass1_0.<RealizeService>b__0(ServiceProviderEngineScope) + 0xd1
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type, ServiceProviderEngineScope) + 0xce
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type) + 0x4c
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider) + 0x90
   at Microsoft.EntityFrameworkCore.Internal.DbContextServices.CreateModel() + 0x74
   at Microsoft.EntityFrameworkCore.Internal.DbContextServices.get_Model() + 0x9c
   at Microsoft.EntityFrameworkCore.Infrastructure.EntityFrameworkServicesBuilder.<>c.<TryAddCoreServices>b__7_1(IServiceProvider) + 0x3b
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite, ServiceProviderEngineScope) + 0x54
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite, TArgument) + 0xae
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite, ServiceProviderEngineScope) + 0xd4
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite, TArgument) + 0x236
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, ServiceProviderEngineScope) + 0xbf
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite, TArgument) + 0x152
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite, ServiceProviderEngineScope) + 0xd4
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite, TArgument) + 0x236
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(IServiceCallSite, ServiceProviderEngineScope) + 0x3a
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass1_0.<RealizeService>b__0(ServiceProviderEngineScope) + 0xd1
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type, ServiceProviderEngineScope) + 0xce
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type) + 0x4c
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider, Type) + 0xdb
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider) + 0x86
   at Microsoft.EntityFrameworkCore.DbContext.get_DbContextDependencies() + 0x6b
   at Microsoft.EntityFrameworkCore.DbContext.get_InternalServiceProvider() + 0x2c3
   at Microsoft.EntityFrameworkCore.DbContext.get_DbContextDependencies() + 0x57
   at Microsoft.EntityFrameworkCore.DbContext.EntryWithoutDetectChanges[TEntity](TEntity) + 0x33
   at Microsoft.EntityFrameworkCore.DbContext.SetEntityState[TEntity](TEntity, EntityState) + 0x53
   at Microsoft.EntityFrameworkCore.DbContext.Add[TEntity](TEntity) + 0x83
   at Microsoft.EntityFrameworkCore.Internal.InternalDbSet`1.Add(TEntity) + 0x85
   at hello.Program.Main(String[]) + 0xa3
   at hello!<BaseAddress>+0x76f2d6
   at hello!<BaseAddress>+0x76f363

However, when I try to add Microsoft.EntityFrameworkCore.Infrastructure.RelationalModelCustomizer as a <Type> within <Assembly Name="Microsoft.EntityFrameworkCore">, I get a parse exception of some sort:

[rubin@KURO dot-hello (master)]$ dotnet publish -r osx-x64 -c debug
Microsoft (R) Build Engine version 15.8.169+g1ccb72aefa for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restore completed in 37.31 ms for /Users/rubin/Syncthing/Source/Rubin/dot-hello/hello.csproj.
  hello -> /Users/rubin/Syncthing/Source/Rubin/dot-hello/bin/debug/netcoreapp2.1/osx-x64/hello.dll
  /usr/bin/clang
  Generating native code
EXEC : warning : RD.XML processing will change before release (https://github.com/dotnet/corert/issues/5001) [/Users/rubin/Syncthing/Source/Rubin/dot-hello/hello.csproj]
EXEC : error : [TEMPORARY EXCEPTION MESSAGE] ClassLoadGeneral: Microsoft.EntityFrameworkCore.Infrastructure.RelationalModelCustomizer, Microsoft.EntityFrameworkCore, Version=2.1.4.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 [/Users/rubin/Syncthing/Source/Rubin/dot-hello/hello.csproj]
  Internal.TypeSystem.TypeSystemException+TypeLoadException: [TEMPORARY EXCEPTION MESSAGE] ClassLoadGeneral: Microsoft.EntityFrameworkCore.Infrastructure.RelationalModelCustomizer, Microsoft.EntityFrameworkCore, Version=2.1.4.0, Culture=neutral, PublicKeyToken=adb9793829ddae60
     at Internal.TypeSystem.ThrowHelper.ThrowTypeLoadException(ExceptionStringID id, String typeName, String assemblyName)
     at Internal.TypeSystem.Ecma.EcmaModule.GetType(String nameSpace, String name, Boolean throwIfNotFound)
     at Internal.TypeSystem.CustomAttributeTypeNameParser.ResolveCustomAttributeTypeDefinitionName(String name, ModuleDesc module, Boolean throwIfNotFound)
     at Internal.TypeSystem.CustomAttributeTypeNameParser.GetTypeByCustomAttributeTypeName(ModuleDesc module, String name, Boolean throwIfNotFound, Func`4 resolver)
     at ILCompiler.RdXmlRootProvider.ProcessTypeDirective(IRootingServiceProvider rootProvider, ModuleDesc containingModule, XElement typeElement)
     at ILCompiler.RdXmlRootProvider.ProcessAssemblyDirective(IRootingServiceProvider rootProvider, XElement assemblyElement)
     at ILCompiler.RdXmlRootProvider.AddCompilationRoots(IRootingServiceProvider rootProvider)
     at ILCompiler.Compilation..ctor(DependencyAnalyzerBase`1 dependencyGraph, NodeFactory nodeFactory, IEnumerable`1 compilationRoots, ILProvider ilProvider, DebugInformationProvider debugInformationProvider, DevirtualizationManager devirtualizationManager, Logger logger)
     at ILCompiler.RyuJitCompilationBuilder.ToCompilation()
     at ILCompiler.Program.Run(String[] args)
     at ILCompiler.Program.Main(String[] args)
/Users/rubin/.nuget/packages/microsoft.dotnet.ilcompiler/1.0.0-alpha-27024-03/build/Microsoft.NETCore.Native.targets(200,5): error MSB3073: The command ""/Users/rubin/.nuget/packages/runtime.osx-x64.microsoft.dotnet.ilcompiler/1.0.0-alpha-27024-03/tools/ilc" @"obj/debug/netcoreapp2.1/osx-x64/native/hello.ilc.rsp"" exited with code 1. [/Users/rubin/Syncthing/Source/Rubin/dot-hello/hello.csproj]

Sorry for taking up so much of your time man, I'm a bit stubborn when trying to get something to work :-) Thanks a lot anyways.

@MichalStrehovsky
Copy link
Member

Your patience is admirable :).

Seems like that type lives in the Microsoft.EntityFrameworkCore.Relational assembly.

@rubin55
Copy link
Author

rubin55 commented Oct 25, 2018

Hi Michal, So I marched on and am now stumped by the following Generic Method related exception:

'Microsoft.EntityFrameworkCore.Metadata.Internal.ClrAccessorFactory<Microsoft.EntityFrameworkCore.Metadata.Internal.IClrPropertyGetter>.CreateGeneric<hello.Blog,System.Int32,System.Int32>(System.Reflection.PropertyInfo,Microsoft.EntityFrameworkCore.Metadata.IPropertyBase)'

I've commited the latest rd.xml to my dot-hello repository. If I extrapolate the above and add the following:

<Type Name="Microsoft.EntityFrameworkCore.Metadata.Internal.ClrAccessorFactory`1[[Microsoft.EntityFrameworkCore.Metadata.Internal.IClrPropertyGetter,Microsoft.EntityFrameworkCore]]">
                <Method Name="CreateGeneric" Dynamic="Required">
                    <GenericArgument Name="hello.Blog,hello" />
                    <GenericArgument Name="System.Int32,System" />
                    <GenericArgument Name="System.Int32,System" />
                </Method>
</Type>

For the Getter, which is incidentally almost identical to the Setter variant I've added previously, I get this:

EXEC : error : [TEMPORARY EXCEPTION MESSAGE] ClassLoadGeneral: System.Int32, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 [/Users/rubin/Syncthing/Source/Rubin/dot-hello/hello.csproj]
  Internal.TypeSystem.TypeSystemException+TypeLoadException: [TEMPORARY EXCEPTION MESSAGE] ClassLoadGeneral: System.Int32, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
     at Internal.TypeSystem.ThrowHelper.ThrowTypeLoadException(ExceptionStringID id, String typeName, String assemblyName)
     at Internal.TypeSystem.Ecma.EcmaModule.GetType(String nameSpace, String name, Boolean throwIfNotFound)
     at Internal.TypeSystem.CustomAttributeTypeNameParser.ResolveCustomAttributeTypeDefinitionName(String name, ModuleDesc module, Boolean throwIfNotFound)
     at Internal.TypeSystem.CustomAttributeTypeNameParser.GetTypeByCustomAttributeTypeName(ModuleDesc module, String name, Boolean throwIfNotFound, Func`4 resolver)
     at ILCompiler.RdXmlRootProvider.ProcessMethodDirective(IRootingServiceProvider rootProvider, ModuleDesc containingModule, TypeDesc containingType, XElement methodElement)
     at ILCompiler.RdXmlRootProvider.ProcessTypeDirective(IRootingServiceProvider rootProvider, ModuleDesc containingModule, XElement typeElement)
     at ILCompiler.RdXmlRootProvider.ProcessAssemblyDirective(IRootingServiceProvider rootProvider, XElement assemblyElement)
     at ILCompiler.RdXmlRootProvider.AddCompilationRoots(IRootingServiceProvider rootProvider)
     at ILCompiler.Compilation..ctor(DependencyAnalyzerBase`1 dependencyGraph, NodeFactory nodeFactory, IEnumerable`1 compilationRoots, ILProvider ilProvider, DebugInformationProvider debugInformationProvider, DevirtualizationManager devirtualizationManager, Logger logger)
     at ILCompiler.RyuJitCompilationBuilder.ToCompilation()
     at ILCompiler.Program.Run(String[] args)
     at ILCompiler.Program.Main(String[] args)

It seems to tell me it cannot find System.Int32; Am I specifying it wrongly? As always much appreciated!

@MichalStrehovsky
Copy link
Member

Thanks for keeping at it! I'm wondering how much RD.XML will be needed in the end myself :). Int32 should be referenced as <GenericArgument Name="System.Int32,System.Runtime" />

@rubin55
Copy link
Author

rubin55 commented Oct 26, 2018

Hey Michal, I reached the end! :-). I ran into a total of 99 MissingMetadataExceptions and added 103 Type entries to rd.xml.

To run, git clone my dot-hello repository, and:

dotnet ef database update

and then:

bin\debug\netcoreapp2.1\win-x64\hello.exe

It seems to work, but once (could not reproduce) I ran into the following:

debug\netcoreapp2.1\win-x64\publish\hello.exe
Let's do some SQlite!
Generic virtual method pointer lookup failure.

Declaring type handle: EEType:0x00007FF665345FD0
Target type handle: EEType:0x00007FF6651D33C8
Method name: CreateGeneric
Instantiation:
  Argument 00000000: EEType:0x00007FF6651925A0
  Argument 00000001: EEType:0x00007FF6651BECB8
  Argument 00000002: EEType:0x00007FF6651BECB8

When it works, it outputs (multiple runs done before this):

Let's do some SQlite!
1 records saved to database

All blogs in database:
 - http://blogs.msdn.com/adonet
 - http://blogs.msdn.com/adonet
 - http://blogs.msdn.com/adonet
 - http://blogs.msdn.com/adonet
 - http://blogs.msdn.com/adonet
 - http://blogs.msdn.com/adonet
 - http://blogs.msdn.com/adonet
 - http://blogs.msdn.com/adonet
 - http://blogs.msdn.com/adonet

Cool that I got it working but two questions:

  1. I suppose this is considered a suboptimal way of working, at least, I could not imagine anyone sane doing this by hand for all possible code-paths in a non-trivial program, so how will this be/work in the future?

  2. How does dotnet/ILCompiler know/handle the SQLite dependency which must be a native (C? C++?) library DLL? I checked the native executable and it only depends kernel32.dll, advapi32.dll and ole32.dll (which is awesome imho).

@MichalStrehovsky
Copy link
Member

Hey Michal, I reached the end! :-)

This is awesome! Good job and thank you! If you would like to contribute your work, we would be happy to accept a pull request adding this to the /samples/ directory of the repo.

It seems to work, but once (could not reproduce) I ran into the following

That's unfortunate. If you still had the EXE/PDB, we could use the EEType values in the message to look up what's missing. If you hit it again, please save them somewhere!

I suppose this is considered a suboptimal way of working, at least, I could not imagine anyone sane doing this by hand for all possible code-paths in a non-trivial program, so how will this be/work in the future?

The RD.XML you came up with is a "minimal RD.XML" - it results in the best executable size and fastest compilation speed, because we only compile what's really needed. You could e.g. replace the 38 Type directives in the <Assembly Name="Microsoft.EntityFrameworkCore.Relational"> section with a single <Assembly Name="Microsoft.EntityFrameworkCore.Relational" Dynamic="Required All" /> line that says "compile everything in the Microsoft.EntityFrameworkCore.Relational assembly, even if it's not needed". This makes it more likely that the app will work even if you don't exercise all the code paths in testing, at the cost of a bigger executable (measuring the size difference for EF could be interesting).

If you do this for all the assemblies in your project, the only way you could hit a reflection related issue is in MakeGenericMethod and MakeGenericType - for that, we have a feature in our backlog that again - at the cost of some EXE size, can make things work by generating less efficient, but universal generic code that works for any instantiation.

How does dotnet/ILCompiler know/handle the SQLite dependency which must be a native (C? C++?) library DLL? I checked the native executable and it only depends kernel32.dll, advapi32.dll and ole32.dll (which is awesome imho).

I assume there's a P/invoke to the SQLite library somewhere - what the compiler/runtime does is that it uses LoadLibrary/GetProcAddress to locate entrypoints in the library at runtime. We have a workitem tracking making this configurable (#2454) - this will make it possible for the compiler to generate a proper import for the SQLite library. LoadLibrary is usually a better choice because it lets us throw a managed exception if we fail to locate the library/entrypoint. If this is generated as a proper import, Windows wouldn't even let the process start up if the library/entrypoint was missing.

@daniel-packard
Copy link

Wow - thanks @rubin55, for all your hard work tracking this down and sharing your solution!

I was able to resolve my own issues with MissingMetadataException using your rd.xml as a starting point ( linked here as a convenience for future devs: https://raw.githubusercontent.com/rubin55/dot-hello/master/rd.xml )

@MichalStrehovsky - is this the recommended/correct way of resolving this issue? Is there continuing work on this front?

@MichalStrehovsky
Copy link
Member

@MichalStrehovsky - is this the recommended/correct way of resolving this issue? Is there continuing work on this front?

There's no happy place for Entity Framework right now. It requires writing RD.XML that is specific to each app because the reflection patterns end up being different.

Hopefully when Source generators become available, EF will switch to using those instead of reflection. Things will become faster in general and will work better with AOT. (The reflection would be replaced by code generated as part of the build - so instead of reflection-setting each property of the object in the object model, it would be a simple property assignment.)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants