New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce start up time by loading finished code first models from a persistent cache #275

Closed
divega opened this Issue May 15, 2017 · 21 comments

Comments

Projects
None yet
@divega
Member

divega commented May 15, 2017

(Copied description from PR #40)

Ported from CodePlex Pull Request 8468
Fix for CodePlex Work Items #1876 and #2631
Reduce start up time by loading finished Code First models from a persistent cache
Incorporates DbModelStore persistent cache to reduce Code First Startup Time

With these changes, first AppDomain calls to context.Database.Initialize for a model with just over 600 models and a null initializer dropped from 12-14 seconds to about 1.9 seconds after the edmx was written, saving 10-12 seconds on initialization. The first call to write the edmx still ran in 12-14 seconds (no noticeable delay added).

Includes code from the patch by emilcicos attached to work item #1876, with modifications to fix known issues.

Code changed after applying the patch:

  • DefaultDbModelStore invalidates (deletes) the persistent store based on comparing context dll and edmx file last updated timestamps.
  • EdmxWriter was updated to read from the DbModelStore when available. This fixed a null reference when using DropCreateAlways initializer with a DbModelStore (compiledModel.CachedModelBuilder is null when the model is loaded from the store). Added function DbModelStore.TryGetEdmx to allow the EdmxWriter to read the edmx directly from storage.
  • LazyInternalContext.ModelMatches changes in the patch were removed as they were unnecessary after the EdmxWriter fix.
  • Added unit and functional test coverage for code changes including the patch code. DefaultDbModelStoreTests functional test covers the end-to-end functionality of using a DbModelStore.

@divega divega added this to the 6.2.0 milestone May 15, 2017

@divega divega closed this May 15, 2017

@divega

This comment has been minimized.

Show comment
Hide comment
@divega

divega May 15, 2017

Member

Fixed with @j-Edge contribution at #40.

Member

divega commented May 15, 2017

Fixed with @j-Edge contribution at #40.

@tlk

This comment has been minimized.

Show comment
Hide comment
@tlk

tlk May 22, 2017

What is the recommended way to get notified when this is released (6.2)?

tlk commented May 22, 2017

What is the recommended way to get notified when this is released (6.2)?

@ErikEJ

This comment has been minimized.

Show comment
Hide comment
@ErikEJ

ErikEJ May 22, 2017

Contributor

@tlk Watch the repo ?

Contributor

ErikEJ commented May 22, 2017

@tlk Watch the repo ?

@tlk

This comment has been minimized.

Show comment
Hide comment
@tlk

tlk May 22, 2017

@ErikEJ I would prefer to receive announcements only instead of project-wide chit-chat like this, but I'll watch the repo.

Looking very much forward to the improvement on startup time. Thanks!

tlk commented May 22, 2017

@ErikEJ I would prefer to receive announcements only instead of project-wide chit-chat like this, but I'll watch the repo.

Looking very much forward to the improvement on startup time. Thanks!

@ErikEJ

This comment has been minimized.

Show comment
Hide comment
@ErikEJ

ErikEJ May 22, 2017

Contributor

@tlk Understand, this repo however is very quiet. Alternatively, subscribe to the .NET blog RSS feed.

Contributor

ErikEJ commented May 22, 2017

@tlk Understand, this repo however is very quiet. Alternatively, subscribe to the .NET blog RSS feed.

@divega divega changed the title from Reduce start up time by loading finished Code First models from a persistent cache to Reduce start up time by loading finished code first models from a persistent cache May 23, 2017

@NicoJuicy

This comment has been minimized.

Show comment
Hide comment
@NicoJuicy

NicoJuicy Jun 27, 2017

@divega How would i set this on DbMigrationsConfiguration(Of DataContext)?

This is in DbConfiguration, but not in DbMigrationsConfiguration.

this.SetModelStore(new DefaultDbModelStore(Directory.GetCurrentDirectory())); 

NicoJuicy commented Jun 27, 2017

@divega How would i set this on DbMigrationsConfiguration(Of DataContext)?

This is in DbConfiguration, but not in DbMigrationsConfiguration.

this.SetModelStore(new DefaultDbModelStore(Directory.GetCurrentDirectory())); 
@michalczerwinski

This comment has been minimized.

Show comment
Hide comment
@michalczerwinski

michalczerwinski Jul 5, 2017

I have a question: is the DefaultDbModelStore suppose to be set as default in 6.2?

michalczerwinski commented Jul 5, 2017

I have a question: is the DefaultDbModelStore suppose to be set as default in 6.2?

@divega

This comment has been minimized.

Show comment
Hide comment
@divega

divega Jul 9, 2017

Member

@NicoJuicy this can only be set in a DbConfiguration

@michalczerwinski it is not set by default, you have to opt in explicitly. That is something we could consider for a future release if the serialization as well as the invalidation mechanism proves to be very reliable.

Member

divega commented Jul 9, 2017

@NicoJuicy this can only be set in a DbConfiguration

@michalczerwinski it is not set by default, you have to opt in explicitly. That is something we could consider for a future release if the serialization as well as the invalidation mechanism proves to be very reliable.

@TimRowe

This comment has been minimized.

Show comment
Hide comment
@TimRowe

TimRowe Aug 15, 2017

This is as improvement of the code first mode , why you say it has to do with edmx file? edmx file isn't with the db first mode?

TimRowe commented Aug 15, 2017

This is as improvement of the code first mode , why you say it has to do with edmx file? edmx file isn't with the db first mode?

@divega

This comment has been minimized.

Show comment
Hide comment
@divega

divega Aug 16, 2017

Member

@TimRowe the improvement serializes the computed code first model to an artifact that uses the edmx file format. I.e. the edmx becomes an implementation detail of code first when you are using this option.

Member

divega commented Aug 16, 2017

@TimRowe the improvement serializes the computed code first model to an artifact that uses the edmx file format. I.e. the edmx becomes an implementation detail of code first when you are using this option.

@brux88

This comment has been minimized.

Show comment
Hide comment
@brux88

brux88 Sep 13, 2017

Hi,but if i use code first migrations , how i can do use persistent caching? My database configuration use dbmigrationsconfiguration and not use dbconfiguration

brux88 commented Sep 13, 2017

Hi,but if i use code first migrations , how i can do use persistent caching? My database configuration use dbmigrationsconfiguration and not use dbconfiguration

@abdelrady

This comment has been minimized.

Show comment
Hide comment
@abdelrady

abdelrady commented Oct 28, 2017

@SylwesterZarebski

This comment has been minimized.

Show comment
Hide comment
@SylwesterZarebski

SylwesterZarebski Nov 2, 2017

@divega commented on 9 lip 2017, 09:19 CEST:

@NicoJuicy this can only be set in a DbConfiguration

It means that is not possible to use persistent caching with DbMigrationsConfiguration?

SylwesterZarebski commented Nov 2, 2017

@divega commented on 9 lip 2017, 09:19 CEST:

@NicoJuicy this can only be set in a DbConfiguration

It means that is not possible to use persistent caching with DbMigrationsConfiguration?

@brux88

This comment has been minimized.

Show comment
Hide comment
@brux88

brux88 commented Nov 2, 2017

@SylwesterZarebski you can use both

@SylwesterZarebski

This comment has been minimized.

Show comment
Hide comment
@SylwesterZarebski

SylwesterZarebski Nov 2, 2017

Could You share some code? Now i have:

Database.SetInitializer(new Migrations.CheckAndMigrateDatabaseToLatestVersion<AppContext, Migrations.Configuration>());

where CheckAndMigrateDatabaseToLatestVersion is:

internal sealed class CheckAndMigrateDatabaseToLatestVersion<TContext, TMigrationsConfiguration> : IDatabaseInitializer<TContext>
        where TContext : DbContext
        where TMigrationsConfiguration : DbMigrationsConfiguration<TContext>, new()

How can i add persistant caching?

SylwesterZarebski commented Nov 2, 2017

Could You share some code? Now i have:

Database.SetInitializer(new Migrations.CheckAndMigrateDatabaseToLatestVersion<AppContext, Migrations.Configuration>());

where CheckAndMigrateDatabaseToLatestVersion is:

internal sealed class CheckAndMigrateDatabaseToLatestVersion<TContext, TMigrationsConfiguration> : IDatabaseInitializer<TContext>
        where TContext : DbContext
        where TMigrationsConfiguration : DbMigrationsConfiguration<TContext>, new()

How can i add persistant caching?

@brux88

This comment has been minimized.

Show comment
Hide comment
@brux88

brux88 Nov 2, 2017

sure.
I have two configuration file for my context (SednaContext).

this configuration for migrations:

public sealed class Configuration : DbMigrationsConfiguration<UnitOfWork.SednaContext>
  {
      public Configuration()
      {
          AutomaticMigrationsEnabled = false;
         //  AutomaticMigrationDataLossAllowed = true;
          MigrationsDirectory = @"Migrations_SednaContext";
      }


      protected override void Seed(UnitOfWork.SednaContext context)
      {
    
      }

  }

and this configuration for chaching:

 public sealed class ConfigurationCaching : DbConfiguration
    { 
        public ConfigurationCaching(): base()
        {
            var pathEDMX = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData), @"Sedna\EDMX");

            if (!Directory.Exists(pathEDMX))
            {
                DirectorySecurity securityRules = new DirectorySecurity();
                securityRules.AddAccessRule(new FileSystemAccessRule(
                                            new SecurityIdentifier(WellKnownSidType.WorldSid, null),
                                            FileSystemRights.FullControl, InheritanceFlags.ObjectInherit |
                                            InheritanceFlags.ContainerInherit, PropagationFlags.NoPropagateInherit, AccessControlType.Allow));
                Directory.CreateDirectory(pathEDMX, securityRules);
            }
            this.SetModelStore(new DefaultDbModelStore(pathEDMX/*Directory.GetCurrentDirectory()*/));
        }

    }

then you set attribute on your DbContext file:

  [DbConfigurationType(typeof(ConfigurationCaching))]
    public class SednaContext : DbContext
    {
...

brux88 commented Nov 2, 2017

sure.
I have two configuration file for my context (SednaContext).

this configuration for migrations:

public sealed class Configuration : DbMigrationsConfiguration<UnitOfWork.SednaContext>
  {
      public Configuration()
      {
          AutomaticMigrationsEnabled = false;
         //  AutomaticMigrationDataLossAllowed = true;
          MigrationsDirectory = @"Migrations_SednaContext";
      }


      protected override void Seed(UnitOfWork.SednaContext context)
      {
    
      }

  }

and this configuration for chaching:

 public sealed class ConfigurationCaching : DbConfiguration
    { 
        public ConfigurationCaching(): base()
        {
            var pathEDMX = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData), @"Sedna\EDMX");

            if (!Directory.Exists(pathEDMX))
            {
                DirectorySecurity securityRules = new DirectorySecurity();
                securityRules.AddAccessRule(new FileSystemAccessRule(
                                            new SecurityIdentifier(WellKnownSidType.WorldSid, null),
                                            FileSystemRights.FullControl, InheritanceFlags.ObjectInherit |
                                            InheritanceFlags.ContainerInherit, PropagationFlags.NoPropagateInherit, AccessControlType.Allow));
                Directory.CreateDirectory(pathEDMX, securityRules);
            }
            this.SetModelStore(new DefaultDbModelStore(pathEDMX/*Directory.GetCurrentDirectory()*/));
        }

    }

then you set attribute on your DbContext file:

  [DbConfigurationType(typeof(ConfigurationCaching))]
    public class SednaContext : DbContext
    {
...
@SylwesterZarebski

This comment has been minimized.

Show comment
Hide comment
@SylwesterZarebski

SylwesterZarebski Nov 2, 2017

Thanks, i'll try to use it that way.

SylwesterZarebski commented Nov 2, 2017

Thanks, i'll try to use it that way.

@srudin

This comment has been minimized.

Show comment
Hide comment
@srudin

srudin Nov 21, 2017

This is not working for me.

What I did is very simple and I think exactly what has been suggested above:

I saved the model using
EdmxWriter.WriteEdmx(context, writer);

In my DbConfiguration implementation I set
SetModelStore(new DbModelStore());

DbModelStore is my implementation of DefaultDbModelStore where I overwrite GetFilePath().

When making the first call to EntitySet I get
InvalidOperationException: The entity type EdmMetadata is not part of the model for the current context.
System.Data.Entity.Internal.InternalContext.UpdateEntitySetMappingsForType(Type entityType) +273
System.Data.Entity.Internal.InternalContext.GetEntitySetAndBaseTypeForType(Type entityType) +31
System.Data.Entity.Internal.Linq.InternalSet1.Initialize() +77 System.Data.Entity.Internal.Linq.InternalSet1.AsNoTracking() +29
System.Data.Entity.Infrastructure.DbQuery1.AsNoTracking() +75 System.Data.Entity.Internal.EdmMetadataRepository.QueryForModelHash(Func2 createContext) +159
System.Data.Entity.Internal.ModelCompatibilityChecker.CompatibleWithModel(InternalContext internalContext, ModelHashCalculator modelHashCalculator, Boolean throwIfNoMetadata, DatabaseExistenceState existenceState) +120
System.Data.Entity.Internal.InternalContext.CompatibleWithModel(Boolean throwIfNoMetadata, DatabaseExistenceState existenceState) +77
System.Data.Entity.CreateDatabaseIfNotExists1.InitializeDatabase(TContext context) +155 System.Data.Entity.Internal.InternalContext.PerformInitializationAction(Action action) +71 System.Data.Entity.Internal.InternalContext.PerformDatabaseInitialization() +482 System.Data.Entity.Internal.RetryAction1.PerformAction(TInput input) +174
System.Data.Entity.Internal.LazyInternalContext.InitializeDatabaseAction(Action1 action) +273 System.Data.Entity.Internal.InternalContext.GetEntitySetAndBaseTypeForType(Type entityType) +20 System.Data.Entity.Internal.Linq.InternalSet1.Initialize() +77
System.Data.Entity.Internal.Linq.InternalSet1.AsNoTracking() +29 System.Data.Entity.Infrastructure.DbQuery1.AsNoTracking() +75
[my call to EntitySet]

I have been able to get it to work when I create a DbCompiledModel instead:
var model = EdmxReader.Read(reader, "dbo");

I can then pass the model to the constructor of the context. However this is much more complicated mostly because of the stupid null check of the model in the constructor (!) so I have to make sure to call the right constructor overload depending on whether I already have a saved model or not.

A fix for the null check or an explanation why the DefaultDbModelStore is not working would be appreciated. I consider both to be bugs.

srudin commented Nov 21, 2017

This is not working for me.

What I did is very simple and I think exactly what has been suggested above:

I saved the model using
EdmxWriter.WriteEdmx(context, writer);

In my DbConfiguration implementation I set
SetModelStore(new DbModelStore());

DbModelStore is my implementation of DefaultDbModelStore where I overwrite GetFilePath().

When making the first call to EntitySet I get
InvalidOperationException: The entity type EdmMetadata is not part of the model for the current context.
System.Data.Entity.Internal.InternalContext.UpdateEntitySetMappingsForType(Type entityType) +273
System.Data.Entity.Internal.InternalContext.GetEntitySetAndBaseTypeForType(Type entityType) +31
System.Data.Entity.Internal.Linq.InternalSet1.Initialize() +77 System.Data.Entity.Internal.Linq.InternalSet1.AsNoTracking() +29
System.Data.Entity.Infrastructure.DbQuery1.AsNoTracking() +75 System.Data.Entity.Internal.EdmMetadataRepository.QueryForModelHash(Func2 createContext) +159
System.Data.Entity.Internal.ModelCompatibilityChecker.CompatibleWithModel(InternalContext internalContext, ModelHashCalculator modelHashCalculator, Boolean throwIfNoMetadata, DatabaseExistenceState existenceState) +120
System.Data.Entity.Internal.InternalContext.CompatibleWithModel(Boolean throwIfNoMetadata, DatabaseExistenceState existenceState) +77
System.Data.Entity.CreateDatabaseIfNotExists1.InitializeDatabase(TContext context) +155 System.Data.Entity.Internal.InternalContext.PerformInitializationAction(Action action) +71 System.Data.Entity.Internal.InternalContext.PerformDatabaseInitialization() +482 System.Data.Entity.Internal.RetryAction1.PerformAction(TInput input) +174
System.Data.Entity.Internal.LazyInternalContext.InitializeDatabaseAction(Action1 action) +273 System.Data.Entity.Internal.InternalContext.GetEntitySetAndBaseTypeForType(Type entityType) +20 System.Data.Entity.Internal.Linq.InternalSet1.Initialize() +77
System.Data.Entity.Internal.Linq.InternalSet1.AsNoTracking() +29 System.Data.Entity.Infrastructure.DbQuery1.AsNoTracking() +75
[my call to EntitySet]

I have been able to get it to work when I create a DbCompiledModel instead:
var model = EdmxReader.Read(reader, "dbo");

I can then pass the model to the constructor of the context. However this is much more complicated mostly because of the stupid null check of the model in the constructor (!) so I have to make sure to call the right constructor overload depending on whether I already have a saved model or not.

A fix for the null check or an explanation why the DefaultDbModelStore is not working would be appreciated. I consider both to be bugs.

@srudin

This comment has been minimized.

Show comment
Hide comment
@srudin

srudin Nov 22, 2017

Created a new issue instead: #404

srudin commented Nov 22, 2017

Created a new issue instead: #404

@ivanovevgeny

This comment has been minimized.

Show comment
Hide comment
@ivanovevgeny

ivanovevgeny Nov 30, 2017

My project's start time is about 10 sec.
I use migrations.

I added this model caching feature

public class ProductsDbConfiguration : DbConfiguration
    {
        public ProductsDbConfiguration() : base()
        {
            this.SetModelStore(new DefaultDbModelStore(AppDomain.CurrentDomain.BaseDirectory));
        }
    }

    [DbConfigurationType(typeof(ProductsDbConfiguration))]
    public class ProductsContext : DbContext
    {...}

Edmx file created succesffully on first run.
Than i changed some service, model is located in another project in solution and has not been changed. I had not see any improvements on loading time - 10 sec.

What i'm doing wrong?

Migration init code:

// Global.asax.cs --------
Database.SetInitializer(new MigrateDatabaseToLatestVersion<ProductsContext, Products.Infrastructure.Migrations.Configuration>());
// ---------

namespace Products.Infrastructure.Migrations
{
    using System.Data.Entity.Migrations;

    public sealed class Configuration : DbMigrationsConfiguration<Repositories.ProductsContext>
    {
        public Configuration()
        {
            AutomaticMigrationsEnabled = false;
            CommandTimeout = 60 * 5; 
        }
    }
}

ivanovevgeny commented Nov 30, 2017

My project's start time is about 10 sec.
I use migrations.

I added this model caching feature

public class ProductsDbConfiguration : DbConfiguration
    {
        public ProductsDbConfiguration() : base()
        {
            this.SetModelStore(new DefaultDbModelStore(AppDomain.CurrentDomain.BaseDirectory));
        }
    }

    [DbConfigurationType(typeof(ProductsDbConfiguration))]
    public class ProductsContext : DbContext
    {...}

Edmx file created succesffully on first run.
Than i changed some service, model is located in another project in solution and has not been changed. I had not see any improvements on loading time - 10 sec.

What i'm doing wrong?

Migration init code:

// Global.asax.cs --------
Database.SetInitializer(new MigrateDatabaseToLatestVersion<ProductsContext, Products.Infrastructure.Migrations.Configuration>());
// ---------

namespace Products.Infrastructure.Migrations
{
    using System.Data.Entity.Migrations;

    public sealed class Configuration : DbMigrationsConfiguration<Repositories.ProductsContext>
    {
        public Configuration()
        {
            AutomaticMigrationsEnabled = false;
            CommandTimeout = 60 * 5; 
        }
    }
}
@ajcvickers

This comment has been minimized.

Show comment
Hide comment
@ajcvickers

ajcvickers Nov 30, 2017

Member

@ivanovevgeny It could be that Code First model creation is not the perf bottleneck. It might be worth reading EF perf white paper and checking to see if any of the guidance there can help.

Member

ajcvickers commented Nov 30, 2017

@ivanovevgeny It could be that Code First model creation is not the perf bottleneck. It might be worth reading EF perf white paper and checking to see if any of the guidance there can help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment