Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

[Proposal] Schema cache dump #5162

Merged
merged 5 commits into from

4 participants

@kennyj
Collaborator

In my experience, if we had many models (ex. one hundred), Rails boot was slowly.
According to production log, it seems that AR's schema data loading is slowly especially.

Thus I've implemented schema cache dumping. Please review it.
I guess this implementation has many fixing point ;)

Usage:

$ edit config/environments/production.rb
config.use_schema_cache_dump = true
$ RAILS_ENV=production bundle rake db:schema:cache:dump
=> generate db/schema_cache.dump
$ RAILS_ENV=production rails s
@tenderlove
Owner

I like this idea, but can we change a few things?

First, can we just implement marshal_dump and marshal_load on the SchemaCache object? Second, I'm not sure that loading every model is the best idea for the schema cache. What about asking for all the tables and populating the cache that way? For example:

schema_cache.connection.tables.each do |table|
  schema_cache.populate(table)
end

Maybe not a populate method, but something. I don't really like the idea of requiring every model in order to get the schema cache.

I have another idea that is related to this: can we enable schema caching by default? We can use the migration version to determine if the cache should be expired. Maybe add a version method to the schema cache object.

Anyway, I really like this feature.

@kennyj
Collaborator

Thank you for comment ! I'll improve the implement :)

@kennyj
Collaborator

Hi @tenderlove

Done!
Please review new some commits.

@kennyj

A hash with default_proc can't be dumped.

@tenderlove tenderlove merged commit 447ecb0 into rails:master
@josevalim

This configuration should not be here. It is specific to Active Record and therefore should be defined in Active Record railtie.

Collaborator

Certainly, I agree with you.. Do you mean kennyj@82bd05a ?

@dhh
Owner

Can you provide some benchmarks for this optimization? How much does it actually speed things up?

@kennyj
Collaborator

I'll provide it, but I've many works during this week. Please, just wait a moment a few days.

@kennyj
Collaborator

Sorry for keeping you waiting for this reply.

I tested about this performance.
But this result was not expected one.

・building environment steps
https://gist.github.com/3730757
・test result
https://gist.github.com/3730759

In my experience on Oracle, the queries to data dictionary were very slow when having many data.
Thus, by similar approache, we solved that problem.

I'll try to research a little more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
View
23 activerecord/CHANGELOG.md
@@ -1,5 +1,28 @@
## Rails 4.0.0 (unreleased) ##
+* Added the schema cache dump feature.
+
+ `Schema cache dump` feature was implemetend. This feature can dump/load internal state of `SchemaCache` instance
+ because we want to boot rails more quickly when we have many models.
+
+ Usage notes:
+
+ 1) execute rake task.
+ RAILS_ENV=production bundle exec rake db:schema:cache:dump
+ => generate db/schema_cache.dump
+
+ 2) add config.use_schema_cache_dump = true in config/production.rb. BTW, true is default.
+
+ 3) boot rails.
+ RAILS_ENV=production bundle exec rails server
+ => use db/schema_cache.db
+
+ 4) If you remove clear dumped cache, execute rake task.
+ RAILS_ENV=production bundle exec rake db:schema:cache:clear
+ => remove db/schema_cache.dump
+
+ *kennyj*
+
* Added support for partial indices to PostgreSQL adapter
The `add_index` method now supports a `where` option that receives a
View
5 activerecord/lib/active_record/connection_adapters/abstract_adapter.rb
@@ -86,6 +86,11 @@ def lease
end
end
+ def schema_cache=(cache)
+ cache.connection = self
+ @schema_cache = cache
+ end
+
def expire
@in_use = false
end
View
64 activerecord/lib/active_record/connection_adapters/schema_cache.rb
@@ -1,26 +1,17 @@
module ActiveRecord
module ConnectionAdapters
class SchemaCache
- attr_reader :columns, :columns_hash, :primary_keys, :tables
- attr_reader :connection
+ attr_reader :columns, :columns_hash, :primary_keys, :tables, :version
+ attr_accessor :connection
def initialize(conn)
@connection = conn
- @tables = {}
- @columns = Hash.new do |h, table_name|
- h[table_name] = conn.columns(table_name)
- end
-
- @columns_hash = Hash.new do |h, table_name|
- h[table_name] = Hash[columns[table_name].map { |col|
- [col.name, col]
- }]
- end
-
- @primary_keys = Hash.new do |h, table_name|
- h[table_name] = table_exists?(table_name) ? conn.primary_key(table_name) : nil
- end
+ @columns = {}
+ @columns_hash = {}
+ @primary_keys = {}
+ @tables = {}
+ prepare_default_proc
end
# A cached lookup for table existence.
@@ -30,12 +21,22 @@ def table_exists?(name)
@tables[name] = connection.table_exists?(name)
end
+ # Add internal cache for table with +table_name+.
+ def add(table_name)
+ if table_exists?(table_name)
+ @primary_keys[table_name]
+ @columns[table_name]
+ @columns_hash[table_name]
+ end
+ end
+
# Clears out internal caches
def clear!
@columns.clear
@columns_hash.clear
@primary_keys.clear
@tables.clear
+ @version = nil
end
# Clear out internal caches for table with +table_name+.
@@ -45,6 +46,37 @@ def clear_table_cache!(table_name)
@primary_keys.delete table_name
@tables.delete table_name
end
+
+ def marshal_dump
+ # if we get current version during initialization, it happens stack over flow.
+ @version = ActiveRecord::Migrator.current_version
+ [@version] + [:@columns, :@columns_hash, :@primary_keys, :@tables].map do |val|
+ self.instance_variable_get(val).inject({}) { |h, v| h[v[0]] = v[1]; h }
+ end
+ end
+
+ def marshal_load(array)
+ @version, @columns, @columns_hash, @primary_keys, @tables = array
+ prepare_default_proc
+ end
+
+ private
+
+ def prepare_default_proc
+ @columns.default_proc = Proc.new do |h, table_name|
+ h[table_name] = connection.columns(table_name)
+ end
+
+ @columns_hash.default_proc = Proc.new do |h, table_name|
+ h[table_name] = Hash[columns[table_name].map { |col|
+ [col.name, col]
+ }]
+ end
+
+ @primary_keys.default_proc = Proc.new do |h, table_name|
+ h[table_name] = table_exists?(table_name) ? connection.primary_key(table_name) : nil
+ end
+ end
end
end
end
View
17 activerecord/lib/active_record/railtie.rb
@@ -107,7 +107,7 @@ class Railtie < Rails::Railtie
config.watchable_files.concat ["#{app.root}/db/schema.rb", "#{app.root}/db/structure.sql"]
end
- config.after_initialize do
+ config.after_initialize do |app|
ActiveSupport.on_load(:active_record) do
ActiveRecord::Base.instantiate_observers
@@ -115,6 +115,21 @@ class Railtie < Rails::Railtie
ActiveRecord::Base.instantiate_observers
end
end
+
+ ActiveSupport.on_load(:active_record) do
+ if app.config.use_schema_cache_dump
+ filename = File.join(app.config.paths["db"].first, "schema_cache.dump")
+ if File.file?(filename)
+ cache = Marshal.load(open(filename, 'rb') { |f| f.read })
+ if cache.version == ActiveRecord::Migrator.current_version
+ ActiveRecord::Base.connection.schema_cache = cache
+ else
+ warn "schema_cache.dump is expired. Current version is #{ActiveRecord::Migrator.current_version}, but cache version is #{cache.version}."
+ end
+ end
+ end
+ end
+
end
end
end
View
19 activerecord/lib/active_record/railties/databases.rake
@@ -372,6 +372,25 @@ db_namespace = namespace :db do
task :load_if_ruby => 'db:create' do
db_namespace["schema:load"].invoke if ActiveRecord::Base.schema_format == :ruby
end
+
+ namespace :cache do
+ desc 'Create a db/schema_cache.dump file.'
+ task :dump => :environment do
+ con = ActiveRecord::Base.connection
+ filename = File.join(Rails.application.config.paths["db"].first, "schema_cache.dump")
+
+ con.schema_cache.clear!
+ con.tables.each { |table| con.schema_cache.add(table) }
+ open(filename, 'wb') { |f| f.write(Marshal.dump(con.schema_cache)) }
+ end
+
+ desc 'Clear a db/schema_cache.dump file.'
+ task :clear => :environment do
+ filename = File.join(Rails.application.config.paths["db"].first, "schema_cache.dump")
+ FileUtils.rm(filename) if File.exists?(filename)
+ end
+ end
+
end
namespace :structure do
View
15 activerecord/test/cases/connection_adapters/schema_cache_test.rb
@@ -39,6 +39,21 @@ def test_clearing
assert_equal 0, @cache.tables.size
assert_equal 0, @cache.primary_keys.size
end
+
+ def test_dump_and_load
+ @cache.columns['posts']
+ @cache.columns_hash['posts']
+ @cache.tables['posts']
+ @cache.primary_keys['posts']
+
+ @cache = Marshal.load(Marshal.dump(@cache))
+
+ assert_equal 12, @cache.columns['posts'].size
+ assert_equal 12, @cache.columns_hash['posts'].size
+ assert @cache.tables['posts']
+ assert_equal 'id', @cache.primary_keys['posts']
+ end
+
end
end
end
View
3  railties/lib/rails/application/configuration.rb
@@ -11,7 +11,7 @@ class Configuration < ::Rails::Engine::Configuration
:force_ssl, :helpers_paths, :logger, :log_tags, :preload_frameworks,
:railties_order, :relative_url_root, :secret_token,
:serve_static_assets, :ssl_options, :static_cache_control, :session_options,
- :time_zone, :reload_classes_only_on_change
+ :time_zone, :reload_classes_only_on_change, :use_schema_cache_dump
attr_writer :log_level
attr_reader :encoding
@@ -41,6 +41,7 @@ def initialize(*)
@file_watcher = ActiveSupport::FileUpdateChecker
@exceptions_app = nil
@autoflush_log = true
+ @use_schema_cache_dump = true
@assets = ActiveSupport::OrderedOptions.new
@assets.enabled = false
View
26 railties/test/application/initializers/frameworks_test.rb
@@ -193,5 +193,31 @@ def from_bar_helper
require "#{app_path}/config/environment"
assert_nil defined?(ActiveRecord::Base)
end
+
+ test "use schema cache dump" do
+ Dir.chdir(app_path) do
+ `rails generate model post title:string`
+ `bundle exec rake db:migrate`
+ `bundle exec rake db:schema:cache:dump`
+ end
+ require "#{app_path}/config/environment"
+ ActiveRecord::Base.connection.drop_table("posts") # force drop posts table for test.
+ assert ActiveRecord::Base.connection.schema_cache.tables["posts"]
+ end
+
+ test "expire schema cache dump" do
+ Dir.chdir(app_path) do
+ `rails generate model post title:string`
+ `bundle exec rake db:migrate`
+ `bundle exec rake db:schema:cache:dump`
+
+ `bundle exec rake db:rollback`
+ end
+ silence_warnings {
+ require "#{app_path}/config/environment"
+ assert !ActiveRecord::Base.connection.schema_cache.tables["posts"]
+ }
+ end
+
end
end
View
19 railties/test/application/rake_test.rb
@@ -138,5 +138,24 @@ def test_rake_dump_structure_should_respect_db_structure_env_variable
end
assert File.exists?(File.join(app_path, 'db', 'my_structure.sql'))
end
+
+ def test_rake_dump_schema_cache
+ Dir.chdir(app_path) do
+ `rails generate model post title:string`
+ `rails generate model product name:string`
+ `bundle exec rake db:migrate`
+ `bundle exec rake db:schema:cache:dump`
+ end
+ assert File.exists?(File.join(app_path, 'db', 'schema_cache.dump'))
+ end
+
+ def test_rake_clear_schema_cache
+ Dir.chdir(app_path) do
+ `bundle exec rake db:schema:cache:dump`
+ `bundle exec rake db:schema:cache:clear`
+ end
+ assert !File.exists?(File.join(app_path, 'db', 'schema_cache.dump'))
+ end
+
end
end
Something went wrong with that request. Please try again.