Permalink
Browse files

Basic API for connection switching

This PR adds the ability to 1) connect to multiple databases in a model,
and 2) switch between those connections using a block.

To connect a model to a set of databases for writing and reading use
the following API. This API supercedes `establish_connection`. The
`writing` and `reading` keys represent handler / role names and
`animals` and `animals_replica` represents the database key to look up
the configuration hash from.

```
class AnimalsBase < ApplicationRecord
  connects_to database: { writing: :animals, reading: :animals_replica }
end
```

Inside the application - outside the model declaration - we can switch
connections with a block call to `connected_to`.

If we want to connect to a db that isn't default (ie readonly_slow) we
can connect like this:

Outside the model we may want to connect to a new database (one that is
not in the default writing/reading set) - for example a slow replica for
making slow queries. To do this we have the `connected_to` method that
takes a `database` hash that matches the signature of `connects_to`. The
`connected_to` method also takes a block.

```
AcitveRecord::Base.connected_to(database: { slow_readonly: :primary_replica_slow }) do
  ModelInPrimary.do_something_thats_slow
end
```

For models that are already loaded and connections that are already
connected, `connected_to` doesn't need to pass in a `database` because
you may want to run queries against multiple databases using a specific
role/handler.

In this case `connected_to` can take a `role` and use that to swap on
the connection passed. This simplies queries - and matches how we do it
in GitHub. Once you're connected to the database you don't need to
re-connect, we assume the connection is in the pool and simply pass the
handler we'd like to swap on.

```
ActiveRecord::Base.connected_to(role: :reading) do
  Dog.read_something_from_dog
  ModelInPrimary.do_something_from_model_in_primary
end
```
  • Loading branch information...
eileencodes committed Sep 28, 2018
1 parent 58999af commit 31021a8c85bb80300deb01db96876858ff417f4a
View
@@ -1,3 +1,39 @@
* Add basic API for connection switching to support multiple databases.
1) Adds a `connects_to` method for models to connect to multiple databases. Example:
```
class AnimalsModel < ApplicationRecord
self.abstract_class = true
connects_to database: { writing: :animals_primary, reading: :animals_replica }
end
class Dog < AnimalsModel
# connected to both the animals_primary db for writing and the animals_replica for reading
end
```
2) Adds a `connected_to` block method for switching connection roles or connecting to
a database that the model didn't connect to. Connecting to the database in this block is
useful when you have another defined connection, for example `slow_replica` that you don't
want to connect to by default but need in the console, or a specific code block.
```
ActiveRecord::Base.connected_to(role: :reading) do
Dog.first # finds dog from replica connected to AnimalsBase
Book.first # doesn't have a reading connection, will raise an error
end
```
```
ActiveRecord::Base.connected_to(database: :slow_replica) do
SlowReplicaModel.first # if the db config has a slow_replica configuration this will be used to do the lookup, otherwise this will throw an exception
end
```
*Eileen M. Uchitelle*
* Enum raises on invalid definition values
When defining a Hash enum it can be easy to use [] instead of {}. This
@@ -47,6 +47,92 @@ module ConnectionHandling
# The exceptions AdapterNotSpecified, AdapterNotFound and +ArgumentError+
# may be returned on an error.
def establish_connection(config_or_env = nil)
config_hash = resolve_config_for_connection(config_or_env)
connection_handler.establish_connection(config_hash)
end
# Connects a model to the databases specified. The +database+ keyword
# takes a hash consisting of a +role+ and a +database_key+.
#
# This will create a connection handler for switching between connections,
# look up the config hash using the +database_key+ and finally
# establishes a connection to that config.
#
# class AnimalsModel < ApplicationRecord
# self.abstract_class = true
#
# connects_to database: { writing: :primary, reading: :primary_replica }
# end
#
# Returns an array of established connections.
def connects_to(database: {})
connections = []
database.each do |role, database_key|
config_hash = resolve_config_for_connection(database_key)
handler = lookup_connection_handler(role.to_sym)
connections << handler.establish_connection(config_hash)
end
connections
end
# Connects to a database or role (ex writing, reading, or another
# custom role) for the duration of the block.
#
# If a role is passed, Active Record will look up the connection
# based on the requested role:
#
# ActiveRecord::Base.connected_to(role: :writing) do
# Dog.create! # creates dog using dog connection
# end
#
# ActiveRecord::Base.connected_to(role: :reading) do
# Dog.create! # throws exception because we're on a replica
# end
#
# ActiveRecord::Base.connected_to(role: :unknown_ode) do
# # raises exception due to non-existent role
# end
#
# For cases where you may want to connect to a database outside of the model,
# you can use +connected_to+ with a +database+ argument. The +database+ argument
# expects a symbol that corresponds to the database key in your config.
#
# This will connect to a new database for the queries inside the block.
#
# ActiveRecord::Base.connected_to(database: :animals_slow_replica) do
# Dog.run_a_long_query # runs a long query while connected to the +animals_slow_replica+
# end
def connected_to(database: nil, role: nil, &blk)
if database && role
raise ArgumentError, "connected_to can only accept a database or role argument, but not both arguments."
elsif database
config_hash = resolve_config_for_connection(database)
handler = lookup_connection_handler(database.to_sym)
with_handler(database.to_sym) do
handler.establish_connection(config_hash)
return yield
end
elsif role
with_handler(role.to_sym, &blk)
else
raise ArgumentError, "must provide a `database` or a `role`."
end
end
def lookup_connection_handler(handler_key) # :nodoc:
connection_handlers[handler_key] ||= ActiveRecord::ConnectionAdapters::ConnectionHandler.new
end
def with_handler(handler_key, &blk) # :nodoc:
handler = lookup_connection_handler(handler_key)
swap_connection_handler(handler, &blk)
end
def resolve_config_for_connection(config_or_env) # :nodoc:
raise "Anonymous class is not allowed." unless name
config_or_env ||= DEFAULT_ENV.call.to_sym
@@ -57,7 +143,7 @@ def establish_connection(config_or_env = nil)
config_hash = resolver.resolve(config_or_env, pool_name).symbolize_keys
config_hash[:name] = pool_name
connection_handler.establish_connection(config_hash)
config_hash
end
# Returns the connection currently associated with the class. This can
@@ -118,5 +204,14 @@ def clear_cache! # :nodoc:
delegate :clear_active_connections!, :clear_reloadable_connections!,
:clear_all_connections!, :flush_idle_connections!, to: :connection_handler
private
def swap_connection_handler(handler, &blk) # :nodoc:
old_handler, ActiveRecord::Base.connection_handler = ActiveRecord::Base.connection_handler, handler
yield
ensure
ActiveRecord::Base.connection_handler = old_handler
end
end
end
@@ -124,6 +124,8 @@ def self.configurations
mattr_accessor :belongs_to_required_by_default, instance_accessor: false
mattr_accessor :connection_handlers, instance_accessor: false, default: {}
class_attribute :default_connection_handler, instance_writer: false
self.filter_attributes = []
@@ -137,6 +139,7 @@ def self.connection_handler=(handler)
end
self.default_connection_handler = ConnectionAdapters::ConnectionHandler.new
self.connection_handlers = { writing: ActiveRecord::Base.default_connection_handler }
end
module ClassMethods
@@ -0,0 +1,193 @@
# frozen_string_literal: true
require "cases/helper"
require "models/person"
module ActiveRecord
module ConnectionAdapters
class ConnectionHandlersMultiDbTest < ActiveRecord::TestCase
self.use_transactional_tests = false
fixtures :people
def setup
@handlers = { writing: ConnectionHandler.new, reading: ConnectionHandler.new }
@rw_handler = @handlers[:writing]
@ro_handler = @handlers[:reading]
@spec_name = "primary"
@rw_pool = @handlers[:writing].establish_connection(ActiveRecord::Base.configurations["arunit"])
@ro_pool = @handlers[:reading].establish_connection(ActiveRecord::Base.configurations["arunit"])
end
def teardown
ActiveRecord::Base.connection_handlers = { writing: ActiveRecord::Base.default_connection_handler }
end
class MultiConnectionTestModel < ActiveRecord::Base
end
def test_multiple_connection_handlers_works_in_a_threaded_environment
tf_writing = Tempfile.open "test_writing"
tf_reading = Tempfile.open "test_reading"
MultiConnectionTestModel.connects_to database: { writing: { database: tf_writing.path, adapter: "sqlite3" }, reading: { database: tf_reading.path, adapter: "sqlite3" } }
MultiConnectionTestModel.connection.execute("CREATE TABLE `test_1` (connection_role VARCHAR (255))")
MultiConnectionTestModel.connection.execute("INSERT INTO test_1 VALUES ('writing')")
ActiveRecord::Base.connected_to(role: :reading) do
MultiConnectionTestModel.connection.execute("CREATE TABLE `test_1` (connection_role VARCHAR (255))")
MultiConnectionTestModel.connection.execute("INSERT INTO test_1 VALUES ('reading')")
end
read_latch = Concurrent::CountDownLatch.new
write_latch = Concurrent::CountDownLatch.new
MultiConnectionTestModel.connection
thread = Thread.new do
MultiConnectionTestModel.connection
write_latch.wait
assert_equal "writing", MultiConnectionTestModel.connection.select_value("SELECT connection_role from test_1")
read_latch.count_down
end
ActiveRecord::Base.connected_to(role: :reading) do
write_latch.count_down
assert_equal "reading", MultiConnectionTestModel.connection.select_value("SELECT connection_role from test_1")
read_latch.wait
end
thread.join
ensure
tf_reading.close
tf_reading.unlink
tf_writing.close
tf_writing.unlink
end
unless in_memory_db?
def test_establish_connection_using_3_levels_config
previous_env, ENV["RAILS_ENV"] = ENV["RAILS_ENV"], "default_env"
config = {
"default_env" => {
"readonly" => { "adapter" => "sqlite3", "database" => "db/readonly.sqlite3" },
"primary" => { "adapter" => "sqlite3", "database" => "db/primary.sqlite3" }
}
}
@prev_configs, ActiveRecord::Base.configurations = ActiveRecord::Base.configurations, config
ActiveRecord::Base.connects_to(database: { writing: :primary, reading: :readonly })
assert_not_nil pool = ActiveRecord::Base.connection_handlers[:writing].retrieve_connection_pool("primary")
assert_equal "db/primary.sqlite3", pool.spec.config[:database]
assert_not_nil pool = ActiveRecord::Base.connection_handlers[:reading].retrieve_connection_pool("primary")
assert_equal "db/readonly.sqlite3", pool.spec.config[:database]
ensure
ActiveRecord::Base.configurations = @prev_configs
ActiveRecord::Base.establish_connection(:arunit)
ENV["RAILS_ENV"] = previous_env
end
def test_switching_connections_via_handler
previous_env, ENV["RAILS_ENV"] = ENV["RAILS_ENV"], "default_env"
config = {
"default_env" => {
"readonly" => { "adapter" => "sqlite3", "database" => "db/readonly.sqlite3" },
"primary" => { "adapter" => "sqlite3", "database" => "db/primary.sqlite3" }
}
}
@prev_configs, ActiveRecord::Base.configurations = ActiveRecord::Base.configurations, config
ActiveRecord::Base.connects_to(database: { writing: :primary, reading: :readonly })
ActiveRecord::Base.connected_to(role: :reading) do
@ro_handler = ActiveRecord::Base.connection_handler
assert_equal ActiveRecord::Base.connection_handler, ActiveRecord::Base.connection_handlers[:reading]
end
ActiveRecord::Base.connected_to(role: :writing) do
assert_equal ActiveRecord::Base.connection_handler, ActiveRecord::Base.connection_handlers[:writing]
assert_not_equal @ro_handler, ActiveRecord::Base.connection_handler
end
ensure
ActiveRecord::Base.configurations = @prev_configs
ActiveRecord::Base.establish_connection(:arunit)
ENV["RAILS_ENV"] = previous_env
end
def test_connects_to_with_single_configuration
config = {
"development" => { "adapter" => "sqlite3", "database" => "db/primary.sqlite3" },
}
@prev_configs, ActiveRecord::Base.configurations = ActiveRecord::Base.configurations, config
ActiveRecord::Base.connects_to database: { writing: :development }
assert_equal 1, ActiveRecord::Base.connection_handlers.size
assert_equal ActiveRecord::Base.connection_handler, ActiveRecord::Base.connection_handlers[:writing]
ensure
ActiveRecord::Base.configurations = @prev_configs
ActiveRecord::Base.establish_connection(:arunit)
end
def test_connects_to_using_top_level_key_in_two_level_config
config = {
"development" => { "adapter" => "sqlite3", "database" => "db/primary.sqlite3" },
"development_readonly" => { "adapter" => "sqlite3", "database" => "db/readonly.sqlite3" }
}
@prev_configs, ActiveRecord::Base.configurations = ActiveRecord::Base.configurations, config
ActiveRecord::Base.connects_to database: { writing: :development, reading: :development_readonly }
assert_not_nil pool = ActiveRecord::Base.connection_handlers[:reading].retrieve_connection_pool("primary")
assert_equal "db/readonly.sqlite3", pool.spec.config[:database]
ensure
ActiveRecord::Base.configurations = @prev_configs
ActiveRecord::Base.establish_connection(:arunit)
end
end
def test_connection_pools
assert_equal([@rw_pool], @handlers[:writing].connection_pools)
assert_equal([@ro_pool], @handlers[:reading].connection_pools)
end
def test_retrieve_connection
assert @rw_handler.retrieve_connection(@spec_name)
assert @ro_handler.retrieve_connection(@spec_name)
end
def test_active_connections?
assert_not_predicate @rw_handler, :active_connections?
assert_not_predicate @ro_handler, :active_connections?
assert @rw_handler.retrieve_connection(@spec_name)
assert @ro_handler.retrieve_connection(@spec_name)
assert_predicate @rw_handler, :active_connections?
assert_predicate @ro_handler, :active_connections?
@rw_handler.clear_active_connections!
assert_not_predicate @rw_handler, :active_connections?
@ro_handler.clear_active_connections!
assert_not_predicate @ro_handler, :active_connections?
end
def test_retrieve_connection_pool
assert_not_nil @rw_handler.retrieve_connection_pool(@spec_name)
assert_not_nil @ro_handler.retrieve_connection_pool(@spec_name)
end
def test_retrieve_connection_pool_with_invalid_id
assert_nil @rw_handler.retrieve_connection_pool("foo")
assert_nil @ro_handler.retrieve_connection_pool("foo")
end
end
end
end

0 comments on commit 31021a8

Please sign in to comment.