New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Part 2: Multi-db improvements, Refactor Active Record configurations #33637

Merged
merged 1 commit into from Aug 30, 2018

Conversation

Projects
None yet
5 participants
@eileencodes
Member

eileencodes commented Aug 16, 2018

While the three-tier config makes it easier to define databases for
multiple database applications, it quickly became clear to offer full
support for multiple databases we need to change the way the connections
hash was handled.

A three-tier config means that when Rails needed to choose a default
configuration (in the case a user doesn't ask for a specific
configuration) it wasn't clear to Rails which the default was. I
bandaid fixed this so the rake tasks could work but that fix
wasn't correct because it actually doubled up the configuration hashes.

Instead of attemping to manipulate the hashes @tenderlove and I decided
that it made more sense if we converted the hashes to objects so we can
easily ask those object questions. In a three tier config like this:

development:
  primary:
    database: "my_primary_db"
  animals:
    database; "my_animals_db"

We end up with an object like this:

  @configurations=[
    #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbded10
      @env_name="development",@spec_name="primary",
      @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>,
    #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbdea90
      @env_name="development",@spec_name="animals",
      @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>
]>

The configurations setter takes the database configuration set by your
application and turns them into an
ActiveRecord::DatabaseConfigurations object that has one getter -
@configurations which is an array of all the database objects.

The configurations getter returns this object by default since it acts
like a hash in most of the cases we need. For example if you need to
access the default development database we can simply request it as we
did before:

ActiveRecord::Base.configurations["development"]

This will return primary development database configuration hash:

{ "database" => "my_primary_db" }

Internally all of Active Record has been converted to use the new
objects. I've built this to be backwards compatible but allow for
accessing the hash if needed for a deprecation period. To get the
original hash instead of the object you can either add to_h on the
configurations call or pass legacy: true to `configurations.

ActiveRecord::Base.configurations.to_h
=> { "development => { "database" => "my_primary_db" } }

ActiveRecord::Base.configurations(legacy: true)
=> { "development => { "database" => "my_primary_db" } }

The new configurations object allows us to iterate over the Active
Record configurations without losing the known environment or
specification name for that configuration. You can also select all the
configs for an env or env and spec. With this we can always ask
any object what environment it belongs to:

db_configs = ActiveRecord::Base.configurations.configurations_for("development")
=> #<ActiveRecord::DatabaseConfigurations:0x00007fd1acbdf800
  @configurations=[
    #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbded10
      @env_name="development",@spec_name="primary",
      @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>,
    #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbdea90
      @env_name="development",@spec_name="animals",
      @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>
]>

db_config.env_name
=> "development"

db_config.spec_name
=> "primary"

db_config.config
=> { "adapter"=>"sqlite3", "database"=>"db/development.sqlite3" }

The configurations object is more flexible than the configurations hash
and will allow us to build on top of the connection management in order
to add support for primary/replica connections, sharding, and
constructing queries for associations that live in multiple databases.

cc/ @tenderlove @matthewd @rafaelfranca

PS This is easier to view in the split view.

Left to do:

  • I'm not sure if the docs are accurate, I might need to improve them but the majority of changes are internal implementation and not a whole ton should change for the average Rails app.
  • Let's see what tests fail 馃槄
  • Decide what to do about backwards compatibility: added a method missing for most commonly used hash accessors. In general this is for library authors as apps are less likely to mess with the connection hash, and if you do it's pretty simply to use the new object version.
  • changelog entry

I think there's more code that can be deleted, especially in ConnectionSpecification but I didn't want to go too far as I've been working on this since February.

@eileencodes eileencodes added this to the 6.0.0 milestone Aug 16, 2018

@eileencodes eileencodes self-assigned this Aug 16, 2018

@@ -291,7 +292,6 @@ class Base
extend Aggregations::ClassMethods
include Core
include DatabaseConfigurations

This comment has been minimized.

@eileencodes

eileencodes Aug 16, 2018

Member

This is no longer a module.

@eileencodes

eileencodes Aug 16, 2018

Member

This is no longer a module.

Show outdated Hide outdated activerecord/lib/active_record/core.rb Outdated
@@ -127,36 +127,36 @@ class AnimalsBase < ActiveRecord::Base
test "db:create and db:drop works on all databases for env" do
require "#{app_path}/config/environment"
ActiveRecord::Base.configurations[Rails.env].each do |namespace, config|
db_create_and_drop namespace, config["database"]
ActiveRecord::Base.configurations.configs_for(Rails.env).each do |db_config|

This comment has been minimized.

@eileencodes

eileencodes Aug 16, 2018

Member

I changed these because this is what is in the database tasks code so we ensure all this keeps working.

@eileencodes

eileencodes Aug 16, 2018

Member

I changed these because this is what is in the database tasks code so we ensure all this keeps working.

config ||= DEFAULT_ENV.call.to_sym
spec_name = self == Base ? "primary" : name
self.connection_specification_name = spec_name
config_or_env ||= DEFAULT_ENV.call.to_sym

This comment has been minimized.

@eileencodes

eileencodes Aug 16, 2018

Member

Overall we found that these could be either a configuration hash or an environment. In order to reduce the confusing around the code in connection handling we change a bunch of the variable name so we could keep track.

@eileencodes

eileencodes Aug 16, 2018

Member

Overall we found that these could be either a configuration hash or an environment. In order to reduce the confusing around the code in connection handling we change a bunch of the variable name so we could keep track.

@eileencodes eileencodes changed the title from Refactors Active Record connection management to Refactor Active Record connection management Aug 16, 2018

Show resolved Hide resolved activerecord/test/cases/tasks/legacy_database_tasks_test.rb Outdated
Show resolved Hide resolved activerecord/test/cases/tasks/legacy_database_tasks_test.rb Outdated
Show resolved Hide resolved ...record/lib/active_record/connection_adapters/connection_specification.rb Outdated
Show resolved Hide resolved activerecord/lib/active_record/database_configurations/database_config.rb Outdated
Refactors Active Record connection management
While the three-tier config makes it easier to define databases for
multiple database applications, it quickly became clear to offer full
support for multiple databases we need to change the way the connections
hash was handled.

A three-tier config means that when Rails needed to choose a default
configuration (in the case a user doesn't ask for a specific
configuration) it wasn't clear to Rails which the default was. I
[bandaid fixed this so the rake tasks could work](#32271) but that fix
wasn't correct because it actually doubled up the configuration hashes.

Instead of attemping to manipulate the hashes @tenderlove and I decided
that it made more sense if we converted the hashes to objects so we can
easily ask those object questions. In a three tier config like this:

```
development:
  primary:
    database: "my_primary_db"
  animals:
    database; "my_animals_db"
```

We end up with an object like this:

```
  @Configurations=[
    #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbded10
      @env_name="development",@spec_name="primary",
      @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>,
    #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbdea90
      @env_name="development",@spec_name="animals",
      @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>
]>
```

The configurations setter takes the database configuration set by your
application and turns them into an
`ActiveRecord::DatabaseConfigurations` object that has one getter -
`@Configurations` which is an array of all the database objects.

The configurations getter returns this object by default since it acts
like a hash in most of the cases we need. For example if you need to
access the default `development` database we can simply request it as we
did before:

```
ActiveRecord::Base.configurations["development"]
```

This will return primary development database configuration hash:

```
{ "database" => "my_primary_db" }
```

Internally all of Active Record has been converted to use the new
objects. I've built this to be backwards compatible but allow for
accessing the hash if needed for a deprecation period. To get the
original hash instead of the object you can either add `to_h` on the
configurations call or pass `legacy: true` to `configurations.

```
ActiveRecord::Base.configurations.to_h
=> { "development => { "database" => "my_primary_db" } }

ActiveRecord::Base.configurations(legacy: true)
=> { "development => { "database" => "my_primary_db" } }
```

The new configurations object allows us to iterate over the Active
Record configurations without losing the known environment or
specification name for that configuration. You can also select all the
configs for an env or env and spec. With this we can always ask
any object what environment it belongs to:

```
db_configs = ActiveRecord::Base.configurations.configurations_for("development")
=> #<ActiveRecord::DatabaseConfigurations:0x00007fd1acbdf800
  @Configurations=[
    #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbded10
      @env_name="development",@spec_name="primary",
      @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>,
    #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbdea90
      @env_name="development",@spec_name="animals",
      @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>
]>

db_config.env_name
=> "development"

db_config.spec_name
=> "primary"

db_config.config
=> { "adapter"=>"sqlite3", "database"=>"db/development.sqlite3" }
```

The configurations object is more flexible than the configurations hash
and will allow us to build on top of the connection management in order
to add support for primary/replica connections, sharding, and
constructing queries for associations that live in multiple databases.

@eileencodes eileencodes changed the title from Refactor Active Record connection management to Refactor Active Record configurations Aug 30, 2018

@eileencodes eileencodes merged commit 8f2caec into rails:master Aug 30, 2018

1 of 2 checks passed

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
codeclimate All good!
Details

@eileencodes eileencodes deleted the eileencodes:ar-connection-management-refactoring branch Aug 30, 2018

@schneems

This comment has been minimized.

Show comment
Hide comment
@schneems

schneems Aug 30, 2018

Member

馃帀馃帀馃帀

Member

schneems commented Aug 30, 2018

馃帀馃帀馃帀

@eileencodes eileencodes changed the title from Refactor Active Record configurations to Part 2: Multi-db, Refactor Active Record configurations Aug 31, 2018

@eileencodes eileencodes changed the title from Part 2: Multi-db, Refactor Active Record configurations to Part 2: Multi-db improvements, Refactor Active Record configurations Aug 31, 2018

bogdanvlviv added a commit to bogdanvlviv/rails that referenced this pull request Sep 4, 2018

Fix tests in `activerecord/test/cases/tasks/database_tasks_test.rb`
After rails#33637 some tests in `activerecord/test/cases/tasks/database_tasks_test.rb`
don't assert anything.
We used to stub `ActiveRecord::Base::configurations` method in those
tests like `ActiveRecord::Base.stub(:configurations, @Configurations) {}`.
Since rails#33637 `ActiveRecord::Base::configurations` is a `ActiveRecord::DatabaseConfigurations`
object(not a Hash object) we can't do so anymore.
`ActiveRecord::DatabaseConfigurations` object builds during `ActiveRecord::Base::configurations=`.
We can replace `ActiveRecord::Base.stub(:configurations, @Configurations) {}` to
```
begin
  old_configurations = ActiveRecord::Base.configurations
  ActiveRecord::Base.configurations = @Configurations
  # ...
ensure
 ActiveRecord::Base.configurations = old_configurations
end
```

Also I fixed tests in `activerecord/test/cases/tasks/legacy_database_tasks_test.rb`
But currently It looks like duplication of
`activerecord/test/cases/tasks/database_tasks_test.rb`.
We should improve those tests or remove them.

I've tried (in `activerecord/test/cases/tasks/legacy_database_tasks_test.rb` file):
```
def with_stubbed_configurations
  old_configurations = ActiveRecord::Base.configurations.to_h
  ActiveRecord::Base.configurations = @Configurations

  ActiveRecord::Base.stub(:configurations, ActiveRecord::Base.configurations.to_h) do
    yield
  end
ensure
  ActiveRecord::Base.configurations = old_configurations
end
```
but it causes erros in tests cases.

After discussion we decided to remove
`activerecord/test/cases/tasks/legacy_database_tasks_test.rb`

Related to rails#33637

eileencodes added a commit that referenced this pull request Sep 9, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment