Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Part 2: Multi-db improvements, Refactor Active Record configurations #33637

Merged

Conversation

eileencodes
Copy link
Member

@eileencodes eileencodes commented Aug 16, 2018

While the three-tier config makes it easier to define databases for
multiple database applications, it quickly became clear to offer full
support for multiple databases we need to change the way the connections
hash was handled.

A three-tier config means that when Rails needed to choose a default
configuration (in the case a user doesn't ask for a specific
configuration) it wasn't clear to Rails which the default was. I
bandaid fixed this so the rake tasks could work but that fix
wasn't correct because it actually doubled up the configuration hashes.

Instead of attemping to manipulate the hashes @tenderlove and I decided
that it made more sense if we converted the hashes to objects so we can
easily ask those object questions. In a three tier config like this:

development:
  primary:
    database: "my_primary_db"
  animals:
    database; "my_animals_db"

We end up with an object like this:

  @configurations=[
    #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbded10
      @env_name="development",@spec_name="primary",
      @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>,
    #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbdea90
      @env_name="development",@spec_name="animals",
      @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>
]>

The configurations setter takes the database configuration set by your
application and turns them into an
ActiveRecord::DatabaseConfigurations object that has one getter -
@configurations which is an array of all the database objects.

The configurations getter returns this object by default since it acts
like a hash in most of the cases we need. For example if you need to
access the default development database we can simply request it as we
did before:

ActiveRecord::Base.configurations["development"]

This will return primary development database configuration hash:

{ "database" => "my_primary_db" }

Internally all of Active Record has been converted to use the new
objects. I've built this to be backwards compatible but allow for
accessing the hash if needed for a deprecation period. To get the
original hash instead of the object you can either add to_h on the
configurations call or pass legacy: true to `configurations.

ActiveRecord::Base.configurations.to_h
=> { "development => { "database" => "my_primary_db" } }

ActiveRecord::Base.configurations(legacy: true)
=> { "development => { "database" => "my_primary_db" } }

The new configurations object allows us to iterate over the Active
Record configurations without losing the known environment or
specification name for that configuration. You can also select all the
configs for an env or env and spec. With this we can always ask
any object what environment it belongs to:

db_configs = ActiveRecord::Base.configurations.configurations_for("development")
=> #<ActiveRecord::DatabaseConfigurations:0x00007fd1acbdf800
  @configurations=[
    #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbded10
      @env_name="development",@spec_name="primary",
      @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>,
    #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbdea90
      @env_name="development",@spec_name="animals",
      @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>
]>

db_config.env_name
=> "development"

db_config.spec_name
=> "primary"

db_config.config
=> { "adapter"=>"sqlite3", "database"=>"db/development.sqlite3" }

The configurations object is more flexible than the configurations hash
and will allow us to build on top of the connection management in order
to add support for primary/replica connections, sharding, and
constructing queries for associations that live in multiple databases.

cc/ @tenderlove @matthewd @rafaelfranca

PS This is easier to view in the split view.

Left to do:

  • I'm not sure if the docs are accurate, I might need to improve them but the majority of changes are internal implementation and not a whole ton should change for the average Rails app.
  • Let's see what tests fail 馃槄
  • Decide what to do about backwards compatibility: added a method missing for most commonly used hash accessors. In general this is for library authors as apps are less likely to mess with the connection hash, and if you do it's pretty simply to use the new object version.
  • changelog entry

I think there's more code that can be deleted, especially in ConnectionSpecification but I didn't want to go too far as I've been working on this since February.

@eileencodes eileencodes added this to the 6.0.0 milestone Aug 16, 2018
@eileencodes eileencodes force-pushed the ar-connection-management-refactoring branch 2 times, most recently from 336f6bf to aaec0d6 Compare August 16, 2018 20:46
@eileencodes eileencodes self-assigned this Aug 16, 2018
@@ -291,7 +292,6 @@ class Base
extend Aggregations::ClassMethods

include Core
include DatabaseConfigurations
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is no longer a module.

@@configurations.to_h
else
@@configurations
end
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I waffled back and forth on this something like 800 times. I opted for default being to use the new version because 99% of applications don't muck with the configurations hash. In addition I added some features (blank?, empty?, and []) to the returned object so the disruption should be small.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I'm happy to change this without a deprecation: it's a low-traffic API, and we're fundamentally rearranging enough that anyone relying on the legacy version is going to have a deeply limited view of the world, at best.

I think we should ditch the legacy: true option entirely, though: ActiveRecord::Base.configurations.to_h will work just fine, and is a better spelling for "be like the old one" because it works on the old one too -- the only reasonable reason to want the old hash style is because you're trying to support multiple versions.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some method missing and removed the legacy stuff.

@@ -127,36 +127,36 @@ class AnimalsBase < ActiveRecord::Base

test "db:create and db:drop works on all databases for env" do
require "#{app_path}/config/environment"
ActiveRecord::Base.configurations[Rails.env].each do |namespace, config|
db_create_and_drop namespace, config["database"]
ActiveRecord::Base.configurations.configs_for(Rails.env).each do |db_config|
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed these because this is what is in the database tasks code so we ensure all this keeps working.

config ||= DEFAULT_ENV.call.to_sym
spec_name = self == Base ? "primary" : name
self.connection_specification_name = spec_name
config_or_env ||= DEFAULT_ENV.call.to_sym
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall we found that these could be either a configuration hash or an environment. In order to reduce the confusing around the code in connection handling we change a bunch of the variable name so we could keep track.

@eileencodes eileencodes force-pushed the ar-connection-management-refactoring branch from aaec0d6 to fead0d8 Compare August 16, 2018 22:54
@eileencodes eileencodes changed the title Refactors Active Record connection management Refactor Active Record connection management Aug 16, 2018
@eileencodes eileencodes force-pushed the ar-connection-management-refactoring branch 3 times, most recently from 2fdc18e to cb5a7c1 Compare August 30, 2018 13:15
While the three-tier config makes it easier to define databases for
multiple database applications, it quickly became clear to offer full
support for multiple databases we need to change the way the connections
hash was handled.

A three-tier config means that when Rails needed to choose a default
configuration (in the case a user doesn't ask for a specific
configuration) it wasn't clear to Rails which the default was. I
[bandaid fixed this so the rake tasks could work](rails#32271) but that fix
wasn't correct because it actually doubled up the configuration hashes.

Instead of attemping to manipulate the hashes @tenderlove and I decided
that it made more sense if we converted the hashes to objects so we can
easily ask those object questions. In a three tier config like this:

```
development:
  primary:
    database: "my_primary_db"
  animals:
    database; "my_animals_db"
```

We end up with an object like this:

```
  @Configurations=[
    #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbded10
      @env_name="development",@spec_name="primary",
      @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>,
    #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbdea90
      @env_name="development",@spec_name="animals",
      @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>
]>
```

The configurations setter takes the database configuration set by your
application and turns them into an
`ActiveRecord::DatabaseConfigurations` object that has one getter -
`@configurations` which is an array of all the database objects.

The configurations getter returns this object by default since it acts
like a hash in most of the cases we need. For example if you need to
access the default `development` database we can simply request it as we
did before:

```
ActiveRecord::Base.configurations["development"]
```

This will return primary development database configuration hash:

```
{ "database" => "my_primary_db" }
```

Internally all of Active Record has been converted to use the new
objects. I've built this to be backwards compatible but allow for
accessing the hash if needed for a deprecation period. To get the
original hash instead of the object you can either add `to_h` on the
configurations call or pass `legacy: true` to `configurations.

```
ActiveRecord::Base.configurations.to_h
=> { "development => { "database" => "my_primary_db" } }

ActiveRecord::Base.configurations(legacy: true)
=> { "development => { "database" => "my_primary_db" } }
```

The new configurations object allows us to iterate over the Active
Record configurations without losing the known environment or
specification name for that configuration. You can also select all the
configs for an env or env and spec. With this we can always ask
any object what environment it belongs to:

```
db_configs = ActiveRecord::Base.configurations.configurations_for("development")
=> #<ActiveRecord::DatabaseConfigurations:0x00007fd1acbdf800
  @Configurations=[
    #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbded10
      @env_name="development",@spec_name="primary",
      @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>,
    #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbdea90
      @env_name="development",@spec_name="animals",
      @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>
]>

db_config.env_name
=> "development"

db_config.spec_name
=> "primary"

db_config.config
=> { "adapter"=>"sqlite3", "database"=>"db/development.sqlite3" }
```

The configurations object is more flexible than the configurations hash
and will allow us to build on top of the connection management in order
to add support for primary/replica connections, sharding, and
constructing queries for associations that live in multiple databases.
@eileencodes eileencodes force-pushed the ar-connection-management-refactoring branch from cb5a7c1 to fdf3f0b Compare August 30, 2018 14:07
@eileencodes eileencodes changed the title Refactor Active Record connection management Refactor Active Record configurations Aug 30, 2018
@eileencodes eileencodes merged commit 8f2caec into rails:master Aug 30, 2018
@eileencodes eileencodes deleted the ar-connection-management-refactoring branch August 30, 2018 17:07
@schneems
Copy link
Member

馃帀馃帀馃帀

@eileencodes eileencodes changed the title Refactor Active Record configurations Part 2: Multi-db, Refactor Active Record configurations Aug 31, 2018
@eileencodes eileencodes changed the title Part 2: Multi-db, Refactor Active Record configurations Part 2: Multi-db improvements, Refactor Active Record configurations Aug 31, 2018
bogdanvlviv added a commit to bogdanvlviv/rails that referenced this pull request Sep 4, 2018
After rails#33637 some tests in `activerecord/test/cases/tasks/database_tasks_test.rb`
don't assert anything.
We used to stub `ActiveRecord::Base::configurations` method in those
tests like `ActiveRecord::Base.stub(:configurations, @Configurations) {}`.
Since rails#33637 `ActiveRecord::Base::configurations` is a `ActiveRecord::DatabaseConfigurations`
object(not a Hash object) we can't do so anymore.
`ActiveRecord::DatabaseConfigurations` object builds during `ActiveRecord::Base::configurations=`.
We can replace `ActiveRecord::Base.stub(:configurations, @Configurations) {}` to
```
begin
  old_configurations = ActiveRecord::Base.configurations
  ActiveRecord::Base.configurations = @Configurations
  # ...
ensure
 ActiveRecord::Base.configurations = old_configurations
end
```

Also I fixed tests in `activerecord/test/cases/tasks/legacy_database_tasks_test.rb`
But currently It looks like duplication of
`activerecord/test/cases/tasks/database_tasks_test.rb`.
We should improve those tests or remove them.

I've tried (in `activerecord/test/cases/tasks/legacy_database_tasks_test.rb` file):
```
def with_stubbed_configurations
  old_configurations = ActiveRecord::Base.configurations.to_h
  ActiveRecord::Base.configurations = @Configurations

  ActiveRecord::Base.stub(:configurations, ActiveRecord::Base.configurations.to_h) do
    yield
  end
ensure
  ActiveRecord::Base.configurations = old_configurations
end
```
but it causes erros in tests cases.

After discussion we decided to remove
`activerecord/test/cases/tasks/legacy_database_tasks_test.rb`

Related to rails#33637
@ioquatix
Copy link
Contributor

I know I'm late to the party but https://github.com/ioquatix/activerecord-configurations is my solution to this problem and this is probably a breaking change for my use css which is a bit of a pain. I'll need to add Rails 6 to the test matrix.

@ioquatix
Copy link
Contributor

Just for xref: #32135

@ioquatix
Copy link
Contributor

ioquatix commented Mar 19, 2019

So, as expected, it fails:

ActiveRecord::Configurations
  should establish connection (FAILED - 1)

Failures:

  1) ActiveRecord::Configurations should establish connection
     Failure/Error: spec = resolver.resolve(config).symbolize_keys
     
     NoMethodError:
       undefined method `find_db_config' for #<Hash:0x00007fbb7cbf65a0>
     # /Users/samuel/.rvm/gems/ruby-2.6.1/gems/activerecord-6.0.0.beta3/lib/active_record/connection_adapters/connection_specification.rb:246:in `resolve_symbol_connection'
     # /Users/samuel/.rvm/gems/ruby-2.6.1/gems/activerecord-6.0.0.beta3/lib/active_record/connection_adapters/connection_specification.rb:219:in `resolve_connection'
     # /Users/samuel/.rvm/gems/ruby-2.6.1/gems/activerecord-6.0.0.beta3/lib/active_record/connection_adapters/connection_specification.rb:140:in `resolve'
     # ./lib/active_record/configurations.rb:37:in `establish_connection'
     # ./lib/active_record/configurations.rb:133:in `setup_connection'
     # ./spec/active_record/establish_connection_spec.rb:49:in `block (2 levels) in <top (required)>'

If the expectation was that connections should be a hash, then this PR is breaking that interface. However, maybe I was the only person depending on that :)

NickLaMuro added a commit to NickLaMuro/manageiq that referenced this pull request Nov 11, 2020
Rails 6 changed database configuration to support multi-db environments:

rails/rails#33637

So how one accesses the `@config` hash has changed.

The original error suggested the following:

    NotImplementedError: `ActiveRecord::Base.configurations` in Rails 6
    now returns an object instead of a hash. The `fetch_path` method is
    not supported. Please use `configs_for` or consult the documentation
    for supported methods.

However, there is no `[]` or `host` method, so it would require calling
`configs_for(env_name: ENV["RAILS_ENV"]).config['host']`.

This solution seemed like the more terse solution.
NickLaMuro added a commit to NickLaMuro/manageiq that referenced this pull request Nov 16, 2020
Rails 6 changed database configuration to support multi-db environments:

rails/rails#33637

So how one accesses the `@config` hash has changed.

The original error suggested the following:

    NotImplementedError: `ActiveRecord::Base.configurations` in Rails 6
    now returns an object instead of a hash. The `fetch_path` method is
    not supported. Please use `configs_for` or consult the documentation
    for supported methods.

However, there is no `[]` or `host` method, so it would require calling
`configs_for(env_name: ENV["RAILS_ENV"]).config['host']`.

This solution seemed like the more terse solution.
NickLaMuro added a commit to NickLaMuro/manageiq that referenced this pull request Nov 17, 2020
Rails 6 changed database configuration to support multi-db environments:

rails/rails#33637

So how one accesses the `@config` hash has changed.

The original error suggested the following:

    NotImplementedError: `ActiveRecord::Base.configurations` in Rails 6
    now returns an object instead of a hash. The `fetch_path` method is
    not supported. Please use `configs_for` or consult the documentation
    for supported methods.

However, there is no `[]` or `host` method, so it would require calling
`configs_for(env_name: ENV["RAILS_ENV"]).config['host']`.

This solution seemed like the more terse solution.
NickLaMuro added a commit to NickLaMuro/manageiq that referenced this pull request Nov 23, 2020
Rails 6 changed database configuration to support multi-db environments:

rails/rails#33637

So how one accesses the `@config` hash has changed.

The original error suggested the following:

    NotImplementedError: `ActiveRecord::Base.configurations` in Rails 6
    now returns an object instead of a hash. The `fetch_path` method is
    not supported. Please use `configs_for` or consult the documentation
    for supported methods.

However, there is no `[]` or `host` method, so it would require calling
`configs_for(env_name: ENV["RAILS_ENV"]).config['host']`.

This solution seemed like the more terse solution.
NickLaMuro added a commit to NickLaMuro/manageiq that referenced this pull request Nov 24, 2020
Rails 6 changed database configuration to support multi-db environments:

rails/rails#33637

So how one accesses the `@config` hash has changed.

The original error suggested the following:

    NotImplementedError: `ActiveRecord::Base.configurations` in Rails 6
    now returns an object instead of a hash. The `fetch_path` method is
    not supported. Please use `configs_for` or consult the documentation
    for supported methods.

However, there is no `[]` or `host` method, so it would require calling
`configs_for(env_name: ENV["RAILS_ENV"]).config['host']`.

This solution seemed like the more terse solution.
NickLaMuro added a commit to NickLaMuro/manageiq that referenced this pull request Nov 30, 2020
Rails 6 changed database configuration to support multi-db environments:

rails/rails#33637

So how one accesses the `@config` hash has changed.

The original error suggested the following:

    NotImplementedError: `ActiveRecord::Base.configurations` in Rails 6
    now returns an object instead of a hash. The `fetch_path` method is
    not supported. Please use `configs_for` or consult the documentation
    for supported methods.

However, there is no `[]` or `host` method, so it would require calling
`configs_for(env_name: ENV["RAILS_ENV"]).config['host']`.

This solution seemed like the more terse solution.
NickLaMuro added a commit to NickLaMuro/manageiq that referenced this pull request Dec 1, 2020
Rails 6 changed database configuration to support multi-db environments:

rails/rails#33637

So how one accesses the `@config` hash has changed.

The original error suggested the following:

    NotImplementedError: `ActiveRecord::Base.configurations` in Rails 6
    now returns an object instead of a hash. The `fetch_path` method is
    not supported. Please use `configs_for` or consult the documentation
    for supported methods.

However, there is no `[]` or `host` method, so it would require calling
`configs_for(env_name: ENV["RAILS_ENV"]).config['host']`.

This solution seemed like the more terse solution.
NickLaMuro added a commit to NickLaMuro/manageiq that referenced this pull request Dec 2, 2020
Rails 6 changed database configuration to support multi-db environments:

rails/rails#33637

So how one accesses the `@config` hash has changed.

The original error suggested the following:

    NotImplementedError: `ActiveRecord::Base.configurations` in Rails 6
    now returns an object instead of a hash. The `fetch_path` method is
    not supported. Please use `configs_for` or consult the documentation
    for supported methods.

However, there is no `[]` or `host` method, so it would require calling
`configs_for(env_name: ENV["RAILS_ENV"]).config['host']`.

This solution seemed like the more terse solution.
NickLaMuro added a commit to NickLaMuro/manageiq that referenced this pull request Dec 3, 2020
Rails 6 changed database configuration to support multi-db environments:

rails/rails#33637

So how one accesses the `@config` hash has changed.

The original error suggested the following:

    NotImplementedError: `ActiveRecord::Base.configurations` in Rails 6
    now returns an object instead of a hash. The `fetch_path` method is
    not supported. Please use `configs_for` or consult the documentation
    for supported methods.

However, there is no `[]` or `host` method, so it would require calling
`configs_for(env_name: ENV["RAILS_ENV"]).config['host']`.

This solution seemed like the more terse solution.
NickLaMuro added a commit to NickLaMuro/manageiq that referenced this pull request Dec 7, 2020
Rails 6 changed database configuration to support multi-db environments:

rails/rails#33637

So how one accesses the `@config` hash has changed.

The original error suggested the following:

    NotImplementedError: `ActiveRecord::Base.configurations` in Rails 6
    now returns an object instead of a hash. The `fetch_path` method is
    not supported. Please use `configs_for` or consult the documentation
    for supported methods.

However, there is no `[]` or `host` method, so it would require calling
`configs_for(env_name: ENV["RAILS_ENV"]).config['host']`.

This solution seemed like the more terse solution.
NickLaMuro added a commit to NickLaMuro/manageiq that referenced this pull request Dec 10, 2020
Rails 6 changed database configuration to support multi-db environments:

rails/rails#33637

So how one accesses the `@config` hash has changed.

The original error suggested the following:

    NotImplementedError: `ActiveRecord::Base.configurations` in Rails 6
    now returns an object instead of a hash. The `fetch_path` method is
    not supported. Please use `configs_for` or consult the documentation
    for supported methods.

However, there is no `[]` or `host` method, so it would require calling
`configs_for(env_name: ENV["RAILS_ENV"]).config['host']`.

This solution seemed like the more terse solution.
NickLaMuro added a commit to NickLaMuro/manageiq that referenced this pull request Jan 5, 2021
Rails 6 changed database configuration to support multi-db environments:

rails/rails#33637

So how one accesses the `@config` hash has changed.

The original error suggested the following:

    NotImplementedError: `ActiveRecord::Base.configurations` in Rails 6
    now returns an object instead of a hash. The `fetch_path` method is
    not supported. Please use `configs_for` or consult the documentation
    for supported methods.

However, there is no `[]` or `host` method, so it would require calling
`configs_for(env_name: ENV["RAILS_ENV"]).config['host']`.

This solution seemed like the more terse solution.
kbrock pushed a commit to kbrock/manageiq that referenced this pull request Jan 27, 2021
Rails 6 changed database configuration to support multi-db environments:

rails/rails#33637

So how one accesses the `@config` hash has changed.

The original error suggested the following:

    NotImplementedError: `ActiveRecord::Base.configurations` in Rails 6
    now returns an object instead of a hash. The `fetch_path` method is
    not supported. Please use `configs_for` or consult the documentation
    for supported methods.

However, there is no `[]` or `host` method, so it would require calling
`configs_for(env_name: ENV["RAILS_ENV"]).config['host']`.

This solution seemed like the more terse solution.
HeyNonster added a commit to zendesk/active_record_shards that referenced this pull request Sep 22, 2021
ActiveRecord::Base.configurations used to return a hash before Rails 6 but now
returns an object of configurations. We can return a hash like we used to expect
by calling .to_h on the object. For Rails < 6 .to_h will return the hash unchanged.

Relevant Rails PR: rails/rails#33637
mjankowski added a commit to mjankowski/mastodon that referenced this pull request Mar 15, 2023
The way that db options are access changed:

rails/rails#33637
mjankowski added a commit to mjankowski/mastodon that referenced this pull request Mar 15, 2023
The way that db options are access changed:

rails/rails#33637
mjankowski added a commit to mjankowski/mastodon that referenced this pull request Mar 16, 2023
The way that db options are access changed:

rails/rails#33637
mjankowski added a commit to mjankowski/mastodon that referenced this pull request Mar 16, 2023
The way that db options are access changed:

rails/rails#33637
mjankowski added a commit to mjankowski/mastodon that referenced this pull request Mar 17, 2023
The way that db options are access changed:

rails/rails#33637
mjankowski added a commit to mjankowski/mastodon that referenced this pull request Mar 17, 2023
The way that db options are access changed:

rails/rails#33637
mjankowski added a commit to mjankowski/mastodon that referenced this pull request Mar 20, 2023
The way that db options are access changed:

rails/rails#33637
mjankowski added a commit to mjankowski/mastodon that referenced this pull request Mar 21, 2023
The way that db options are access changed:

rails/rails#33637
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants