Part 1 Easy Multi db in Rails: Add basic rake tasks for multi db setup

[eileencodes]

For multi db applications you always had to create your own rake tasks which made setting up multi db a major PITA. This PR is Part 1 of a many that adds the initial underpinning for supporting multiple databases through the rake db commands. I'm doing this in small PR's so that reviewing is easier.

This app can be used to test out the features here. Just clone and use the commands below to play with testing re rake tasks for create, migrate, drop, and dump. https://github.com/eileencodes/multiple_databases_demo

Examples below are assuming a three-tier database.yml like this:

development:
  primary:
    <<: *default
    database: multiple_databases_development
  animals:
    <<: *default
    database: multiple_databases_development_animals
    migrations_paths: "db/animals_migrate"

test:
  primary:
    <<: *default
    database: multiple_databases_test
  animals:
    <<: *default
    database: multiple_databases_test_animals
    migrations_paths: "db/animals_migrate"

• Creates internal DatabaseConfig objects so it's easier to pass around the configs. We can then ask a DatabaseConfig for it's env or spec or config directly. This will come in handy for a larger refactoring I'm working on.
• Ensures when running bin/rails db:create, bin/rails db:migrate, bin/rails db:drop, and bin/rails db:schema|structure:dump that tasks are run for all relevant envs and all databases in that env so given the above config bin/rails db:create will create the dev and test dbs for both primary and animals configs.
• Adds new Rails db tasks that can perform create/migrate/drop on a specific database in an environment
  • bin/rails db:create:primary or bin/rails db:create:animals
  • bin/rails db:drop:primary or bin/rails db:drop:animals
  • bin/rails db:migrate:primary or bin/rails db:migrate:animals

Future parts of this work will:

• Add more rake task support for multi db as this only does the first 4 major ones
• Add documentation for multi-db three-tier database.yml applications
• Add documentation for the rake tasks
• Refactoring of the connection management to fix the assumption that the config lives directly under development (see #32271 for more info there on what I'm talking about).
• Eventually porting these to Rails commands
• Dealing with rw/ro connection setups

cc/ @matthewd @tenderlove @dhh

[huskylengcb]

Can mysql and PGSQL be used together?

[RouL]

Seeng the question from @huskylengcb, I didn't check the code, but that would be perfect. Being able to use multiple databases on the same server, multiple databases on different servers and even multiple databases with different backends.
But I would guess at least, thats how it is meant anyways. :)

[eileencodes]

We use different mysql databases across different servers at GitHub already, so it "works", it just is really hard to set up. We're getting there but it will be a long process before multi db is easy in Rails.

[schneems]

Great work! Are there any preliminary docs around using multi-db with 6.0?

[eileencodes]

Thanks @schneems 😄

I haven't started docs because I think a lot more is going to change. This revealed some issues with connection management in Rails that I'm working on refactoring. Once I get more of that work done I can start the docs. I didn't want to start them before the code was written because I think it has a lot of potential to change in the coming months.

[Edouard-chin]

Thanks a lot for all the work you put so far in managing multiple db setup ♥️ .

A small heads up in case this wasn't thought of; loading the database.yml without evaluating erb first will break in case the database configuration contains erb loop or conditions

test:
  primary:
    <<: *default
    database: multiple_databases_test
  animals:
    <<: *default
    <% if something %> # This will break as it's invalid YML
    database: multiple_databases_test_animals
    migrations_paths: "db/animals_migrate"
    <% end %>

production:
  <% all_shards.each do |shard| %> # This will break as it's invalid YAML
  <%= shard.name %>:
    username: ...
  <% end %>

[eileencodes]

sigh I really think we need to limit what's possible in a database.yml. There are far too many edge cases. Do folks do this in their apps?

[Edouard-chin]

I really think we need to limit what's possible in a database.yml.

That would make our lives easier indeed, but not sure how we can do that since the config accepts erb tags, it should be able to handle any form of erb.

Do folks do this in their apps?

Not entirely sure how broad this is used but I have seen it few times and couple blog posts mentioning it to DRY the configuration. We also do that at Shopify in our config

[eileencodes]

Oh wait a minute. We don't evaluate it on the first round in for_each but we evaluate it once the task is created. The only thing we use the un-eval'd yaml for is to create the namespaces tasks like :animals, but then after that we load the eval'd yaml. So I think this is fine.

[eileencodes]

Er I see the prod db has different shards and needs the shard name for each task name. Does it actually break and throw an error or just not create the tasks?

[Edouard-chin]

It throws an error when trying to run the task (the yaml can't be parsed)

Psych::SyntaxError: (/app/config/database.yml): could not find expected ':' while scanning a simple key at line 387 column 3

[schneems]

sigh I really think we need to limit what's possible in a database.yml

If we're re-thinking the database experience, maybe it makes sense to do a non-yaml based config? Maybe config objects help eliminate some of these ambiguities?

[tenderlove]

I think we could eval the YAML before creating the rake tasks. The problem is that some people add conditionals based on the RAILS_ENV in the YAML file. Since we're trying to generate rake tasks independent of the RAILS_ENV, we can't set the RAILS_ENV, so their conditionals could be wrong.

IMO we should just run the YAML through ERB, then through the YAML parser, and if any exceptions happen along the way, we emit a warning and don't create the tasks. Does that sound reasonable?

[tenderlove]

It occurs to me that we could perform the eval but warn or raise an exception if people try to access ENV in the YAML:

require 'erb'

ENV["RAILS_ENV"] = "omg"

template = ERB.new(DATA.read)

require 'delegate'

class Foo
  class EnvDelegate < SimpleDelegator
    def [] key
      if key == "RAILS_ENV"
        warn "do not access RAILS_ENV"
        super
      else
        super
      end
    end
  end

  ENV = EnvDelegate.new(::ENV)

  def eval_template template
    template.result binding
  end
end

Foo.new.eval_template(template)

__END__
development:
  <% if ENV["RAILS_ENV"] == "omg" %>
  - wow
  <% else %>
  - neat
  <% end %>

Output:

[aaron@TC git]$ ruby testing.rb
do not access RAILS_ENV

[Edouard-chin]

Ah yes ok I understand the problem about the RAILS_ENV, thanks for the snippet.
The problem would happen for any condition tough, not only when checking the RAILS_ENV, right?

Are we specifically checking the RAILS_ENV because it might be one of the most common use case?

development:
  <% if something %>
  - wow
  <% else %>
  - neat
  <% end %>

[tenderlove]

@Edouard-chin ya. As long as the conditional doesn't depend on RAILS_ENV, then it should be fine to eval the ERB. I think we only need to protect against code that changes depending on the RAILS_ENV. In your example, as long as the something method call does not depend on the RAILS_ENV, then it's output should not change regardless of when or where we eval the ERB.

We don't know what the RAILS_ENV should be when defining the Rake tasks. However, as long as the ERB code does the same thing regardless of RAILS_ENV, then it should be OK to execute.