Rails 6 multiple database tasks throws exception when environment config is used

[morgoth]

In Rails 6 with multiple database support, rake tasks are now build dynamically based on db config:

rails/activerecord/lib/active_record/railties/databases.rake

Lines 26 to 33 in 9e34df0

	ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name|
	desc "Create #{spec_name} database for current environment"
	task spec_name => :load_config do
	db_config = ActiveRecord::Base.configurations.configs_for(env_name: Rails.env, spec_name: spec_name)
	ActiveRecord::Tasks::DatabaseTasks.create(db_config.config)
	end
	end
	end

This causes problems when evaluating config/database.yml that uses some configuration set on app level:

# config/environments/development.rb
Rails.application.configure do
  config.some_config = "I'm set"
end

default: &default
  some_config: <%= Rails.configuration.some_config %>

as in time of building rake task, app environment is not loaded yet, thus some_config method is not defined.

@eileencodes I'm wondering if those task names could be built without loading env, but I guess to get them, database.yml has to be evaluated. Other option would be to load the env in time of building them, but this looks like a too big overhead on rake -T in example.

[eileencodes]

Can you make a basic app on Rails 6 that reproduces this and I will take a look.

Unfortunately we cannot avoid loading ENV at the time we do - I tried for weeks to work around that, it's just not possible.

[morgoth]

Here you go https://github.com/morgoth/ar-bug#readme :-)

[eileencodes]

Is this the error you're seeing or a different one?

rake aborted!
NoMethodError: Cannot load database configuration:
undefined method `max_threads' for #<Rails::Application::Configuration:0x00007fc39d0b96c8>
(erb):9:in `<main>'
/Users/eileencodes/open_source/rails_apps/ar-bug/Rakefile:6:in `<top (required)>'

[morgoth]

yes, it's that error

[eileencodes]

So there's a deeper problem we identified with setting any environment specific config that we really should have never allowed this behavior to work in the first place.

While it used to work without blowing up it presents some problems. If you set max_threads in development to 10 and in test to 5 and then load the configurations the configuration settings will change based on the environment you load that configuration in.

So if you load the configuration in test it will set max_threads in dev and test to 5 because Rails can't read a different environment when you're loading another environment.

I'm not sure we can support this going forward because it was a broken system to begin with. I will try to figure out how to not raise an unhelpful error.

[eileencodes]

Actually, I'll first experiment a bit more with not blowing up but having been down this road before I'm not sure it's possible or recommended to support this going forward.

[morgoth]

So if you load the configuration in test it will set max_threads in dev and test to 5 because Rails can't read a different environment when you're loading another environment.

Yes, I know about it. it works the same way as config_for

[eileencodes]

We have a chicken and egg problem here and I don't know how to solve this. Possible solutions are:

1. Just load the YAML to get the namespaces without the ERB. This won't work though if you have multi-line erb, yaml can't parse it and it will blow up with a Psych error.
2. Load the environment when we call for_each. Sure this is possible and it works but it means we're loading the env for bin/rake -T which may be a performance hit we're not willing to take.
3. Have the app pre-provide the namespaces. If we had a config option like config.database_namespaces = [:primary, :animals] we could then just loop through those and create the tasks, but this is extra work app engineers need to do.

Clearly it's not acceptable to just break this but I also don't want to release Rails 6 without the rake tasks for multiple databases 😢

[eileencodes]

@tenderlove @rafaelfranca and/or @matthewd do you have any ideas how I can create the rake tasks for multiple database like db:create:animals without loading the env or without parsing the erb in the yaml file?

We can load the yaml and ignore the ERB but if an app uses multi-line erb like we do at GitHub it will blow up with Psych::SyntaxError: (<unknown>): found character that cannot start any token while scanning for the next token at line 11 column 3.

If we try to load the rake tasks assuming there's no Rails configuration we get this issue. Because the env isn't loaded max_threads isn't defined.

[tenderlove]

We could parse the ERB, but either not evaluate the Ruby code, or just put in dummy values. Are the namespaces going to be dynamic? Nobody can configure the namespaces via Rails.application.configure { }, right?