Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

ActiveRecord::ConnectionNotEstablished with rails 3.2 #2

Closed
antage opened this Issue · 11 comments

2 participants

@antage

I try to use multidb in production environment with rails 3.2 but got following error for any request:

ActiveRecord::ConnectionNotEstablished (ActiveRecord::ConnectionNotEstablished):
  activerecord (3.2.11) lib/active_record/connection_adapters/abstract/connection_pool.rb:323:in `checkout_new_connection'
  activerecord (3.2.11) lib/active_record/connection_adapters/abstract/connection_pool.rb:247:in `block (2 levels) in checkout'
  activerecord (3.2.11) lib/active_record/connection_adapters/abstract/connection_pool.rb:242:in `loop'
  activerecord (3.2.11) lib/active_record/connection_adapters/abstract/connection_pool.rb:242:in `block in checkout'
  /usr/lib/rbenv/versions/1.9.3-p327/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
  activerecord (3.2.11) lib/active_record/connection_adapters/abstract/connection_pool.rb:239:in `checkout'
  activerecord (3.2.11) lib/active_record/connection_adapters/abstract/connection_pool.rb:102:in `block in connection'
  /usr/lib/rbenv/versions/1.9.3-p327/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
  activerecord (3.2.11) lib/active_record/connection_adapters/abstract/connection_pool.rb:101:in `connection'
  /home/avaxhome/app.com/shared/bundle/ruby/1.9.1/bundler/gems/multidb-2a9ad2f341fd/lib/multidb/balancer.rb:23:in `connection'
  /home/avaxhome/app.com/shared/bundle/ruby/1.9.1/bundler/gems/multidb-2a9ad2f341fd/lib/multidb/balancer.rb:87:in `current_connection'
  /home/avaxhome/app.com/shared/bundle/ruby/1.9.1/bundler/gems/multidb-2a9ad2f341fd/lib/multidb/model_extensions.rb:18:in `connection_with_multidb'
  activerecord (3.2.11) lib/active_record/query_cache.rb:67:in `rescue in call'
  activerecord (3.2.11) lib/active_record/query_cache.rb:61:in `call'
  activerecord (3.2.11) lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in `call'
  actionpack (3.2.11) lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
  activesupport (3.2.11) lib/active_support/callbacks.rb:405:in `_run__2689738203702554028__call__2455757002539370000__callbacks'
  activesupport (3.2.11) lib/active_support/callbacks.rb:405:in `__run_callback'
  activesupport (3.2.11) lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
  activesupport (3.2.11) lib/active_support/callbacks.rb:81:in `run_callbacks'
  actionpack (3.2.11) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  rack (1.4.3) lib/rack/sendfile.rb:102:in `call'
  actionpack (3.2.11) lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
  actionpack (3.2.11) lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
  actionpack (3.2.11) lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
  railties (3.2.11) lib/rails/rack/logger.rb:32:in `call_app'
  railties (3.2.11) lib/rails/rack/logger.rb:16:in `block in call'
  activesupport (3.2.11) lib/active_support/tagged_logging.rb:22:in `tagged'
  railties (3.2.11) lib/rails/rack/logger.rb:16:in `call'
  actionpack (3.2.11) lib/action_dispatch/middleware/request_id.rb:22:in `call'
  rack (1.4.3) lib/rack/methodoverride.rb:21:in `call'
  rack (1.4.3) lib/rack/runtime.rb:17:in `call'
  rack (1.4.3) lib/rack/lock.rb:15:in `call'
  rack-cache (1.2) lib/rack/cache/context.rb:136:in `forward'
  rack-cache (1.2) lib/rack/cache/context.rb:245:in `fetch'
  rack-cache (1.2) lib/rack/cache/context.rb:185:in `lookup'
  rack-cache (1.2) lib/rack/cache/context.rb:66:in `call!'
  rack-cache (1.2) lib/rack/cache/context.rb:51:in `call'
  airbrake (3.1.5) lib/airbrake/rack.rb:41:in `call'
  airbrake (3.1.5) lib/airbrake/user_informer.rb:12:in `call'
  railties (3.2.11) lib/rails/engine.rb:479:in `call'
  railties (3.2.11) lib/rails/application.rb:223:in `call'
  railties (3.2.11) lib/rails/railtie/configurable.rb:30:in `method_missing'
  unicorn (4.3.0) lib/unicorn/http_server.rb:530:in `process_client'
  unicorn (4.3.0) lib/unicorn/http_server.rb:605:in `worker_loop'
  newrelic_rpm (3.5.3.25) lib/new_relic/agent/instrumentation/unicorn_instrumentation.rb:18:in `call'
  newrelic_rpm (3.5.3.25) lib/new_relic/agent/instrumentation/unicorn_instrumentation.rb:18:in `block (4 levels) in <top (required)>'
  unicorn (4.3.0) lib/unicorn/http_server.rb:487:in `spawn_missing_workers'
  unicorn (4.3.0) lib/unicorn/http_server.rb:137:in `start'
  unicorn (4.3.0) bin/unicorn:121:in `<top (required)>'
  /home/avaxhome/app.com/shared/bundle/ruby/1.9.1/bin/unicorn:23:in `load'
  /home/avaxhome/app.com/shared/bundle/ruby/1.9.1/bin/unicorn:23:in `<main>'

I use Multidb.use(:slave, &block) in around_filter.

database.yml:

login: &login
  adapter: postgresql
  host: aaa.bbb.ccc.ddd
  username: app
  password: ***
  encoding: UTF8

development:
  database: app
  <<: *login

# Warning: The database defined as 'test' will be erased and
# re-generated from your development database when you run 'rake'.
# Do not set this db to the same as development or production.
test:
  database: app_test
  <<: *login

production:
  database: app
  <<: *login
  multidb:
    databases:
      slave:
        host: aaa.bbb.ccc.eee

I changed password and host fields.

I have no idea what's happening.

@antage

I use my fork https://github.com/antage/multidb because without it I can't start rails environment.

@antage

I would to clarify - I got error only for actions with around_filter switching multidb to slave.

@atombender
Owner

I will look into it tomorrow.

@antage

@alexstaubo Ok, thanks.

I investigated that a root of problem is unicorn forking workers.
If I start unicorn with preload_app false - it's ok.
But if I start unicorn with following config I got errors:

preload_app true                                                                                                        

before_fork do |server, worker|
  # the following is highly recomended for Rails + "preload_app true"
  # as there's no need for the master process to hold a connection
  if defined?(::ActiveRecord::Base)
    ::ActiveRecord::Base.connection.disconnect!
  end
end

after_fork do |server, worker|
  ##
  # Unicorn master loads the app then forks off workers - because of the way
  # Unix forking works, we need to make sure we aren't using any of the parent's
  # sockets, e.g. db connection

  if defined?(::ActiveRecord::Base)
    ::ActiveRecord::Base.establish_connection
  end
end

Any ideas how to restart connections properly with multidb?

@atombender
Owner

Try calling Multidb.install! -- or if that doesn't work, Multidb.configure!.

@antage

@alexstaubo Multidb.configure! doesn't help but Multidb.install! works. Thanks!

@antage antage closed this
@antage antage reopened this
@antage

Same problem with delayed_job. After DJ forks worker it can't see connection to DB. I wrote monkey patch for Delayed::Backend::ActiveRecord::Job.after_fork but it's so dirty :)
Could Multidb check a connection and reestablished it if the connection is closed?

@atombender
Owner

How does your delayed_job fix work? The problem is knowing whether a process has forked, which one can't. Multidb just uses the connection pool and assumes that it's operational. The responsibility is really with the Unicorn config, I think.

This, by the way, is why I wrote Hupper, a simple gem that allows libraries to register their own after_fork handlers so that Unicorn etc. don't need to know about them. But it's not like anyone else has been using it.

@antage

How does your delayed_job fix work?

I invoke Multidb.install! in Delayed::Backend::ActiveRecord::Job.after_fork.

The problem is knowing whether a process has forked, which one can't.

You're right. But Octopus works fine with forking workers. If multidb hooked AR::B.clear_all_connections! and AR::B.establish_connection it would refresh connection pools in Balancer.

But if you want to keep code simple just close #1 and #2 :)

@atombender
Owner

I believe this works now. Can you try?

@atombender
Owner

No response, so closing for now.

@atombender atombender closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.