Octopus.using not working on production/staging

[swapab]

Octopus.using doesn't seem to work on Heroku staging environment. Per octopus.rb, connection should be of type Octopus::Proxy i.e. conn.is_a?(Octopus::Proxy) should be true but it returns false.

I checked on console
Development:

2.0.0p195 :001 > conn = ActiveRecord::Base.connection;nil
 => nil 
2.0.0p195 :002 > conn.class
 => Octopus::Proxy 

Staging:

irb(main):003:0> conn = ActiveRecord::Base.connection;nil
=> nil
irb(main):005:0> conn.class
=> ActiveRecord::ConnectionAdapters::PostgreSQLAdapter

Hence it doesn't use the shard(slave).
Somewhat similar to #82

When restarting a dyno on heroku I can see these logs

2014-03-26T01:14:29.357474+00:00 app[web.2]: => 1 database enabled as read-only slave
2014-03-26T01:14:29.357562+00:00 app[web.2]:  * ROSE follower

I followed the wiki Replication-with-Rails-on-Heroku

• config/initializers/octopus.rb
• config/shards.yml

Also as said is the wiki I have used following HEAD
gem "ar-octopus", github: "rocketmobile/octopus", require: "octopus"

Any help, what extra configs do I need to make it work ?

[mumoshu]

Hi @swapnilabnave

Would you mind removing establish_connection somewhere in your code if it exists?

---

My guess is that you call establish_connection anywhere in your code which is run only in staging environment, and that's causing the problem.
I believe that if you call establish_connection, Octopus stops proxying connections regardless of configuration in shards.yml or Octopus.enabled?.

The logs you see:

2014-03-26T01:14:29.357474+00:00 app[web.2]: => 1 database enabled as read-only slave
2014-03-26T01:14:29.357562+00:00 app[web.2]: * ROSE follower

seem to state shards.yml is read correctly, Octopus is enabled.
But that does not mean Octopus always proxies connections.

It would be helpful to see:
https://o.com/rocketmobile/octopus/blob/master/lib/octopus/model.rb#L156
https://github.com/rocketmobile/octopus/blob/master/lib/octopus/model.rb#L67

[mumoshu]

I'm not sure if it relates to the problem, but you seem to be using a fork of Octopus.
(I see you consulted the wiki page but I'm not sure if it's up to date, too)

gem "ar-octopus", github: "rocketmobile/octopus", require: "octopus"

Would you mind using "tchandy/octopus" or the latest "0.8.1" instead?
Almost all the commits in the fork except one are already merged to tchandy's according to the network.

[mumoshu]

Hi @eprothro

Thank you for writing up a detailed document about replication with Rails on Heroku!
Would you mind helping us about rocketmobile/octopus?

In the last update, you added the below notice:

Note: master HEAD may not work with this configuration currently. If you have problems, use the rocketmobile fork HEAD until this issue is resolved (this message will be removed). gem "ar-octopus", github: "rocketmobile/octopus", require: "octopus"

Do you know if the problem is fixed so that we can always use tchandy/octopus now?

[eprothro]

@mumoshu my apologies for not updating the wiki. Unfortunately, we stopped using replication on that project, so I haven't had the occasion to test tchandy/octopus using the configuration in the wiki.

At the time of writing the note that you referenced, some change had been introduced to tchandy/octopus that resulted in an error during application boot using the configuration in the wiki. I reverted to rocketmobile/octopus to workaround the issue and planned on later digging into what the issue was and revising the recommended configuration so it would be compatible with tchandy/octopus. I won't be able to get around to that for quite a while at least, unfortunately.

Long story short, if the configuration mentioned in the wiki works (doesn't result in boot failures) using tchandy/octopus when deployed to Heroku, the note in the wiki can be removed. If it fails, the failure needs to be debugged and configuration recommendation revised for use of tchandy/octopus on Heroku.

[swapab]

@mumoshu Thank You for looking into this.

@eprothro :

At the time of writing the note that you referenced, some change had been introduced to tchandy/octopus that resulted in an error during application boot using the configuration in the wiki.

Yes, thats right I had issues during application boot but was confined to test env. I didn't tracked much on it. But I switched over to rocketmobile/octopus eventually didn't worked out as expected.

For me gem 'ar-octopus', require: 'octopus', github: 'tchandy/octopus' works perfect. So I guess no harm in updating the wiki.

[eprothro]

Excellent, I agree.

-Evan Prothro
Sent from my mobile

On Apr 24, 2014, at 3:03 AM, Swapnil Abnave notifications@github.com wrote:

@mumoshu Thank You for looking into this.

@eprothro :

At the time of writing the note that you referenced, some change had been introduced to tchandy/octopus that resulted in an error during application boot using the configuration in the wiki.

Yes, thats right I had issues during application boot but was confined to test env. I didn't tracked much on it. But I switched over to rocketmobile/octopus eventually didn't worked out as expected.

For me gem 'ar-octopus', require: 'octopus', github: 'tchandy/octopus' works perfect. So I guess no harm in updating the wiki.

—
Reply to this email directly or view it on GitHub.

[sobrinho]

Closing since it appears to be fixed on latest octopus release.

[JigarS91]

Hi There,

We are having issues with connecting to our replica database using Octopus. Issue is occurring on production and works completely fine on QA.

When we run command - Octopus.using(:replica1) { User.last }. We get below error: -

Octopus.using(:replica1) { User.last }
Mysql2::Error: Can't connect to MySQL server on 'XXXXXXXXXXXXXXXXXXXXXXX' (4)
from /usr/local/bundle/gems/mysql2-0.3.18/lib/mysql2/client.rb:70:in connect' from /usr/local/bundle/gems/mysql2-0.3.18/lib/mysql2/client.rb:70:in initialize'
from /usr/local/bundle/gems/activerecord-4.2.7/lib/active_record/connection_adapters/mysql2_adapter.rb:18:in new' from /usr/local/bundle/gems/activerecord-4.2.7/lib/active_record/connection_adapters/mysql2_adapter.rb:18:in mysql2_connection'
from /usr/local/bundle/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:438:in new_connection' from /usr/local/bundle/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:448:in checkout_new_connection'
from /usr/local/bundle/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:422:in acquire_connection' from /usr/local/bundle/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:349:in block in checkout'
from /usr/local/lib/ruby/2.3.0/monitor.rb:214:in mon_synchronize' from /usr/local/bundle/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:348:in checkout'
from /usr/local/bundle/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:263:in block in connection' from /usr/local/lib/ruby/2.3.0/monitor.rb:214:in mon_synchronize'
from /usr/local/bundle/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:262:in connection' from /usr/local/bundle/gems/ar-octopus-0.9.2/lib/octopus/proxy.rb:76:in safe_connection'
from /usr/local/bundle/gems/ar-octopus-0.9.2/lib/octopus/proxy.rb:80:in select_connection' from /usr/local/bundle/gems/ar-octopus-0.9.2/lib/octopus/proxy.rb:25:in quote_table_name'
... 7 levels...
from /usr/local/bundle/gems/activerecord-4.2.7/lib/active_record/querying.rb:3:in last' from (irb):1:in block in irb_binding'
from /usr/local/bundle/gems/ar-octopus-0.9.2/lib/octopus/proxy.rb:86:in block (2 levels) in run_queries_on_shard' from /usr/local/bundle/gems/ar-octopus-0.9.2/lib/octopus/proxy.rb:324:in using_shard'
from /usr/local/bundle/gems/ar-octopus-0.9.2/lib/octopus/proxy.rb:85:in block in run_queries_on_shard' from /usr/local/bundle/gems/ar-octopus-0.9.2/lib/octopus/proxy.rb:308:in keeping_connection_proxy'
from /usr/local/bundle/gems/ar-octopus-0.9.2/lib/octopus/proxy.rb:84:in run_queries_on_shard' from /usr/local/bundle/gems/ar-octopus-0.9.2/lib/octopus.rb:144:in using'
from (irb):1
from /usr/local/bundle/gems/railties-4.2.7/lib/rails/commands/console.rb:110:in start' from /usr/local/bundle/gems/railties-4.2.7/lib/rails/commands/console.rb:9:in start'
from /usr/local/bundle/gems/railties-4.2.7/lib/rails/commands/commands_tasks.rb:68:in console' from /usr/local/bundle/gems/railties-4.2.7/lib/rails/commands/commands_tasks.rb:39:in run_command!'
from /usr/local/bundle/gems/railties-4.2.7/lib/rails/commands.rb:17:in <top (required)>' from bin/rails:9:in require'
from bin/rails:9:in `

'

Mysql Version is same on both the databases as well as shards.yml is same.

Below is shards.yml config: -

<%
require 'cgi'
require 'uri'
if ENV['DATABASE_REPLICA_URL'].present?
config = {}
replica_url = ENV['DATABASE_REPLICA_URL']
uri = URI.parse(replica_url)
config['host'] = uri.host
config['username'] = uri.user
config['password'] = uri.password
config['database'] = (uri.path || '').split('/')[1]
end
%>

octopus:
replicated: false
fully_replicated: false
environments:
<%- if config.present? %>
<%= "- #{ENV["RAILS_ENV"] || ENV["RACK_ENV"] || Rails.env}" %>
<%= ENV['RAILS_ENV'] || ENV['RACK_ENV'] || Rails.env %>:
replica1:
adapter: mysql2
host: <%= config['host'] %>
username: <%= config['username'] %>
password: <%= config['password'] %>
database: <%= config['database'] %>
reconnect: true
encoding: utf8mb4
<% else %>
- none
<% end %>

Checked firewall and everything looks good. Unable to figure it out. Please assist. I am able to connect to the Database using mysql command line from the source instance.