Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rake db:migrate should run only on master (shouldn't try to run on slaves) #353

Open
ecylmz opened this issue Feb 9, 2016 · 14 comments
Open

Comments

@ecylmz
Copy link

ecylmz commented Feb 9, 2016

I have fully replicated MySQL Master-Slave databases. I just want to run rake db:migrate command on master shard.

If Octopus try to run rake db:migrate command on slaves, MySQL Slaves stop working.

My migration files and models not include any using(:shard) blocks.

I just want to send all write queries and migration commands to master, and all reads queries to slaves. MySQL is handling replication job.

Detailed information are listed below.

Problem steps:

  • run rake db:migrate RAILS_ENV=development
  • Octopus try to run migration command. Trying order: (slave1, slave2, slave3, master) (I dont want to run migration command on slaves but Octopus trying...)
  • This time MySQL slaves stop working. Because MySQL configured as slaves.

Example MySQL Error:

[ERROR] Slave SQL: Error 'Table 'schema_migrations' already exists' on query. Default database: 'ustad_development'. Query: 'CREATE TABLE `schema_migrations` (`version` varchar(255) NOT NULL) ENGINE=InnoDB', Error_code: 1050

[Warning] Slave: Table 'schema_migrations' already exists Error_code: 1050

[ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysql
-bin.000001' position 310

As a result, I just want to run rake db:migrate command on master shard. What am i missing?

# config/shards.yml

octopus:
  replicated: true
  fully_replicated: true
  environments:
    - development

  development:
    slave1:
      adapter: mysql2
      encoding: utf8
      pool: 10
      username: <%= ENV['MYSQL_SLAVE_1_ENV_MYSQL_USER'] %>
      password: <%= ENV['MYSQL_SLAVE_1_ENV_MYSQL_PASS'] %>
      host: <%= ENV['MYSQL_SLAVE_1_PORT_3306_TCP_ADDR'] %>
      port: <%= ENV['MYSQL_SLAVE_1_PORT_3306_TCP_PORT'] %>
      database: foo_development
    slave2:
      adapter: mysql2
      encoding: utf8
      pool: 10
      username: <%= ENV['MYSQL_SLAVE_2_ENV_MYSQL_USER'] %>
      password: <%= ENV['MYSQL_SLAVE_2_ENV_MYSQL_PASS'] %>
      host: <%= ENV['MYSQL_SLAVE_2_PORT_3306_TCP_ADDR'] %>
      port: <%= ENV['MYSQL_SLAVE_2_PORT_3306_TCP_PORT'] %>
      database: foo_development
    slave3:
      adapter: mysql2
      encoding: utf8
      pool: 10
      username: <%= ENV['MYSQL_SLAVE_3_ENV_MYSQL_USER'] %>
      password: <%= ENV['MYSQL_SLAVE_3_ENV_MYSQL_PASS'] %>
      host: <%= ENV['MYSQL_SLAVE_3_PORT_3306_TCP_ADDR'] %>
      port: <%= ENV['MYSQL_SLAVE_3_PORT_3306_TCP_PORT'] %>
      database: foo_development
# config/database.yml

production:
  adapter: mysql2
  encoding: utf8
  pool: 10
  database: foo
  username: <%= Rails.application.secrets.database['username'] %>
  password: <%= Rails.application.secrets.database['password'] %>
  host: <%= Rails.application.secrets.database['host'] %>
  port: <%= Rails.application.secrets.database['port'] %>

development:
  adapter: mysql2
  encoding: utf8
  pool: 10
  database: foo_development
  username: <%= Rails.application.secrets.database['username'] %>
  password: <%= Rails.application.secrets.database['password'] %>
  host: <%= Rails.application.secrets.database['host'] %>
  port: <%= Rails.application.secrets.database['port'] %>

test:
  adapter: mysql2
  encoding: utf8
  pool: 10
  database: foo_test
  username: <%= Rails.application.secrets.database['username'] %>
  password: <%= Rails.application.secrets.database['password'] %>
  host: <%= Rails.application.secrets.database['host'] %>
  port: <%= Rails.application.secrets.database['port'] %>

@sobrinho: Can you help me? :(

@anhkind
Copy link

anhkind commented Mar 7, 2016

Does the comment from velles here #345 work for you?

@ecylmz
Copy link
Author

ecylmz commented Mar 9, 2016

@anhkind No, this method adds the master shard at the end of the shards list. So, Octopus is trying in that order: slave1, slave2, slave3, master.

@anhkind
Copy link

anhkind commented Mar 9, 2016

Can you paste the content of the migration file that failed the migration here?

@ecylmz
Copy link
Author

ecylmz commented Mar 10, 2016

# migration file example
class CreateUsers < ActiveRecord::Migration
  def change
    create_table :users do |t|
      t.string   :first_name,       null: false, limit: 40
      t.string   :last_name,        null: false, limit: 40
      t.string   :email,            null: false, limit: 100
      t.string   :password_digest,  null: false
      t.string   :picture,          limit: 50
      t.string   :cid,              limit: 20
      t.string   :cell_phone,       limit: 20
      t.string   :address,          limit: 250
      t.string   :utype,            null: false, limit: 20
      t.integer  :institution_id,   null: false
      t.integer  :city_id,          null: false
      t.integer  :gender,           null: false, limit: 1, default: 0
      t.integer  :blood_group,      null: false, limit: 1, default: 0
      t.integer  :status,           null: false, limit: 1, default: 0
      t.date     :birth_date

      t.timestamps null: false
    end

    add_index :users, :institution_id
    add_index :users, :utype
    add_index :users, :email, unique: true
  end
end

All files like this. They only create the table.I didn't make any changes on migration files for the Octopus.

@anhkind
Copy link

anhkind commented Mar 12, 2016

Are you running version 0.8.5 or from github's master branch? Can you try the master branch to see if it helps?

@kritik
Copy link

kritik commented Aug 15, 2016

Didn't work for me with master's code

@francois-tilkee
Copy link

Same issue here

@ryanwjackson
Copy link

ryanwjackson commented Apr 26, 2017

I think I'm having a related issue where I run rake db:migrate:status and it shows all up but the last, newly created migration. And then if I run rake db:migrate it errors because it's trying to run the very first migration.

I'm running postgres and Octopus 0.9.0. The error output from rake db:migrate is because of a now mis-named migration that was run years ago. In every other environment, it doesn't matter as it's already been run.

Here is the slimmed-down output:

root@402af45ba5c7:/paid# rake db:migrate:status --trace
** Invoke db:migrate:status (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke db:load_config (first_time)
** Execute db:load_config
** Execute db:migrate:status

database: app

 Status   Migration ID    Migration Name
--------------------------------------------------
   up     20141017224634  Create extensions
   up     20141117224634  Create API keys
...
   up     20170418194101  Add business support phone to accounts
  down    20170426043725  Add temp to transactions

root@402af45ba5c7:/paid# rake db:migrate --trace
** Invoke db:migrate (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke db:load_config (first_time)
** Execute db:load_config
** Execute db:migrate
rake aborted!
NameError: uninitialized constant CreateAPIKeys
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activesupport-4.2.0/lib/active_support/inflector/methods.rb:261:in `const_get'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activesupport-4.2.0/lib/active_support/inflector/methods.rb:261:in `block in constantize'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activesupport-4.2.0/lib/active_support/inflector/methods.rb:259:in `each'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activesupport-4.2.0/lib/active_support/inflector/methods.rb:259:in `inject'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activesupport-4.2.0/lib/active_support/inflector/methods.rb:259:in `constantize'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activesupport-4.2.0/lib/active_support/core_ext/string/inflections.rb:66:in `constantize'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activerecord-4.2.0/lib/active_record/migration.rb:775:in `load_migration'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activerecord-4.2.0/lib/active_record/migration.rb:770:in `migration'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/migration.rb:146:in `shards'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/migration.rb:96:in `block in migrations_with_octopus'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/migration.rb:96:in `select'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/migration.rb:96:in `migrations_with_octopus'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activerecord-4.2.0/lib/active_record/migration.rb:1001:in `target'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activerecord-4.2.0/lib/active_record/migration.rb:948:in `migrate'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/migration.rb:86:in `migrate_with_octopus'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activerecord-4.2.0/lib/active_record/migration.rb:820:in `up'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/migration.rb:110:in `up_with_octopus'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activerecord-4.2.0/lib/active_record/migration.rb:798:in `migrate'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/migration.rb:104:in `block in migrate_with_octopus'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/proxy.rb:246:in `block (2 levels) in run_queries_on_shard'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/proxy.rb:509:in `using_shard'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/proxy.rb:245:in `block in run_queries_on_shard'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/proxy.rb:492:in `keeping_connection_proxy'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/proxy.rb:244:in `run_queries_on_shard'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/proxy.rb:253:in `block in send_queries_to_multiple_shards'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/proxy.rb:252:in `map'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/proxy.rb:252:in `send_queries_to_multiple_shards'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/migration.rb:103:in `migrate_with_octopus'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activerecord-4.2.0/lib/active_record/tasks/database_tasks.rb:137:in `migrate'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activerecord-4.2.0/lib/active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/task.rb:240:in `call'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/task.rb:240:in `block in execute'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/task.rb:235:in `each'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/task.rb:235:in `execute'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/task.rb:179:in `block in invoke_with_call_chain'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/monitor.rb:211:in `mon_synchronize'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/task.rb:172:in `invoke_with_call_chain'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/task.rb:165:in `invoke'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/application.rb:150:in `invoke_task'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/application.rb:106:in `block (2 levels) in top_level'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/application.rb:106:in `each'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/application.rb:106:in `block in top_level'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/application.rb:115:in `run_with_threads'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/application.rb:100:in `top_level'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/application.rb:78:in `block in run'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/application.rb:176:in `standard_exception_handling'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/application.rb:75:in `run'
/root/.rbenv/versions/2.2.1/bin/rake:33:in `<main>'
Tasks: TOP => db:migrate

@ryanwjackson
Copy link

@ecylmz and @francois-tilkee, did you eve figure out a solution here?

@francois-tilkee
Copy link

@ryanwjackson Nope... I still have to scroll my terminal up for 20 seconds to see what happened in my migration.

@ryanwjackson
Copy link

So, I actually figured out mine. It was an issue with how the migrations were being loaded by Octopus. We had added some inflections, which meant that those classes couldn't be loaded properly. Normal rails migrations apparently doesn't attempt to load in the same way, but Octopus does. So renaming the classes properly worked.

That said, are you doing replication or sharding?

@francois-tilkee
Copy link

Replication

@asafjaffi
Copy link

Hi,
Is there any chance this got fixed somehow?

@tuliang
Copy link

tuliang commented Jan 10, 2019

see #345 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants