- Functionally shard into two databases: user/school/classroom information vs student activity (by model, essentially)
- Shard student activity further by school
- Rails 3.2 & postgres support
- Be able to optionally run tests with the sharding config
- Have factories create instances in the right shard
- Get all tests to pass
- Is this the best/right way to be configuring octopus?
- Would be great to have a nice way to move data between shards
- Clean up, generalize, test, and submit patches to upstream octopus
Octopus - Easy Database Sharding for ActiveRecord
Octopus is a better way to do Database Sharding in ActiveRecord. Sharding allows multiple databases in the same rails application. While there are several projects that implement Sharding (e.g. DbCharmer, DataFabric, MultiDb), each project has its own limitations. The main goal of octopus project is to provide a better way of doing Database Sharding.
The api is designed to be simple as possible. Octopus focuses on the end user, giving the power of multiple databases but with reliable code and flexibility. Octopus is focused on Rails 3, but is compatible with Rails 2.x.
- Sharding (with multiple shards, and grouped shards).
- Replication (Master/slave support, with multiple slaves).
- Moving data between shards with migrations.
- Tools to manage database configurations. (soon)
When using replication, all writes queries will be sent to master, and read queries to slaves. More info could be found at: Wiki
When using sharding, you need to specify which shard to send the query. Octopus supports selecting the shard inside a controller, or manually in each object. More could be found at Wiki
Replication + Sharding
Replication + Sharding isn't supported yet. This is on our TODO list and will be done ASAP. If you need, feel free to fork and implement it.
Install the octopus gem:
sudo gem install ar-octopus
Add this line to enviroment.rb:
config.gem 'ar-octopus', :lib => "octopus"
Add this line to Gemfile:
gem 'ar-octopus', :require => 'octopus'
Runs a bundle install:
How to use Octopus?
First, you need to create a config file, shards.yml, inside your config/ directory. to see the syntax and how this file should look, please checkout this page on wiki.
Octopus adds a method to each AR Class and object: the using method is used to select the shard like this:
User.where(:name => "Thiago").limit(3).using(:slave_one)
Octopus also supports queries within a block. When you pass a block to the using method, all queries inside the block will be sent to the specified shard.
Octopus.using(:slave_two) do User.create(:name => "Mike") end
Each model instance knows which shard it came from so this will work automatically:
# This will find the user in the shard1 @user = User.using(:shard1).find_by_name("Joao") # This will find the user in the master database @user2 = User.find_by_name("Jose") #Sets the name @user.name = "Mike" # Save the user in the correct shard, shard1. @user.save
In migrations, you also have access to the using method. The syntax is basically the same. This migration will run in the brazil and canada shards.
class CreateUsersOnBothShards < ActiveRecord::Migration using(:brazil, :canada) def self.up User.create!(:name => "Both") end def self.down User.delete_all end end
You also could send a migration to a group of shards. This migration will be sent to all shards that belongs to history_shards group, specified in shards.yml:
class CreateUsersOnMultiplesGroups < ActiveRecord::Migration using_group(:history_shards) def self.up User.create!(:name => "MultipleGroup") end def self.down User.delete_all end end
If you want to send a specified action, or all actions from a controller, to a specific shard, use this syntax:
class ApplicationController < ActionController::Base around_filter :select_shard def select_shard(&block) Octopus.using(:brazil, &block) end end
To see the complete list of features and syntax, please check out our Wiki Want to see sample rails applications using octopus features? please check it out: Sharding Example and Replication Example. Also, we have an example that shows how to use Octopus without Rails: Octopus + Sinatra Example.
Occasionally, an application might lose a connection to a database; when this happens, ActiveRecord will raise an exception. Add the following line to your application configuration if you are experiencing this issue:
This will tell Octopus to verify the connection before sending the query.
Mixing Octopus with the Rails multiple database model
If you want to set a custom connection to a specific model, use the normal syntax
#This class sets its own connection class CustomConnection < ActiveRecord::Base establish_connection(:adapter => "mysql", :database => "octopus_shard2") end
Set table names
If you want to use specific table names, use the correct syntax for your version of Rails.
Rails <= 3.1
class Bacon < ActiveRecord::Base set_table_name("yummy") end
Rails >= 3.2
class Bacon < ActiveRecord::Base self.table_name = "yummy" end
self.table_name= syntax isn't supported on versions of Rails <= 3.1.
Contributing with Octopus
Contributors are welcome! To run the test suite, you need mysql, postgresql and sqlite3 installed. This is what you need to setup your Octopus development environment:
git clone http://github.com/tchandy/octopus.git cd octopus bundle install bundle exec rake db:prepare # See below about database users bundle exec rake appraisal:install bundle exec rake spec # This runs with only the dependencies in the main Gemfile
To run with a particular appraisal (gemset), use
bundle exec rake appraisal:[appraisal-name] spec where
[appraisal-name] is one of
rails32. The appraisal readme claims that running
rake appraisal spec will run all the appraisals in turn, but for me it just runs the first appraisal and stops.
If you get errors about database usernames, such as
role "postgres" does not exist, you will probably need to set the environment variables MYSQL_USER and/or POSTGRES_USER. See the Rakefile for more details.
This command will run the spec suite for all rails versions supported (Rails 2.3, Rails 3.0 and Rails 3.1). To run our integrations tests inside sample_app, you need to following commands:
cd sample_app bundle install cucumber
If you are having issues running the octopus spec suite, verify your database users and passwords match those inside the config files and your permissions are correct.
Copyright (c) 2010 Thiago Pradi, released under the MIT license.