Introducing slave groups, can be used concurrently with sharding

[mumoshu]

Hi, @tchandy and commiters

First of all, I would like to thank you for sharing this great library 😄
At CrowdWorks, we are using Octopus in production and it is working like a charm 👍

In this pull request, I'm introducing "slave groups" to make Octopus distribute queries to a specific group of slaves.
They can be used concurrently with shards, e.g. merging this pull request also adds "Replication + Sharding" in your TODO list.
This change is backward-compatible.

---

A slave group is a group of slaves to which queries are distributed.
In other words, all the slaves in the same slave group are load-balanced.
Octopus does not load-balance between slaves across multiple slave groups.

An usecase of slave groups is to distribute/completely seperate loads of SELECT queries from each part of an application to seperate databases.
For example, consider that you have a Web application consists of the A part which serves Web pages for users, and the B part which serves Web pages for admins.
If you just send heavy SELECT queries while serving for admins, they will degrade performance serving Web pages for other users.
This is where slave groups come in.
You can configure seperate slave groups for A and B to make them not affect each other's performance.

You can configure any number of slave groups in shards.yml.
To configure two slave groups slave_group_a and slave_group_b for the development environment, you need shards.yml like:

octopus:
  replicated: true
  environments:
    - development
  development:
    slave_group_a:
      slave1:
        # ...
      slave2:
        # ...
    slave_group_b:
      slave3:
        # ...
      slave4:
        # ...

Then:

• In Octopus.using(slave_group: :slave_group_a) blocks, all the SELECT queries are distributed between slave1 and slave2
• In Octopus.using(slave_group: :slave_group_b) blocks, all the SELECT queries are distribuetd between slave3 and slave4
• Without using, all the SELECT queries go to the development database configured in database.yml

To use with sharding, you need shards.yml like:

octopus:
  replicated: true
  environments:
    - development
  development:
    shard_1:
      slave_group_a:
        slave1:
          # ...
        slave2:
          # ...
      slave_group_b:
        slave3:
          # ...
        slave4:
          # ...
    shard_2:
      slave_group_a:
        slave5:
          # ...

Then:

• In Octopus.using(shard: :shard_1, slave_group: :slave_group_a) blocks, all the SELECT queries are distributed between slave1 and slave2
• In Octopus.using(shard: :shard_1, slave_group: :slave_group_b) blocks, all the SELECT queries are distribuetd between slave3 and slave4
• Without using, all the SELECT queries go to the development database configured in database.yml

[thiagopradi]

hey @mumoshu, awesome feature!

I'll ask the other maintainers to review it. Could you add some documentation about this feature to the README file? Maybe overwriting the section about sharding + replication.

Thanks

[nickmarden]

@tchandy I can spend some time reviewing this over the weekend. Please assign me to the PR?

[mumoshu]

Hi, @tchandy and @nickmarden 😄

Could you add some documentation about this feature to the README file?

Sure!
And I have just added a commit to update README with a small description of slave groups which also mentions a Wiki page for further information.
May I have comments for it? I'm glad to update it so that everyone can try this feature with less hassle.

Thanks

[mumoshu]

@nickmarden

Thank you for reviewing!
I have just added commits in answer to your reviews.
Would you mind reviewing once again?

[nickmarden]

@mumoshu: I have some things to do today and tomorrow, but I will look again tomorrow night or Thursday morning.

[nickmarden]

Looks good, merging.

[mumoshu]

@nickmarden

Thanks for merging!

[mumoshu]

Hi @tchandy

Would you mind releasing it as 0.8.x?

As I have experiences in managing gems, I think I can do it for you if you make me as an additional owner for the gem.

Thanks