fiveruns / data_fabric
watch download tarball
public
Forks8Right
Watchers219Right
Description: Sharding support for ActiveRecord 2.x
Homepage: http://github.com/fiveruns/data_fabric
Clone URL: git://github.com/fiveruns/data_fabric.git
data_fabric / README.rdoc
raw normal view history
ee9ec728 » mperham 2009-05-05 Point to mperham repo 1 Please note this repository has moved to http://github.com/mperham/data_fabric
2
62bdfd97 » mperham 2008-04-16 data_fabric cleanup 3 = data_fabric
4
5 DataFabric provides flexible database connection switching for ActiveRecord.
6
7 We needed two features to scale our mysql database: application-level sharding and
8 master/slave replication. Sharding is the process of splitting a dataset across many
9 independent databases. This often happens based on geographical region (e.g.
10 craigslist) or category (e.g. ebay). Replication provides a near-real-time copy
11 of a database which can be used for fault tolerance and to reduce load on the master
12 node. Combined, you get a scalable database solution which does not require huge
13 hardware to scale to huge volumes. Or: DPAYEIOB - don't put all your eggs in one
14 basket. :-)
15
16 == Installation
17
be845208 » mperham 2008-07-08 Move gem building to Echoe 18 gem install data_fabric
62bdfd97 » mperham 2008-04-16 data_fabric cleanup 19
20 == How does it work?
21
22 You describe the topology for your database infrastructure in your model(s).
23 Different models can use different topologies.
24
25 class MyHugeVolumeOfDataModel < ActiveRecord::Base
14f4433d » mperham 2008-07-09 Change connection_topology ... 26 data_fabric :replicated => true, :shard_by => :city
62bdfd97 » mperham 2008-04-16 data_fabric cleanup 27 end
28
29 There are four supported modes of operation, depending on the options given to the
14f4433d » mperham 2008-07-09 Change connection_topology ... 30 data_fabric method. The plugin will look for connections in your
62bdfd97 » mperham 2008-04-16 data_fabric cleanup 31 config/database.yml with the following convention:
32
33 No connection topology:
34 #{environment} - this is the default, as with ActiveRecord, e.g. "production"
35
14f4433d » mperham 2008-07-09 Change connection_topology ... 36 data_fabric :replicated => true
62bdfd97 » mperham 2008-04-16 data_fabric cleanup 37 #{environment}_#{role} - no sharding, just replication, where role is "master" or "slave", e.g. "production_master"
38
14f4433d » mperham 2008-07-09 Change connection_topology ... 39 data_fabric :shard_by => :city
62bdfd97 » mperham 2008-04-16 data_fabric cleanup 40 #{group}_#{shard}_#{environment} - sharding, no replication, e.g. "city_austin_production"
41
14f4433d » mperham 2008-07-09 Change connection_topology ... 42 data_fabric :replicated => true, :shard_by => :city
62bdfd97 » mperham 2008-04-16 data_fabric cleanup 43 #{group}_#{shard}_#{environment}_#{role} - sharding with replication, e.g. "city_austin_production_master"
44
45
46 When marked as replicated, all write and transactional operations for the model
47 go to the master, whereas read operations go to the slave.
48
49 Since sharding is an application-level concern, your application must set the shard
be845208 » mperham 2008-07-08 Move gem building to Echoe 50 to use based on the current request or environment. The current shard
62bdfd97 » mperham 2008-04-16 data_fabric cleanup 51 is set on a thread local variable. For example, you can set the shard in an
ed83276e » mperham 2008-07-02 Updated documentation 52 ActionController around_filter based on the user as follows:
62bdfd97 » mperham 2008-04-16 data_fabric cleanup 53
54 class ApplicationController < ActionController::Base
55 around_filter :select_shard
56
57 private
58 def select_shard(&block)
ed83276e » mperham 2008-07-02 Updated documentation 59 DataFabric.activate_shard(:city => @current_user.city, &block)
62bdfd97 » mperham 2008-04-16 data_fabric cleanup 60 end
61 end
62
9b899f19 » mperham 2008-07-01 Add note and check for allo... 63
64 == Warnings
65
66 * Sharded models should never be placed in the session store or you will get "Shard not set" errors when the session is persisted.
7d7a2acf » mperham 2008-11-23 Refactor DataFabric to supp... 67 * DataFabric does not support running with ActiveRecord's allow_concurrency = true in AR 2.0 and 2.1. allow_concurrency is gone in AR 2.2.
62bdfd97 » mperham 2008-04-16 data_fabric cleanup 68
ed83276e » mperham 2008-07-02 Updated documentation 69 == Testing and Bug Reports
70
71 If you think you've found a problem with data_fabric, please use the example application to reproduce the bug and send me the diff. The example application is a stock Rails 2.1 application which uses data_fabric in the expected fashion.
72
62bdfd97 » mperham 2008-04-16 data_fabric cleanup 73
74 == Thanks to...
75
76 Rick Olsen of Rails Core:: for the Masochism plugin, which showed me how to bend AR's connection handling to my will
77 Bradley Taylor of RailsMachine:: for the advice to shard while at acts_as_conference
78 FiveRuns:: for paying me to develop this code and allowing its release
79
80
81 == Author
82
83 Mike Perham <mperham@gmail.com>
ed83276e » mperham 2008-07-02 Updated documentation 84 http://mikeperham.com
62bdfd97 » mperham 2008-04-16 data_fabric cleanup 85
86
87 == LICENSE:
88
89 (The FiveRuns License)
90
91 Copyright (c) 2008 FiveRuns Corporation
92
93 Permission is hereby granted, free of charge, to any person obtaining
94 a copy of this software and associated documentation files (the
95 'Software'), to deal in the Software without restriction, including
96 without limitation the rights to use, copy, modify, merge, publish,
97 distribute, sublicense, and/or sell copies of the Software, and to
98 permit persons to whom the Software is furnished to do so, subject to
99 the following conditions:
100
101 The above copyright notice and this permission notice shall be
102 included in all copies or substantial portions of the Software.
103
104 THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
105 EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
106 MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
107 IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
108 CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
109 TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
ed83276e » mperham 2008-07-02 Updated documentation 110 SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.