Browse files

Update demo to work as intended

  • Loading branch information...
1 parent 0b6d8ae commit bcbbd63822123dfb03d1857eb4aeed9b7277a169 @boazavital boazavital committed Sep 26, 2011
Showing with 39 additions and 42 deletions.
  1. +6 −8 README.markdown
  2. +1 −1 config/development.scala
  3. +28 −29 doc/demo.markdown
  4. +4 −4 src/scripts/mkshards.rb
@@ -71,7 +71,7 @@ but there are some pre-requisites. You need:
- java 1.6
- sbt 0.7.4
-- thrift 0.2.0
+- thrift 0.5.0
If you haven't used sbt before, this page has a quick setup:
@@ -80,14 +80,12 @@ My `~/bin/sbt` looks like this:
java -server -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m -Xmx1024m -jar `dirname $0`/sbt-launch-0.7.4.jar "$@"
-Apache Thrift 0.2.0 is pre-requisite for building java stubs of the thrift
+Apache Thrift 0.5.0 is pre-requisite for building java stubs of the thrift
IDL. It can't be installed via jar, so you'll need to install it separately
-before you build. It can be found on the apache incubator site:
-This project is still getting its legs, so it can be hard to find the right
-version. If you have trouble with the mirrors, version 0.2.0 should be hosted
+before you build. It can be found on the apache thrift site:
+You can find the download for 0.5.0 here:
In addition, the tests require a local mysql instance to be running, and for
`DB_USERNAME` and `DB_PASSWORD` env vars to contain login info for it. You can
@@ -75,7 +75,7 @@ new FlockDB {
val databaseConnection = new Credentials {
val hostnames = Seq("localhost")
- val database = "edges_test"
+ val database = "edges_development"
urlOptions = Map("rewriteBatchedStatements" -> "true")
@@ -277,49 +277,48 @@ What shard is user 123456 on?
Hm, but localhost has been behaving strangely lately. Let's move that shard to, which is
-really lightly loaded. First, create the new shard:
+really lightly loaded. To move individual shards we'll take a look at the current topology and then run a `transform-tree` operation to move things where we want them.
- $ gizzmo create -s "INT UNSIGNED" -d "INT UNSIGNED" edges_99_0008_new com.twitter.flockdb.SqlShard
+To see the current topology of graph 99:
-Then, setup a migration:
+ $ gizzmo -T 99 topology
+ 10 com.twitter.gizzard.shards.ReplicatingShard(1) -> com.twitter.flockdb.SqlShard(localhost,1,INT UNSIGNED,INT UNSIGNED)
- $ gizzmo setup-migrate localhost/edges_99_0008_forward_1
- localhost/edges_99_0008_new_migrate_replica
+As we expect, graph 99 is made up of 10 shards, all of which are replicating to just one server, which is also `localhost` for all them. To see all the tables you have in Flock, you can run `gizzmo tables`.
-If you look at the replication subtree for `localhost/edges_99_0008_forward_replicating`, you can
-see that it's added a layer of replication between the old shard and the new one, and the new one is
-behind a write-only barrier:
+Now let's modify shard 8 to replicate only to the new host we want, We'll specify the new topology template we want, and which shard that should apply to:
- $ gizzmo subtree localhost/edges_99_0008_forward_replicating
- localhost/edges_99_0008_forward_replicating
- localhost/edges_99_0008_new_migrate_replica
- localhost/edges_99_0008_forward_1
- localhost/edges_99_0008_new_migrate_write_only
-So let's reload the forwarding table to make sure everyone starts using this replication:
+ $ gizzmo transform-tree "com.twitter.gizzard.shards.ReplicatingShard(1) -> (com.twitter.flockdb.SqlShard(,1,INT UNSIGNED,INT UNSIGNED))" localhost/edges_99_0008_forward_replicating
- $ gizzmo reload
-Now, all writes are going to both places and we can start the copy:
+ com.twitter.gizzard.shards.ReplicatingShard(1) -> com.twitter.flockdb.SqlShard(localhost,1,INT UNSIGNED,INT UNSIGNED) => com.twitter.gizzard.shards.ReplicatingShard(1) -> com.twitter.flockdb.SqlShard(,1,INT UNSIGNED,INT UNSIGNED) :
+ create_shard(com.twitter.flockdb.SqlShard/
+ create_shard(WriteOnlyShard)
+ add_link(WriteOnlyShard -> com.twitter.flockdb.SqlShard/
+ add_link(com.twitter.gizzard.shards.ReplicatingShard -> WriteOnlyShard)
+ copy_shard(com.twitter.flockdb.SqlShard/
+ add_link(com.twitter.gizzard.shards.ReplicatingShard -> com.twitter.flockdb.SqlShard/
+ remove_link(com.twitter.gizzard.shards.ReplicatingShard -> com.twitter.flockdb.SqlShard/localhost)
+ remove_link(WriteOnlyShard -> com.twitter.flockdb.SqlShard/
+ remove_link(com.twitter.gizzard.shards.ReplicatingShard -> WriteOnlyShard)
+ delete_shard(com.twitter.flockdb.SqlShard/localhost)
+ delete_shard(WriteOnlyShard)
- $ gizzmo copy localhost/edges_99_0008_forward_1
+ Continue? (y/n)
-The destination shard will be marked "busy" during the copy, but because we only had 3 edges in it,
-the copy will probably be done before we can even check:
+gizzmo gives us the plan for this transformation so we can approve it before it makes any changes. This looks good so put in `y` and let it run. The destination shard will be marked "busy" during the copy, if you're quick you might be able to see it marked such by checking with:
$ gizzmo busy
-Yep, no busy shards. We can finish the migration, then, to remove the replication layer and the
-source shard.
+After the operation has finished, you can check to see that it did what we expect:
- $ gizzmo finish-migrate localhost/edges_99_0008_forward_1
$ gizzmo subtree localhost/edges_99_0008_forward_replicating
-Sweet! Reload to tell flockdb to stop writing to the old shard.
+Sweet! Reload to make sure flockdb knows about the right topology.
$ gizzmo reload
@@ -47,7 +47,7 @@
config = YAML.load_file(options[:config_filename]) rescue {}
app_host, app_port = (config['app_host'] || 'localhost').split(':')
-app_port ||= 7917
+app_port ||= 7920
namespace = config['namespace'] || nil
db_trees = Array(config['databases'] || 'localhost')
@@ -67,13 +67,13 @@
types = "-s 'INT UNSIGNED' -d 'INT UNSIGNED'"
[ "forward", "backward" ].each do |direction|
- "create localhost #{table_name}_#{direction}_replicating com.twitter.gizzard.shards.ReplicatingShard"
+ "create com.twitter.gizzard.shards.ReplicatingShard localhost/#{table_name}_#{direction}_replicating"
distinct = 1
hosts.each do |host|
host, weight = host.split(':')
- weight ||= 4
- "create #{types} #{host} #{table_name}_#{direction}_#{distinct} com.twitter.flockdb.SqlShard"
+ weight ||= 1
+ "create #{types} com.twitter.flockdb.SqlShard #{host}/#{table_name}_#{direction}_#{distinct}" "addlink localhost/#{table_name}_#{direction}_replicating #{host}/#{table_name}_#{direction}_#{distinct} #{weight}"
distinct += 1

0 comments on commit bcbbd63

Please sign in to comment.