How to use Gizzmo

98tango edited this page Aug 30, 2012 · 19 revisions
Clone this wiki locally

Gizzmo is the command-line tool used to create and manage Gizzard clusters.

Gizzard decides where to store your data by hashing your input key into buckets. Each bucket forwards to a tree topology, which is comprised of virtual and concrete shards that control how and where your data is stored.

Shard topology

Gizzmo commands talk to Gizzard appservers to query and modify their representations of this data (stored persistently on the nameservers), and to create shards and copy data between shards on the storage layer.

Gizzard with Gizzmo diagram

If you were to query the application server above for the current topology of table 0 (tables will be explained later), you would receive something like this:

 $ gizzmo -P ADMIN_PORT -H APP_HOST -T 0 topology
  34 ReplicatingShard(1) -> (AppShard(DB1),AppShard(DB2),AppShard(DB3))
  33 ReplicatingShard(1) -> (AppShard(DB4),AppShard(DB5),AppShard(DB6))
  33 ReplicatingShard(1) -> (AppShard(DB7),AppShard(DB8),AppShard(DB9))
  ...

The number preceding each topology is the number of shards that fall under it. This topology representation is precise, but can become fairly verbose to view and specify. For a given application with user-specified weights and datatypes and a slightly more complex tree structure, it can look more like this:

$ gizzmo -P ADMIN_PORT -H APP_HOST -T 0 topology
  34 ReplicatingShard(1) -> (com.myapp.shards.AppShard(DB1,1,INT UNSIGNED,INT UNSIGNED), com.myapp.shards.AppShard(DB2,1,INT UNSIGNED,INT UNSIGNED), WriteOnlyShard(1) -> com.myapp.shards.AppShard(DB3,1,INT UNSIGNED,INT UNSIGNED))
  33 ReplicatingShard(1) -> (com.myapp.shards.AppShard(DB4,1,INT UNSIGNED,INT UNSIGNED), com.myapp.shards.AppShard(DB5,1,INT UNSIGNED,INT UNSIGNED), WriteOnlyShard(1) -> com.myapp.shards.AppShard(DB6,1,INT UNSIGNED,INT UNSIGNED))
  33 ReplicatingShard(1) -> (com.myapp.shards.AppShard(DB7,1,INT UNSIGNED,INT UNSIGNED), com.myapp.shards.AppShard(DB8,1,INT UNSIGNED,INT UNSIGNED), WriteOnlyShard(1) -> com.myapp.shards.AppShard(DB9,1,INT UNSIGNED,INT UNSIGNED))
  ...

For this reason there's an option to simplify the representation of these trees, sacrificing some information and taking for granted the structure of your topology for the sake clarity. For the above command it might look like this:

$ gizzmo --simple -P ADMIN_PORT -H APP_HOST -T 0 topology
  34 DB1,DB2,DB3+write_only
  33 DB4,DB5,DB6+write_only
  33 DB7,DB8,DB9+write_only
  ...

The first step in administering a Gizzard cluster with Gizzmo is to create a table or tables with a set of shards. This is as simple as issuing a create-table command. Tables in Gizzard are separate datasets that live on the same cluster. They're a flexible construct you can choose to use for your application. If you only need one simple dataset, having one table with id 0 is probably sufficient. An example of the benefit of tables is FlockDB, a graph datastore that uses pairs of tables (e.g. 1,-1) to model the positive and negative directional edges in a graph store.

Creating a table

You can create a table by specifying pairs of weights and topologies (shown below in both complex and simple formats)

 $ gizzmo -P ADMIN_PORT -H APP_HOST -T 0 create-table --shards=100 \
 > 1 "ReplicatingShard(1) -> (com.myapp.shards.AppShard(DB1,1,INT UNSIGNED,INT UNSIGNED), com.myapp.shards.AppShard(DB2,1,INT UNSIGNED,INT UNSIGNED), WriteOnlyShard(1) -> com.myapp.shards.AppShard(DB3,1,INT UNSIGNED,INT UNSIGNED))" \
 > 1 "ReplicatingShard(1) -> (com.myapp.shards.AppShard(DB4,1,INT UNSIGNED,INT UNSIGNED), com.myapp.shards.AppShard(DB5,1,INT UNSIGNED,INT UNSIGNED), WriteOnlyShard(1) -> com.myapp.shards.AppShard(DB6,1,INT UNSIGNED,INT UNSIGNED))" \
 > 1 "ReplicatingShard(1) -> (com.myapp.shards.AppShard(DB7,1,INT UNSIGNED,INT UNSIGNED), com.myapp.shards.AppShard(DB8,1,INT UNSIGNED,INT UNSIGNED), WriteOnlyShard(1) -> com.myapp.shards.AppShard(DB9,1,INT UNSIGNED,INT UNSIGNED))"

 $ gizzmo --simple -P ADMIN_PORT -H APP_HOST -T 0 create-table --shards=100 --concrete=com.myapp.shards.AppShard --source-type="INT UNSIGNED" --dest-type="INT UNSIGNED" \
 > 1 DB1,DB2,DB3+write_only 1 DB4,DB5,DB6+write_only 1 DB7,DB8,DB9+write_only

Afterwards you'll want to reload the nameserver data onto all your application servers.

$ gizzmo -P ADMIN_PORT -H APP_HOST,APP_HOST2,APP_HOST3... reload

To get a list of all tables, just run

$ gizzmo -P ADMIN_PORT -H APP_HOST tables

Remember that whenever you specify tables with -T, you can specify a comma separated list of tables.

Gizzmo has commands for basic manipulation of topologies, like adding or removing links or creating and deleting shards. It also has some more powerful commands to transform topologies from one structure to another or change the entire composition of the cluster.

Transforming a topology

You can change all shards matching one topology into another topology using the transform command.

$ gizzmo -P ADMIN_PORT -H LIST_OF_ALL_APP_HOSTS -T LIST_OF_TABLES transform \
> "ReplicatingShard(1) -> (AppShard(DB1),AppShard(DB2),AppShard(DB3))" "ReplicatingShard(1) -> (AppShard(DB1),AppShard(DB2),AppShard(DB3), WriteOnlyShard(1) -> AppShard(DBx))"

$ gizzmo --simple -P ADMIN_PORT -H LIST_OF_ALL_APP_HOSTS -T LIST_OF_TABLES transform --concrete=AppShard --source-type=None --dest-type=None \
> DB1,DB2,DB3 DB1,DB2,DB3,DBx+write_only

  PREPARE
    create_shard(AppShard/DBx)
    create_shard(BlockedShard)
    create_shard(WriteOnlyShard)
    add_link(WriteOnlyShard -> BlockedShard)
    add_link(BlockedShard -> AppShard/DBx)
    add_link(ReplicatingShard -> WriteOnlyShard)
  COPY
    copy_shard(AppShard/DB1 -> AppShard/DBx)
  CLEANUP
    add_link(WriteOnlyShard -> AppShard/DBx)
    remove_link(BlockedShard -> AppShard/DBx)
    remove_link(WriteOnlyShard -> BlockedShard)
    delete_shard(BlockedShard)
  Applied to 34 shards

For all transform-related commands, Gizzmo returns a plan telling you what actions it is going to take and asks you to verify whether to continue.

If you need more fine-grained transform control, you can tell Gizzmo to only transform one shard's forwarding using the transform-tree command.

To view all current forwardings and topologies:

$ gizzmo --simple -P ADMIN_PORT -H APP_HOST -T 0 topology --forwardings
[0] 0 = localhost/shard_0_0044_replicating  DB1, DB2, DB3, DBx+write_only
[0] 11eb851eb851eb3 = localhost/shard_0_0088_replicating    DB1, DB2, DB3, DBx+write_only
[0] 147ae147ae147a8 = localhost/shard_0_0063_replicating    DB1, DB2, DB3, DBx+write_only
[0] 170a3d70a3d709d = localhost/shard_0_0071_replicating    DB1, DB2, DB3, DBx+write_only
...

And to transform a specific one, just specify the new topology structure you want for that branch.

$ gizzmo --simple -P ADMIN_PORT -H LIST_OF_ALL_APP_HOSTS transform-tree --concrete=AppShard --source-type=None --dest-type=None \
> DB1,DB2,DB3,DBx+blocked localhost/shard_0_0063_replicating

Rebalancing the cluster

The rebalance command allows you to specify an entirely new set of topologies for your cluster. The syntax is similar to create-table, except it acts on an existing cluster. Gizzmo will attempt to minimize data movement when rebalancing the cluster. This command is generally used to add capacity.

$ gizzmo -P ADMIN_PORT -H LIST_OF_ALL_APP_HOSTS -T LIST_OF_TABLES rebalance --copies-per-host=10 --max-copies=50 \
> 1 "ReplicatingShard(1) -> (com.myapp.shards.AppShard(DB1,1,INT UNSIGNED,INT UNSIGNED), com.myapp.shards.AppShard(DB2,1,INT UNSIGNED,INT UNSIGNED), WriteOnlyShard(1) -> com.myapp.shards.AppShard(DB3,1,INT UNSIGNED,INT UNSIGNED))" \
> 1 "ReplicatingShard(1) -> (com.myapp.shards.AppShard(DB4,1,INT UNSIGNED,INT UNSIGNED), com.myapp.shards.AppShard(DB5,1,INT UNSIGNED,INT UNSIGNED), WriteOnlyShard(1) -> com.myapp.shards.AppShard(DB6,1,INT UNSIGNED,INT UNSIGNED))" \
> 1 "ReplicatingShard(1) -> (com.myapp.shards.AppShard(DB7,1,INT UNSIGNED,INT UNSIGNED), com.myapp.shards.AppShard(DB8,1,INT UNSIGNED,INT UNSIGNED), WriteOnlyShard(1) -> com.myapp.shards.AppShard(DB9,1,INT UNSIGNED,INT UNSIGNED))" \
> 1 "ReplicatingShard(1) -> (com.myapp.shards.AppShard(DBa,1,INT UNSIGNED,INT UNSIGNED), com.myapp.shards.AppShard(DBb,1,INT UNSIGNED,INT UNSIGNED), WriteOnlyShard(1) -> com.myapp.shards.AppShard(DBc,1,INT UNSIGNED,INT UNSIGNED))"

$ gizzmo --simple -P ADMIN_PORT -H LIST_OF_ALL_APP_HOSTS -T LIST_OF_TABLES rebalance --concrete=com.myapp.shards.AppShard --source-type="INT UNSIGNED" --dest-type="INT UNSIGNED" --copies-per-host=10 --max-copies=50 \
> 1 DB1,DB2,DB3+write_only 1 DB4,DB5,DB6+write_only 1 DB7,DB8,DB9+write_only 1 DBa,DBb,DBc+write_only

The above command would add the set of hosts DBa, DBb, and DBc to our previously created table, and balances the cluster such that all sets of hosts now own 1/4 of all shards. The max-copies and copies-per-host options ensure that your machines don't become so busy with copy work that they can't serve regular traffic. Because of the way copies work in Gizzard, all shards involved in a copy should be included in this number. That means if you're copying from shard_1 to shard_2, you have 2 copies in progress.

Adding new sets of machines to a cluster for capacity is a common task, and having to respecify all existing topologies is cumbersome. Because of this, there is an add-partition command which acts exactly like rebalance, except doesn't require you to specify existing topologies. It is useful to consider sets of machines in a topology as a partition. Many Gizzard clusters have partitions consisting of completely mirrored machines, though at the cost of managerial complexity you may choose to stripe machines across partitions.

This command is the same as the rebalance command above:

$ gizzmo -P ADMIN_PORT -H LIST_OF_ALL_APP_HOSTS -T LIST_OF_TABLES add-partition --copies-per-host=10 --max-copies=50 \
> "ReplicatingShard(1) -> (com.myapp.shards.AppShard(DBa,1,INT UNSIGNED,INT UNSIGNED), com.myapp.shards.AppShard(DBb,1,INT UNSIGNED,INT UNSIGNED), WriteOnlyShard(1) -> com.myapp.shards.AppShard(DBc,1,INT UNSIGNED,INT UNSIGNED))"

$ gizzmo --simple -P ADMIN_PORT -H LIST_OF_ALL_APP_HOSTS -T LIST_OF_TABLES add-partition --concrete=com.myapp.shards.AppShard --source-type="INT UNSIGNED" --dest-type="INT UNSIGNED" --copies-per-host=10 --max-copies=50 \
> DBa,DBb,DBc+write_only

You can add multiple partitions at once. There is also an equivalent remove-partition command.

It may be important to note that adding/removing partitions reweights all topologies evenly. Most Gizzard clusters already weigh all partitions evenly, so this should not be an issue. If you wish to add or maintain unevenly weighted partitions, use the rebalance command.

Repairing shards

*** Shard repairs are currently only supported in FlockDB ***

To help with the eventually consistent model of Gizzard, you may want to occasionally run repairs on shards from Gizzmo. A repair is an action to compare a set of shards that should be identical, identify any missing or outdated data between them, and write in the correct data. Repair is conveniently encapsulated in the copy command, as copies in general (in Gizzard) are a special case of repairing.

Using a forwarding shard, like one of those we saw in the transforms section above, you can identify all of its concrete subshards using the subtree command.

$ gizzmo -P ADMIN_PORT -H APP_HOST subtree localhost/shard_0_0071_replicating
DB1/shard_0_0071
DB2/shard_0_0071
DB3/shard_0_0071
DBx/shard_0_0071_write_only
  DBx/shard_0071

To a run a repair across these shards, simply run:

$ gizzmo -P ADMIN_PORT -H APP_HOST copy DB1/shard_0_0071 DB2/shard_0_0071 DB3/shard_0_0071 DBx/shard_0071

Obviously it would be difficult to run this individually on all shards in the cluster. You can use the repair-tables command to repair whole tables at a time.

$ gizzmo -P ADMIN_PORT -H LIST_OF_ALL_APP_HOSTS -T LIST_OF_TABLES repair-tables --max-copies=40

Transform options

--batch-finish

When used with transforms, the --batch-finish flag causes an alternative plan to be used: the goal of which is to copy all data that is going to move before making destination shards active. The plan is executed as:

  1. Add new shards as blocked
  2. Copy data to new shards and mark them write-only
  3. When all copies have finished, ask the operator whether to move to the next step, and then...
  4. Mark added shards readable
  5. Finally, when the operator confirms that it is safe, remove any shards that are no longer needed

--rollback-log

The --rollback-log=NAME flag causes a rollback log to be written to the nameserver database as individual operations for any transform operation: for example, transform-tree: gizzmo --rollback-log="my-transform" transform-tree .... The log of the transform can be manually rolled back by using the gizzmo log-rollback NAME command, which will roll the transform back to the last safe state. When the flag is in use, it is (theoretically) safe to kill the gizzmo process at any time during a transform, assuming you manually run the gizzmo log-rollback command afterward.

The rollback log works by adding 'commit' entries to the gizzmo plan which indicate positions in the transform which it would be safe to roll back to. In particular, when used with the --batch-finish flag, all of steps 1, 2, 3, and 4 could be rolled back (although the copying portion of step 2 becomes a noop.) When not using the --batch-finish flag, only the last partial shard creation can be rolled back: this means that killing a transform that adds 100 shards during the 25th shard, will cause only the partial effort for shard 25 to be rolled back.