Tools for managing large consistent hashing Graphite clusters.
Buckminster (Bucky) Fuller once said,
If you want to teach people a new way of thinking, don’t bother trying to teach them. Instead, give them a tool, the use of which will lead to new ways of thinking.
Carbon molecules known as fullerenes were also named after Bucky due to their geodesic arrangement of atoms. So, this is my contribution to the Graphite naming scheme.
When working large consistent hashing Graphite clusters doing simple maintenance can involve a lot of data moving around. Normally, one would reach for the Carbonate tools:
These are good tools and I highly recommend them.
However, when the terabytes of WSP files and the number of storage nodes stack up you start to find scaling problems:
- The python implementation of whisper-fill is slow
- There is no atomic method to rebalance a single metric resulting in query inconsistency for possible days after the metric was moved
- Handling renames and resulting backfills at scale is still difficult
So I wanted to use the speed and concurrency of Go to build more efficient tools to help manage large consistent hashing cluster.
These are the tools included and their functionality.
- bucky-pickle-relay -- A daemon that accepts and decodes Graphite's Pickle protocol and will tcp stream the equivalent plaintext version to a carbon-relay. Useful with carbon-c-relay.
- bucky-fill -- A
whisper-fillcompatible utility that is nearly an order of magnitude faster.
- bucky-isempty -- A utility for discovering WSP databases that contain no valid data points.
- buckyd -- A daemon for each Graphite node that tracks the configuration of the hash ring and exposes a REST API for interacting with the raw metric DBs on disk.
- bucky -- Command line Graphite cluster manager. Modules:
- backfill -- Backfill old metrics into new names.
- delete -- Delete metrics via list or regular expression.
- du -- Measure the storage consumed by a list of regular expression of metrics.
- inconsistent -- Find metrics that are stored in the wrong server according to the hash ring.
- json -- Convert newline separated lists to JSON arrays.
- list -- Discover and verify metrics.
- locate -- Calculate metric locations from the hash ring.
- rebalance -- Move inconsistent metrics to the correct location and delete the source immediately after successful backfill.
- restore -- Restore from a tar archive.
- severs -- List each server's known hash ring and verify that all hash rings are consistent.
- tar -- Make an archive of a list or regular expression of metric names and dump it in tar format to STDOUT.
- gentestmetrics -- Command that generates random Graphite style metrics to stdout purely for testing.
The heavy lifting commands use a set of worker threads to do IO work which can be set at the command line with -w.
These tools assume the following are true:
- You Graphite carbon-cache servers have one Whisper DB store. Not multiple mount points with carbon-cache configured with their own DB store.
- Your hash ring is set to a
These aren't set in stone, just what I was working with as I built the tool. I very much hope that some of these will be solved with further development.
Each data storage node in the Graphite cluster needs a buckyd daemon running as the same user as the other Graphite tools. I use an Upstart job to keep mine running. The important bit here is that you must pass to the daemon as arguments the members of the consistent hash ring.
$ cat /etc/init/buckyd.conf description "Buckyd Daemon for Managing Graphite Clusters" author "Jack Neely <email@example.com>" setuid graphite exec /path/to/buckyd --node graphite010-g5 \ -b 192.168.1.1:5678 \ graphite010-g5:a graphite010-g5:b \ graphite011-g5:a graphite011-g5:b \ graphite012-g5:a graphite012-g5:b
--node is the name of this Graphite node (if different from what
is derived from the host name).
--bind is the address to bind
to. You can also specify
--prefix`` where your Whisper data store is and --tmpdir` where the daemon can write temporary files.
This exposes a REST API that is documented in REST_API_NOTES.md.
The bucky tool is self documenting. You can run:
to see a list of modules and available flags and what arguments are needed. Detailed help is available by specifying a module name:
bucky help backfill
Most commands need a
-h flag to specify the initial Graphite
host to connect to where the client will discover the entire hash ring.
You can also set the
BUCKYHOST environment variable rather than
specify this flag for each command.
Other common flags are:
-sOperate only on the initial Graphite host.
-fRequests the remote daemon to refresh its cache of local metrics.
-jRead from STDIN or dump to STDOUT JSON data rather than text.
-rRegular expression mode.
-wNumber of worker threads.
Rebalance a cluster with newly added storage nodes:
GOMAXPROCS=4 bucky rebalance -h graphite010-g5:4242 \ -w 100 2>&1 | tee rebalance.log
Discover the exact storage used by a set of metrics:
export BUCKYHOST=-h graphite010-g5:4242 bucky du -r '^1min\.ipvs\.'
To Do / Bugs
Authentication -- Negotiate and Kerberos support. Probably Basic as well.
Code clean up -- I wrote a lot of this quickly, it needs love in a lot of places.
Make all modules aware of possible duplicate metrics.
Speed testing and improvements.
Move the lower level GET, POST, DELETE, HEAD functions into a single common file/place.
Rebalance needs to optionally be aware of machines not in the hash ring that the rebalance should vacate.
Graceful restarts and shutdowns? https://github.com/facebookgo/grace
graphite-project/carbon's master branch contains this change:
This will cause a few metrics to be assigned a different position in the hash ring. We need to account for this algorithm change somehow.