The purpose of this utility is to drain documents from Elasticsearch nodes in an AutoScaling Group.
This will help you do Elasticsearch node replacement while keeping the cluster healthy. This is useful if you want to change the instance type of your nodes, or if you use custom AMIs and need to rollout a new AMI.
Consider the following deployment procedure:
- Start with an AutoScaling Group with Elasticsearch nodes in a cluster
- Create a new AutoScaling Group with Elasticsearch nodes that join the above cluster
- Drain all data off older AutoScaling Group and remove instances from the AutoScaling Group and terminate instances
How does it work?
- Get the list of instances we want to remove from the cluster
- In this case it's an entire AutoScaling Group
- Ask the cluster for the
_id(s) of those instances
- Then, tell the cluster to exclude these nodes from routing, which effectively removes all documents from the nodes
- Loop on these nodes asking the cluster how many documents each node is storing, when one reaches 0 we move on to the next step
- Remove the instance from the AutoScaling Group
- Terminate the instance
- Wait a moment and go back to step 4 and continue until there are 0 instances in the AutoScaling Group
$ gem install elasticsearch-drain
- Create a new AutoScaling Group and populate with the same number of instances as the previous AutoScaling Group
- Run the tool, to start draining:
$ drain asg --asg="test-asg-0" --region="us-east-1" --host="localhost:9200"
- Remove a single node from the cluster
- Drain only mode
Install all dependencies:
gem install bundler bundle install
To enable the tests that will hit the AWS APIs pass
Run test tests (unit and style):
Or on a tight loop with guard:
bundle exec guard
If you need to make a new http request or refresh the fixtures you will need to start a test cluster.
By default the test cluster install is version
1.7.2, this can be changed by setting the
ES_VERSION enviroment variable.
Install and Start the Cluster:
rake elasticsearch:install elasticsearch:start
Run the tests:
Stop the Cluster:
And, to wrap all that up:
- Fork it ( https://github.com/rapid7/elasticsearch-drain/fork )
- Create your feature branch (
git checkout -b my-new-feature)
- Commit your changes (
git commit -am 'Add some feature')
- Push to the branch (
git push origin my-new-feature)
- Create a new Pull Request