Elasticsearch Node Controller with AWS Auto Scaling Group
Go Makefile
Switch branches/tags
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
aws
cmd
es
version Create version package Mar 16, 2017
.gitignore
.travis.yml Install glide explicitly Apr 17, 2017
CHANGELOG.md
LICENSE Initial commit Mar 6, 2017
Makefile Use Glide instead of dep Apr 17, 2017
README.md Write about Es v5 on in README Mar 16, 2017
docker-compose.yml Add docker-compose.yml for local Es development Mar 7, 2017
glide.lock Use aws-sdk-go v1.7 Apr 17, 2017
glide.yaml
main.go

README.md

esnctl

Build Status codecov

Elasticsearch Node Controller with AWS Auto Scaling Group

Why

Graceful Elasticsearch node addition/removal requires several steps.

Add node

  1. Disable shard reallocation
  2. Add node
  3. Enable shard reallocation

Remove node

  1. Remove node from load balancer
  2. Wait for connection draining
  3. Remove node from shard allocation targets
  4. Wait for that shards on target node escape to other nodes
  5. (Es 1.x only) Shut down node

So far we have conducted this by hand. However, it sometimes causes operation errors. We realize that these operations should be automated and conducted by ONE action.

Required environment

  • Elasticsearch 1.x / 2.x / 5.x
  • Elasticsearch cluster is running on AWS EC2 instances
  • EC2 instances are managed by AWS Auto Scaling Groups
    • Instances (= Nodes) can be added/removed by modifying DesiredCapacity
  • EC2 instances and Auto Scaling Group are attached to Target Group
    • Cluster can be accessed through Application Load Balancer

(TODO: architecture image here)

Installation

Precompiled binary

Precompiled binaries for Windows, OS X, Linux are available at Releases.

From source

$ go get -d github.com/dtan4/esnctl
$ cd $GOPATH/src/github.com/dtan4/esnctl
$ make deps
$ make install

Usage

To run esnctl add or esnctl remove, you need to set valid AWS credentials beforehand.

export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
export AWS_REGION=xx-yyyy-0

esnctl list

List nodes

$ esnctl add \
  --cluster-url http://elasticsearch.example.com
ip-10-0-1-21.ap-northeast-1.compute.internal
ip-10-0-1-22.ap-northeast-1.compute.internal
ip-10-0-1-23.ap-northeast-1.compute.internal
Option Description
--cluster-url=CLUSTERURL Elasticsearch cluster URL

esnctl add

Add nodes

$ esnctl add \
  --cluster-url http://elasticsearch.example.com \
  --group elasticsearch \
  -n 2
===> Disabling shard reallocation...
===> Launching 2 instances on elasticsearch...
===> Waiting for nodes join to Elasticsearch cluster...
........................
===> Enabling shard reallocation...
===> Finished!
Option Description
--group=GROUP Auto Scaling Group
--cluster-url=CLUSTERURL Elasticsearch cluster URL
-n, --number=NUMBER Number to add instances
--region=REGION AWS region

esnctl remove

Remove a node

Only 1 node can be removed at the same time.

$ esnctl remove \
  --cluster-url http://elasticsearch.example.com \
  --group elasticsearch \
  --node-name ip-10-0-1-21.ap-northeast-1.compute.internal
===> Retrieving target instance ID...
===> Retrieving target group...
===> Detaching instance from target group...
............................................................
===> Excluding target node from shard allocation group...
===> Waiting for shards escape from target node...
..................
===> Shutting down target node...
===> Detaching target instance...
===> Finished!
Option Description
--group=GROUP Auto Scaling Group
--cluster-url=CLUSTERURL Elasticsearch cluster URL
--node-name=NODENAME Elasticsearch node name to remove
--region=REGION AWS region

Author

Daisuke Fujita (@dtan4)

License

MIT License