Skip to content

Gracefully kill Mesos nodes in AWS for autoscaling and inmutable deployments

License

Notifications You must be signed in to change notification settings

alanbover/deathnode

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

56 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Deathnode

Build Status Go Report Card Coverage Status

Purpose

Gracefully Mesos agent destroy system for AWS Autoscaling.

Deathnode ensures that agents from an Autoscaling group in AWS are destroyed only after they have being drained, allowing to implement scale-in or red/black deployments with no impact to the customers.

It's implementation is based using:

How does it do it?

Deathnode monitors the autoscaling groups from the Mesos Agents. Whenever it detects that one instance from an autoscaling group should be removed, it will:

  • Find the best agent to be killed
  • Tag the instance as being deleted
  • Remove instance protection from the instance
  • Set the instance in maintenance mode

Then deathnode will keep monitoring this agent, completing destroy lifecycle once it's drained.

Usage

Here you can find an example of usage:

./deathnode -autoscalingGroupName ${ASG_NAME} -delayDelete 300 -mesosUrl ${MESOS_URL} -polling 60 -protectedFrameworks Eremetic -debug

Constraints

When removing an instance, contraints are used by deathnode to filter which instances are not able to be picked up as candidates (best efford). Multiple contraints can be specified.

  • noContraint: Applies no constraints
  • protectedConstraint: Do not pick instances that has tasks from protected frameworks
  • filterFrameworkConstraint: Do not pick instances that has tasks from the specified framework
  • taskNameRegexpConstraint: Do not pick instances that has tasks that it's name match a certain regexp

Build

To execute the test, run:

make test

To build the app, run:

make build

To build a docker image, run:

DOCKERTAG="version" make docker

Limitations

  • Most of the Mesos frameworks doesn't implement Maintenance primitives.
  • Maintenance primitives are not available through Mesos native integration

About

Gracefully kill Mesos nodes in AWS for autoscaling and inmutable deployments

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published