termination-handler handles termination notices on spot/preemptible instances.
As an example, if deployed on a Kubernetes cluster and a termination notice is issued by the cloud provider,
termination-handler drains the node it is running on before the node is taken away by the cloud provider.
Inspired by pusher/k8s-spot-termination-handler for AWS,
termination-handler surges from the need to operate the same tooling in various distinct environments (providers, orchestrators...)
- Supports multiple cloud providers (AWS, GCP).
- Supports multiple handlers (Kubernetes, Nomad, Slack).
- Small and extensible.
Deploy to Kubernetes
To deploy in clusters using RBAC, please apply all of the manifests (Daemonset, ClusterRole, ClusterRoleBinding and ServiceAccount) in the deploy/k8s folder but uncomment the
serviceAccountName in the Daemonset.
termination-handler to schedule correctly; you will need an identifying label on your spot/preemptible instances.
We add a label
node-role.kubernetes.io/spot-worker to our spot/preemptible instances and hence this is the default value in the node selector of the Daemonset.
nodeSelector: "node-role.kubernetes.io/spot-worker": "true"
To achieve this, add the following flag to your Kubelet:
To define any custom parameters to the drain command you can use
DRAIN_PARAMETERS environment property. If not defined, default parameters are
--grace-period=120 --force --ignore-daemonsets.
env: - name: DRAIN_PARAMETERS value: '--grace-period=120 --force --ignore-daemonsets --delete-local-data'
Deploy to Nomad
A Nomad docker image is available at
The main way to use termination-handler is waiting for the termination notice from the cloud provider. However, termination-handler comes with a demo mode that is can simulate the notice. When deployed it will identify your cloud provider and run your handlers.
To activate termination-handler demo mode on Kubernetes, you can use
DEMO_TERMINATION_HANDLER environment property.
env: - name: DEMO_TERMINATION_HANDLER value: True
Sends a notification message to a specific channel when a termination notice is issued.
How to contribute
- Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug.
- Fork the repository on GitHub to start making your changes to the master branch (or branch off of it).
- Write a test which shows that the bug was fixed or that the feature works as expected.
- Send a pull request and bug me until it gets merged and published.
Some things that would be great to have:
- Support for Azure cloud provider.
- Support for Execution handler (certain command or task)
- Support for notifications (Datadog..)
- Option to omit selected cloud provider discovery.