A powerful testing tool for Kubernetes clusters.
Clone or download
Latest commit f5a150a Aug 31, 2018

README.md

PowerfulSeal

PowerfulSeal adds chaos to your Kubernetes clusters, so that you can detect problems in your systems as early as possible. It kills targeted pods and takes VMs up and down.

It follows the Principles of Chaos Engineering, and is inspired by Chaos Monkey.

Embrace the inevitable failure. Embrace The Seal.

PyPI Travis

Watch us introduce the Seal at Kubecon 2017 Austin

Highlights

  • works with OpenStack, AWS and local machines
  • speaks Kubernetes natively
  • interactive and autonomous, policy-driven mode
  • metric collection and exposition to Prometheus
  • minimal setup, easy yaml-based policies
  • easy to extend

Introduction

PowerfulSeal works in three modes: interactive, autonomous and label.

Interactive mode is designed to allow you to discover your cluster's components, and manually break things to see what happens. It operates on nodes, pods, deployments and namespaces.

Autonomous mode reads a policy file, which can contain any number of pod and node scenarios. Each scenario describes a list of matches, filters and actions to execute on your cluster.

Label mode allows you to specify which pods to kill with a small number of options by adding seal/ labels to pods. This is a more imperative alternative to autonomous mode.

Interactive mode

Here's a sneak peek of what you can do in the interactive mode:

demo nodes

demo pods

Autonomous mode

Autonomous reads the scenarios to execute from the policy file, and runs them:

  1. The matches are combined together and deduplicated to produce an initial working set
  2. They are run through a series of filters
  3. For all the items remaining after the filters, all actions are executed

pipeline

Metric Collection

Autonomous mode also comes with the ability for metrics useful for monitoring to be collected. PowerfulSeal currently has a stdout and Prometheus collector. However, metric collectors are easily extensible so it is easy to add your own. More details can be found here.

Writing policies

A minimal policy file, doing nothing, looks like this:

config:
  minSecondsBetweenRuns: 77
  maxSecondsBetweenRuns: 100

nodeScenarios: []

podScenarios: [] 

The schemas are validated against the powerful JSON schema

A full featured example listing most of the available options can be found in the tests.

Label mode

Label mode is a more imperative alternative to autonomous mode, allowing you to specify which specific per-pod whether a pod should be killed, the days/times it can be killed and the probability of it being killed.

Instructions on how to use label mode can be found in LABELS.md.

Setup

Setup includes:

  • pointing PowerfulSeal at your Kubernetes cluster by giving it a Kubernetes config file
  • pointing PowerfulSeal at your cloud by specifying the cloud driver to use and providing credentials
  • making sure that PowerfulSeal can SSH into the nodes to execute commands on them
  • writing a set of policies

These interactions are available:

pipeline

Getting started

PowerfulSeal is available to install through pip:

pip install powerfulseal
powerfulseal --help # or seal --help

Both Python 2.7 and Python 3 are supported.

Demo mode

The main way to use PowerfulSeal is to write a policy file for Autonomous mode which reflects realistic failures in your system. However, PowerfulSeal comes with a demo mode to demonstrate how it can cause chaos on your Kubernetes cluster. Demo mode gets all the pods in the cluster, selects those which are using the most resources, then kills them based on a probability.

Demo mode requires Heapster. To run demo mode, use the --demo flag along with --heapster-path (path to heapster without a trailing slash, e.g., http://localhost:8001/api/v1/namespaces-kube-system/services/heapster/proxy). You can also optionally specify --aggressiveness (from 1 (weakest) to 5 (strongest)) inclusive, as well as --[min/max]-seconds-between-runs.

Testing

PowerfulSeal uses tox to test with multiple versions on Python. The recommended setup is to install and locally activate the Python versions under tox.ini with pyenv.

Once the required Python versions are set up and can be discovered by tox (e.g., by having them discoverable in your PATH), you can run the tests by running tox.

More details in TESTING.md

Read about the PowerfulSeal

FAQ

Where can I learn more about Chaos Engineering ?

We found these two links to be a good start:

How is it different from Chaos Monkey ?

PowerfulSeal was inspired by Chaos Monkey, but it differs in a couple of important ways.

The Seal does:

  • speak Kubernetes
  • offer flexible, easy to write YAML scenarios
  • provide interactive mode with awesome tab-completion

The Seal doesn't:

  • need external dependencies (db, Spinnaker), apart from SSH, cloud and Kubernetes API access
  • need you to setup cron

Can I contribute to The Seal ?

We would love you to. In particular, it would be great to get help with:

Check out our CONTRIBUTING.md file for more information about how to contribute.

Why a Seal ?

It might have been inspired by this comic.