Litmus is chaos engineering for stateful workloads on Kubernetes, hopefully without learning curves
Switch branches/tags
Nothing to show
Clone or download
dargasudarshan and gprasath Litmus book to run memory utilization test on OpenEBS volume (#309)
Signed-off-by: Sudarshan <sudarshan.darga@cloudbyte.com>
Latest commit 6d6bff0 Oct 16, 2018
Permalink
Failed to load latest commit information.
.github [Added] PULL_REQUEST_TEMPLATE.md Jul 3, 2018
apps Litmus book to run memory utilization test on OpenEBS volume (#309) Oct 16, 2018
chaoslib Include block to drain/taint based chaoslib (#278) Oct 15, 2018
common/utils Renamed the util file names Sep 14, 2018
docs makes the contents inside readme more generic (#43) May 9, 2018
executor refactor(yaml): Cleanup executor code in litmus Sep 10, 2018
funclib Litmusbook to create openebs snapshot and restore the volume (#290) Oct 11, 2018
hack feat(install): setup litmus-prerequisites on cluster Sep 11, 2018
images Reorder the Build Status tag to the top of the readme file. Jun 20, 2018
k8s Added the Node type in variable for eks cluster create playbook (#310) Oct 15, 2018
pkg Create unit test for pkg/time/time.go (#306) Oct 16, 2018
providers/openebs/installers refactor(yaml): update openebs provider setup from master to take ver… Oct 11, 2018
tools refactor(dockerfile): include aws-iam-authenticator tool in ansible-r… Oct 15, 2018
vendor add godog runner tools and a sample test Apr 26, 2018
.gitignore Create unit test for pkg/time/time.go (#306) Oct 16, 2018
.travis.yml makes the contents inside readme more generic (#43) May 9, 2018
CODE_OF_CONDUCT.md Create CODE_OF_CONDUCT.md Apr 27, 2018
CONTRIBUTING.md Create CONTRIBUTING.md Apr 27, 2018
Gopkg.lock add godog runner tools and a sample test Apr 26, 2018
Gopkg.toml add godog runner tools and a sample test Apr 26, 2018
LICENSE fix(licence): Update LICENCE fields Oct 3, 2018
Makefile fix(ci): update source for got lint in build Oct 15, 2018
NOTICE.md Create NOTICE.md Apr 27, 2018
README.md Small improvements to Readme (#296) Oct 14, 2018
Vagrantfile restructure the repo contents Apr 25, 2018
master.yaml fix(yaml): fix the elements in app_tags list Sep 12, 2018

README.md

Litmus

Build Status Docker Pulls

Litmus is chaos engineering for stateful workloads on Kubernetes -> hopefully without learning curves. Our vision includes enabling end users to quickly specify needed scenarios using English descriptions.

https://www.openebs.io/litmus

Overview

The primary objective of Litmus is to ensure a consistent and reliable behavior of workloads running in Kubernetes. It also aims to catch hard-to-test bugs and unacceptable behaviors before users do. Litmus strives to detect real-world issues which escape during unit and integration tests.

While Litmus tests and metrics were developed initially to test if a given Kubernetes deployment is suitable for running on OpenEBS (a kubernetes dynamic storage provisioner); the use cases are broader and overall system resilience can be characterized before and during operations. To learn more about OpenEBS please visit: www.openEBS.io

How Litmus is different from others

Litmus is an overall project that incorporates pieces of a typical chaos engineering environment to deliver a more complete solution to Litmus users.

Also, Litmus incorporates some innovations in translating end user stories directly into scenarios. Litmus accepts user stories in simple English text & converts them to logic. Litmus translates each statement present in a user story into corresponding Kubernetes commands. This provides a transparent view to the users if any particular statement was executed successfully or resulted in failures.

Additionally, test logic is packaged as dedicated containers which of course makes them portable across Kubernetes deployments. This containerization also helps to integrate these containers into CI/CD environments.

There are other aspects to Litmus which are discussed:

Demo: Minio Deployment Using OpenEBS as Persistent Storage

Pre-Requisites for running a specific Test

Users have a Kubernetes environment with a given stateful workload and underlying storage and would like to test a specific scenario:

  • Ensure that the desired storage operators are actually available on a given Kubernetes cluster.

  • Clone the Litmus repo and setup a dedicated RBAC for Litmus.

    git clone https://github.com/openebs/litmus.git
    cd litmus
    kubectl apply -f hack/rbac.yaml
    
  • Create a configmap from the cluster's in-cluster-config (kubeconfig) file with the data placed in "admin.conf". Typically, this file is located at ~/.kube/config or /etc/kubernetes/admin.conf etc. depending on the type of cluster setup. To perform this step, copy the kubeconfig file into "admin.conf" (if it is named differently) and execute the following command:

    kubectl create configmap kubeconfig --from-file=<path/to/admin.conf> -n litmus
    
  • The tests are categorized based on application workloads, with different aspects/use-cases of the application constituting a separate test. Select a workload and follow the instructions under the corresponding <workload>/<usecase>/README.

    For example, to run a MySQL benchmarking test:

    cd apps/percona/tests/mysql_storage_benchmark/
    <Modify the PROVIDER_STORAGE_CLASS in run_litmus_test.yaml>
    kubectl create -f run_litmus_test.yaml
    

    The above test runs a Kubernetes job that:

    • Verifies that the StorageClass mentioned (default: OpenEBS) is loaded in the cluster
    • Launches mysql application with storage
    • Runs a sample TPC-C benchmark against mysql application
    • Provides the benchmark results
    • Reverts system state/performs clean-up by removing deployments launched during the test

As the test ends, the logs of the various storage pods, including the test results of this Kubernetes job are collected and saved in a temporary location. The run_litmus_test.yaml can be customized for the location for saving the logs, type of storage (StorageClass) to be used, etc. This type of deployment test can be used to accelerate the feedback loop when deploying new pieces of a stack, whether underlying cloud or hardware, network, storage, or other.

  • Notes:

    • To run the test, please ensure kubectl create is used as against kubectl apply as the test job uses the generateName API to autogenerate the job name. This is to ensure a re-run of the job w/o deleting the previous instance doesn't throw an error.

    • To delete the job, use the kubectl delete job <jobname>. Deletion using -f spec can complain about non-matching name resources.

Ways to Contribute

Litmus is in alpha stage and needs all the help you can provide to have it cover the ever-growing Kubernetes landscape. Please contribute by raising issues, improving the documentation, contributing to the core framework and tooling, etc.

Another significant area of contribution is for you to describe your experiences/scenarios of running Stateful workloads in your Kubernetes Environment. For example, you can describe feature or scenarios for a new workload or update the scenarios of existing workload as follows:

Feature: MySQL services are not affected due to node failures.
  I need to have at least 3 nodes in my Cluster.
  I need to have enabled Storage solution that supports accessing volume from different nodes.
  I need to have my MySQL running on a persistent volume.
  I need to have MySQL running even when 33% of volume nodes are unavailable.

  Scenario: Node hits an OutOfMemory condition and becomes unresponsive.
    Given I have a Kubernetes cluster with StorageClass installed.
    Given I have a “MySQL” service running and MySQL-client access it from a different node.
    Then I launch memory hog pod on the node where “MySQL” service is running,
    Then wait for "60s",
    And verify MySQL-client can still access data.

For more details on contributing, please refer to CONTRIBUTING.md

Reference Projects

Litmus makes use and extends several open source projects. Below are just some of the most commonly used projects.

For a full list, please checkout the tools directory.

License

Litmus is licensed under the Apache License, Version 2.0. See LICENSE for the full license text. Some of the projects used by the Litmus project may be governed by a different license, please refer to its specific license.