This is very early work in progress on chaos testing utilities for Eventuate and Apache Cassandra. They support running Cassandra clusters and Eventuate applications with Docker and using blockade to easily generate failures like stopping and restarting of containers and introducing network failures such as partitions, packet loss and slow connections.
This repository can be seen as a toolkit or collection of utilities which you can use to test your Eventuate applications. Moreover we are going to describe an examplary test setup that gives an introduction into these tools and serves as a blueprint to build your own more complex test scenarios.
The test scenarios we are about to create mainly facilitate the blockade tool. Blockade is a utility for testing network failures and partitions in distributed applications. Blockade uses Docker containers to run application processes and manages the network from the host system to create various failure scenarios. Docker itself is a container engine that allows you to package and run applications in a hardware-agnostic and platform-agnostic fashion - often called like somewhat lightweight virtual machines.
You can find further information on the blockade github page regarding its configuration and the possibilities you have in addition to what is mentioned in here.
- Docker (tested with docker >= 1.6)
- blockade (currently a fork of the original dcm-oss/blockade)
Given your Docker daemon is running and you have blockade in your $PATH you are basically ready to go. All
you have to do once is to pull or build the necessary Docker containers the test cluster is based on:
# cassandra image
$ docker pull cassandra:2.2.3
# sbt + scala + java8 base image
$ docker build -t eventuate-chaos/sbt .After that is done you can start the minimal docker DNS container that is used to identify the containers with each other:
$ ./start-dns.sh
This script will look for the default docker0 network interface and start a dnsdock image on port 53 for DNS
discovery. You may have to specify the location to your docker socket via $DOCKER_SOCKET in case the script does not
find it by itself:
$ DOCKER_SOCKET=/var/run/docker.sock ./start-dns.sh
These steps only have to be taken once for the initial bootstrapping.
- Vagrant (tested with 1.7.4)
- VirtualBox
As Docker does not run natively on Mac OS you have to use an existing docker machine
setup and follow the steps under Linux or use the pre-configured Vagrant image that ships with this
repository. If you choose the latter you just have to build the image by running vagrant up and ssh into your new
machine:
$ vagrant up
$ vagrant ssh
# inside your virtual machine you can find the repository contents
# mounted under /vagrant as usual
cd /vagrantThe examplary test setup we are describing in the following consists of 2 basic components:
-
Eventuate test appliation:
This is the Eventuate application we are actually testing with. It is an EventsourcedActor that continually (every 2 seconds) emits an event that increments an internal counter by persisting it changes. You can find its implementation in ChaosActor.scala. The container which the scala application is running in is named
location-1. -
Cassandra cluster:
This cassandra cluster contains 3 nodes each running in its own Docker container. The seed node
c1is the initially started one the remaining nodesc2andc3connect with.
The test setup described above is encoded in the blockade.yml YAML configuration file that is used by
blockade to manage the test cluster:
containers:
c1:
image: cassandra:2.2.3
c2:
image: cassandra:2.2.3
start_delay: 60
links:
c1: "c1"
environment:
CASSANDRA_SEEDS: "c1.cassandra.docker"
c3:
image: cassandra:2.2.3
start_delay: 60
links:
c1: "c1"
environment:
CASSANDRA_SEEDS: "c1.cassandra.docker"
location-1:
image: eventuate-chaos/sbt
command: ["test:run-main com.rbmhtechnology.eventuate.chaos.ChaosActor -d"]
volumes:
"${PWD}": "/app"
links: [ "c2", "c3" ]
environment:
CASSANDRA_NODES: "c1.cassandra.docker,c2.cassandra.docker,c3.cassandra.docker"Now that you have all prerequisites fulfilled you can start up the blockade test cluster as it is configured
in the blockade.yml shipped with this repository. From this point on you may freely experiment and modify all given
parameters and configurations as this is just a toolkit to get you started as quickly as possible.
Note that all blockade commands require root permissions.
$ sudo blockade upDepending on the cassandra version (> 2.1) you have to configure a reasonable startup delay for every cassandra node
like 60 seconds (see the
documentation for
further information). This may increase the initial startup time a lot. Once all nodes are up and running you can
inspect your running cluster via blockade status:
$ sudo blockade status
NODE CONTAINER ID STATUS IP NETWORK PARTITION
c1 8ccf90e8f76e UP 172.17.0.3 NORMAL
c2 5c65f3ca62ec UP 172.17.0.5 NORMAL
c3 4b630db6a19e UP 172.17.0.4 NORMAL
location-1 bb8c89f615ef UP 172.17.0.6 NORMALFrom now on you may introduce any kind of failure the blockade tool supports.
$ sudo blockade partition c2
NODE CONTAINER ID STATUS IP NETWORK PARTITION
c1 8ccf90e8f76e UP 172.17.0.3 NORMAL 2
c2 5c65f3ca62ec UP 172.17.0.5 NORMAL 1
c3 4b630db6a19e UP 172.17.0.4 NORMAL 2
location-1 bb8c89f615ef UP 172.17.0.6 NORMAL 2As you can see in the PARTITION column of the command output all cassandra nodes are separated from each other while
the test application (location-1) can reach the cassandra node c2 only.
$ sudo blockade partition c1 c3
NODE CONTAINER ID STATUS IP NETWORK PARTITION
c1 8ccf90e8f76e UP 172.17.0.3 NORMAL 1
c2 5c65f3ca62ec UP 172.17.0.5 NORMAL 3
c3 4b630db6a19e UP 172.17.0.4 NORMAL 2
location-1 bb8c89f615ef UP 172.17.0.6 NORMAL 3$ sudo blockade flaky location-1
NODE CONTAINER ID STATUS IP NETWORK PARTITION
c1 8ccf90e8f76e UP 172.17.0.3 NORMAL 1
c2 5c65f3ca62ec UP 172.17.0.5 NORMAL 3
c3 4b630db6a19e UP 172.17.0.4 NORMAL 2
location-1 bb8c89f615ef UP 172.17.0.6 FLAKY 3You may also restart one or multiple nodes and inspect the effect on the Eventuate application as well:
$ sudo blockade restart c2 c3
While playing with the conditions of the test cluster you can see the Eventuate application output its current state
(being the actual counter value) on stdout. This is my personal preference but I like to inspect the current Eventuate
node's status in a tmux split window while I modify the cluster's condition with tmux split-window "docker logs -f location-1":
counter = 65 (recovery = false)
counter = 66 (recovery = false)
persist failure 696: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive)
persist failure 697: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive)
counter = 67 (recovery = false)
...
counter = 74 (recovery = false)
counter = 75 (recovery = false)
persist failure 698: Cassandra timeout during write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write)
counter = 76 (recovery = false)
counter = 77 (recovery = false)
Just to recapture the current setup, we are dealing with a cassandra cluster of 3 nodes and the Eventuate test
application is configured to initialize its keyspace with a replication-factor of 3 (you may find further information
on data replication in the
cassandra
documentation).
As in this example read and write operations are processed on a QUORUM consistency level (see
documentation) persisting an
event may only succeed if at least 2 of 3 cassandra nodes are reachable from the Eventuate application.
Please keep in mind that any of the following failure scenarios may take a while until the state between application and the cassandra cluster has settled to a stable condition (i.e. reconnect timeouts ...).
Consequently we expect the application to be able to persist any event while any or no cassandra node is partitioned
from the test application container. You may inspect the state of the application with docker logs -f location-1 like
mentioned above or trigger a health check via the ./interact.py python script:
# on success the current state counter of the 'ChaosActor' is written to stdout
$ ./interact.py
250
# the check script exits with a non-zero status code on failure
$ echo $?
0The health check of the interact.py script instructs the test application to persist a special HealthCheck event and
returns with the current counter state if the persist of that given event succeeded within 1 second. The script
basically sends a minor command ("persist") via TCP on port 8080 - so you may achieve the same result via telnet as
well.
On the contrary the Eventuate application must not be able to persist an event while only a minority of the cassandra nodes is available to itself. This happens as soon as you partition a combination of any two nodes like:
# this command creates a partition of c1 and c3 nodes leaving
# the test application with only one reachable node (c2) behind
$ sudo blockade partition c1,c3As soon as you remove all existing partitions or give the test application access to at least 2 cassandra nodes the event persistence should pick up its work again:
# remove all existing partitions
$ sudo blockade joinJust to give a small example on how you might automate such tests you can find a very simple shell script in scenarios/simple-partitions.sh that checks for the expectations described above:
$ scenarios/simple-partitions.sh
*** checking working persistence with 1 node partition each
partition of cassandra 'c1'
waiting 40 seconds for cluster to settle...
checking health...
partition of cassandra 'c2'
waiting 40 seconds for cluster to settle...
checking health...
partition of cassandra 'c3'
waiting 40 seconds for cluster to settle...
checking health...
*** checking non-working persistence with 2 node partitions
partition of test application + 'c1'
waiting 40 seconds for cluster to settle...
checking health...
waiting till cluster is up and running...
partition of test application + 'c2'
waiting 40 seconds for cluster to settle...
checking health...
waiting till cluster is up and running...
partition of test application + 'c3'
waiting 40 seconds for cluster to settle...
checking health...
waiting till cluster is up and running...
*** checking final reconnect
test successfully finishedBelow you can find some random remarks on problems or issues you may run into while chaos testing with the aforementioned methods and tools.
As mentioned before we are using the docker image tonistiigi/dnsdock in order to establish a lightweight DNS service
among the docker containers. This is especially useful when containers are referencing each other in a fashion that
would form a circular dependency of docker links.
You may even integrate this DNS docker service into your host's DNS so that you can reference your containers by name.
Usually you just have to add the docker interface's IP to your /etc/resolv.conf for that to work.
This docker container can be started with the start-dns.sh script and is reachable under the name dnsdock. This
means you can observe its current status via
$ docker logs dnsdock
In case your are using cassandra version > 2.1 (i.e. 2.2.3 like our example) you have to delay the startup of the
cassandra nodes so that the node topology has enough time to settle in. You may alternatively test with an older version
like 2.1.6 and get rid of those start_delay values in your blockade.yml resulting in much faster startup times. If
that makes sense depends completely on what you intend to test. The cassandra
documentation provides
further information on that issue.
While you are using blockade as your 'manager' of your test cluster you are completely free to use other
docker containers in a normal docker-fashion like you are used to. However you should not start/stop/restart
containers of your blockade cluster using the usual docker <cmd> commands. As blockade has to keep track
of the network devices each container is associated with you should use the respective blockade commands like blockade start, blockade stop, blockade restart, blockade up and blockade destroy instead.