amoc is a simple tool for running massively parallel XMPP tests
Erlang Makefile Shell
Clone or download
michalwski Merge pull request #66 from esl/enhance_docker
Change Docker image building process
Latest commit fd9a092 May 12, 2018
Permalink
Failed to load latest commit information.
ansible Some takeovers from #50 Apr 6, 2018
docker
priv
scenarios fix dialyzer warnings for simple_rest scenario Nov 7, 2016
src Fix compiler warnings Apr 4, 2018
test Use dialyzer via Makefile only Apr 9, 2018
.dockerignore Copy code to Doker context while building image instead of cloning May 10, 2018
.gitignore Extend .gitignore Sep 16, 2016
.travis.yml Split test and dialyzer targets in travis Apr 9, 2018
Dockerfile Change ownership of release using RUN May 11, 2018
LICENSE Initial commit Jan 31, 2015
Makefile Enhance readability in Makefile Apr 9, 2018
README.md Make container names more verbose in README.md examples May 11, 2018
REST_API_DOCS.md Fix misleading content in REST API doc Sep 15, 2016
ansible.cfg fix hosts comments and make ansible skip host checking via ansible.cfg Jul 13, 2015
hosts Include file descriptor limit in hosts file Apr 29, 2016
rebar.config warn about missing specs (again) Apr 6, 2018
rebar.config.script Initial commit Jan 31, 2015
rebar.lock Update jiffy to hopefully fix NIF issues Apr 5, 2018
rebar3 Introduce rebar3 Apr 4, 2018
run.sh Some takeovers from #50 Apr 6, 2018

README.md

A Murder of Crows Build Status

A Murder of Crows, aka amoc, is simple tool for running massively parallel XMPP tests. It can be used to load test ESL's MongooseIM.

It uses escalus, the Erlang XMPP client library.

Developing a scenario

All files related to scenarios should be placed in the scenarios directory.

A scenario specification is just an Erlang module that exposes the necessary callback functions.

A typical scenario file will look like this:

-module(my_scenario).

-export([init/0]).
-export([start/1]).

init() ->
    %% initialize some metrics
    ok.

start(Id) ->
    %% connect user
    %% fetch user's history
    %% send some messages
    %% wait a little bit
    %% send some messages again
    ok.

The init/0 function will be called only once per test run, at the very beginning. It can be used for setting up necessary (global) state: metrics, database connections, etc.

The start/1 function describes the actual scenario and will be executed for each user, in the context of that user's process.

Id is the given user's unique integer id. After this function returns, it will be executed again following some delay (60 seconds by default).

For developing XMPP scenarios, we recommend the esl/escalus library.

If additional dependencies are required by your scenario, a rebar.config file can be created inside the scenario dir and deps from that file will be merged with amoc's deps.

Running a scenario

Locally

Everything you need to do is to create the release. To achieve that run: make rel. Now you are ready to test our scenario locally with one amoc node, to do this run _build/default/rel/amoc/bin/amoc console.

amoc_local:do(my_scenario, 1, 10).

Start my_scenario spawning 10 amoc users with IDs from range (1,10) inclusive.

amoc_local:add(10).

Add 10 more user sessions.

amoc_local:remove(10, [{force,true}]).

Remove 10 users.

NOTE: We advise using this mode for the scenario debugging purposes - everything you log using lager is going to be visible in the erlang shell. In this mode the annotations are not working, however this may change in the future.

In a distributed environment

Starting a scenario on multiple nodes is slightly more difficult. You need a number of machines that will be actually running the test (slaves) and one controller machine (master, which might be one of the test nodes).

Moreover, you need a machine with both Erlang and ansible installed in order to do the following:

  1. Edit the hosts file and provide all required information - more information in the file itself.
  2. Run make prepare to configure the slave nodes, you don't need to do that each time you want to run a load test.
  3. Run make deploy in order to build and deploy the release across all nodes. Then go to the master's node and start amoc by executing ~/amoc_master/bin/amoc console, that will run the amoc shell, wait a couple of seconds till all slave nodes are started.

Now instead of amoc_local use amoc_dist - this will tell amoc to distribute and start scenarios on all known nodes (except master).

amoc_dist:do(my_scenario, 1, 100, Opts).

Start my_scenario spawning 100 amoc users with IDs from range (1,100) inclusive. In this case sessions are going to be distributed across all nodes except master. At this moment users are distributed in round-robin fashion - user1 goes to slave1, user2 goes to slave2, ...., user10 goes to slave10, user11 goes to slave1 etc (with the assumption that there are 10 slave nodes configured). We are planning to make the strategy configurable.

Opts may contain:

  • {nodes, Nodes} - list of nodes on which scenario should be started, by default all available slaves
  • {comment, String} - comment displayed used in the graphite annotations
amoc_dist:add(50).

Add 50 more users to the currently started scenario.

amoc_dist:remove(50, Opts).

Remove 50 sessions.

Where Opts may contain:

  • {force, true} - immediately kill the processes responsible for user connections
  • {force, false} (default) - stop user processes gently - do not start them again

Don't stop scenario on exit

There is one problem with the bin/amoc console command. When you exit the Erlang shell the scenario is stopped (in fact the erlang nodes are killed). To prevent that start the amoc node in the background using bin/amoc start. Now you can run commands by attaching to the amoc's node with bin/amoc attach, typing a command and then pressing Ctrl+D to exit the shell. After that the scenario will keep running.

Configuration

amoc is configured through OTP application environment variables that are loaded from the configuration file, operating system environment variables (with prefix AMOC_) and Erlang application environment variables (priv/app.config).

Internally, amoc is using the following settings:

  • interarrival - a delay in ms, for each node, between creating process for two consecutive users. Defaults to 50 ms.
  • repeat_interval - a delay in ms each user process waits before starting the same scenario again. Defaults to 1 minute.
  • repeat_num - number of scenario repetitions for each process. Defaults to infinity.

You can also define your own entries that you might later use in your scenarios.

The amoc_config is a module for getting configuration. Every time we ask for a config value, amoc_config looks into OS environment variables first (with AMOC_ prefix; e.g. if we want to set interarrival by this mechanism we should set OS environment variable AMOC_interarrival), and if it doesn't find it there, it tries to get it from the Erlang application environment variables. If it cannot find it there either and the default value was not supplied, an error it thrown. What's more, we can set variables dynamically (by setting OS variable or application:set_env(amoc, VARIABLE_NAME, VARIABLE_VALUE) in Erlang).

amoc_config provides the following function:

  • get(Name) and get(Name, Default) - return the value for the given config entry according to the aforementioned rules.

Locally (without ansible)

If you are not using ansible, just modify the priv/app.config file. It will later be added to your local release.

Distributed (with ansible)

During the ansible deployment the following file is used as a template: ansible/roles/amoc/templates/app.config.j2.

As long as you need to set only the amoc application variables (including your scenario-specific settings), you can do it in the ansible/group_vars/amoc file. You can also do it later in Erlang console (application:set_env/3) or set OS environment variable (with prefix AMOC_). Remember that OS environment variable will be taken if present even if you set it by application:set_env/3.

A separate file is required for this since dictionaries are not supported in the inventory file.

REST API

There is available REST API for AMOC where we can:

  • retrieve test status from node
  • check whether AMOC cluster is up or down
  • upload source code of scenario to run
  • list available scenarios
  • run scenario
  • check status of nodes in AMOC cluster

API is described here. You can also get an access to REST API by running AMOC and go to (address:port)/api-docs.

Docker

To build a Docker image with Amoc, run the following command from the root of the repository:

docker build -f Dockerfile -t amoc_image:tag .

It is important to start building at project root (it is indicated with dot . at the end of command). It will set build context at the project root. Dockerfile commands expects a context to be set like that:

  • it copies current source code into container to compile it.
  • It looks for files in docker/ relative path.

When image is ready it may be started just with

docker run -d --name amoc_container amoc_image:tag

However, you may want to use Amoc HTTP API for uploading and starting scenarios. In this case port 4000 should be published.

docker run -d -p 4000:4000 -name amoc_container amoc_image:tag

Amoc logs are connected with container standard output, so it is possible to get Amoc logs from running container with:

docker logs amoc_container

Amoc is able to report metrics to Graphite. It is possible to set up reporting endpoint by passing following envromental variables:

docker run -d \
           -e AMOC_GRAPHITE_HOST=192.168.0.1 \
           -e AMOC_GRAPHITE_PORT=2003 \
           --name amoc_container \
           amoc_image:tag

Useful commands with Amoc container:

Starting Amoc remote_console:

docker exec -it amoc_container /home/amoc/amoc/bin/amoc remote_console