Skip to content
A tool for automatically generating system-level test cases
Kotlin Java Other
Branch: master
Clone or download
arcuri82 Merge pull request #128 from EMResearch/time-checks
execution time in progress bar, and fixed memory leak in black-box te…
Latest commit c315d65 Nov 11, 2019
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci increased timeout on CI Oct 20, 2019
client-java 0.4.1 SNAPSHOT Nov 6, 2019
core execution time in progress bar, and fixed memory leak in black-box te… Nov 10, 2019
dbconstraint 0.4.1 SNAPSHOT Nov 6, 2019
docs instructions for making a new release Nov 6, 2019
e2e-tests 0.4.1 SNAPSHOT Nov 6, 2019
experiments 0.4.1 SNAPSHOT Nov 6, 2019
report 0.4.1 SNAPSHOT Nov 6, 2019
resource-rest-experiments 0.4.1 SNAPSHOT Nov 6, 2019
scripts fixed timeout in experiment example script Nov 4, 2019
.gitignore increased test budget for flaky E2E Sep 7, 2019
LICENSE license and readme May 25, 2017
notes.txt Working on instrumentation for exceptions Jun 2, 2017
pom.xml 0.4.1 SNAPSHOT Nov 6, 2019

EvoMaster: A Tool For Automatically Generating System-Level Test Cases

Build Status CircleCI Maven Central

EvoMaster ( is a tool prototype that automatically generates system-level test cases. Internally, it uses an evolutionary algorithm and bytecode analysis to be able to generate effective test cases. The approach is to evolve test cases from an initial population of random ones, trying to maximize measures like code coverage and fault detection.

At the moment, EvoMaster targets RESTful APIs compiled to JVM 8 bytecode (e.g., Java and Kotlin).

This project is in early stage of development. Documentation is still under construction. A short video shows the use of EvoMaster on one of the case studies in EMB.

EvoMaster is currently (2018-2021) funded by a 7.8 million Norwegian Kroner grant by the Research Council of Norway (RCN), as part of the Frinatek project Evolutionary Enterprise Testing.

To compile the project, use the Maven command:

mvn clean install -DskipTests

This should create an evomaster.jar executable under the core/target folder.

Note: if you get an error from the shade-plugin, then make sure to use clean in your Maven commands. Furthermore, if you decide to do not skip the tests, then you will need to have Docker installed and running on your machine.


The following code is an example of one test that was automatically generated by EvoMaster for a REST service called "scout-api" (see EMB). The generated test uses the RestAssured library.

public void test12() throws Exception {
    String location_activities = "";
    String id_0 = given().accept("*/*")
            .header("Authorization", "ApiKey administrator") // administrator
            .body("{\"name\":\"JIg\", \"date_updated\":\"1968-7-28T10:40:58.000Z\", \"description_material\":\"CDasIs\", \"description_prepare\":\"VatRg\", \"description_main\":\"vbhUS\", \"description_safety\":\"mdMZXtHaW6Ac0L7\", \"age_min\":-1639552914, \"age_max\":-546, \"participants_min\":-166, \"time_max\":-728, \"featured\":true, \"activity\":{\"ratings_sum\":-2169794882535544017, \"favourites_count\":2018287764382358555, \"ratings_average\":0.7005221066369205, \"related\":[5230990194698818394, 4025421724722458028, -1291838056, -210322044]}}")
            .post(baseUrlOfSut + "/api/v1/activities")
    location_activities = "/api/v1/activities/" + id_0;
            .header("Authorization", "ApiKey administrator") // administrator
            .body("{\"rating\":7126434, \"favourite\":false}")
            .post(resolveLocation(location_activities, baseUrlOfSut + "/api/v1/activities/-324163273/rating"))
            .header("Authorization", "ApiKey administrator") // administrator
            .delete(resolveLocation(location_activities, baseUrlOfSut + "/api/v1/activities/-324163273/rating"))

The generated tests are self-contained, i.e. they start/stop the REST server by themselves:

    private static SutHandler controller = new;
    private static String baseUrlOfSut;
    public static void initClass() {
        baseUrlOfSut = controller.startSut();
        RestAssured.urlEncodingEnabled = false;
    public static void tearDown() {
    public void initTest() {

EvoMaster Core

Once the file evomaster.jar has been packaged, or the latest release has been downloaded from the releases page, it can be executed directly from a commandline/shell.

Available options can be queried by using:

java -jar evomaster.jar --help

The most important options are:

  • --maxTime <String>
    Maximum allowed time for the search, in the form ?h?m?s, where it can be specified for how many hours (h), minutes (m) and seconds (s) to run the search. For example, 1h10m120s would run the search for 72 minutes. Each component (i.e., h, m and s) is optional, but at least one must be specified. In other words, if you need to run the search for just 30 seconds, you can write 30s instead of 0h0m30s. The more time is allowed, the better results one can expect. But then of course the test generation will take longer.

  • --outputFolder <String>
    The path directory of where the generated test classes should be saved to.

  • --testSuiteFileName <String>
    The name of generated file with the test cases, without file type extension. In JVM languages, if the name contains ., folders will be created to represent the given package structure.

EvoMaster Driver

Note, to generate tests, you need an EvoMaster Driver up and running before executing evomaster.jar. These drivers have to be built manually for each system under test (SUT). See EMB for a set of existing SUTs with drivers.

To build a client driver in Java (or any JVM language), you need to import the EvoMaster Java client library. For example, in Maven:


For the latest version, check Maven Central Repository. The latest version number should also appear at the top of this page. If you are compiling directly from the EvoMaster source code, make sure to use mvn install to install the snapshot version of the Java client into your local Maven repository (e.g., under ~/.m2).

Once the client library is imported, you need to create a class that extends either or Both these classes extend SutController. The difference is on whether the SUT is started in the same JVM of the EvoMaster driver (embedded), or in a separated JVM (external).

The easiest approach is to use the embedded version, especially when dealing with frameworks like Spring and DropWizard. However, when the presence of the EvoMaster client library gives side-effects (although its third-party libraries are shaded, side-effects might still happen), or when it is not possible (or too complicate) to start the SUT directly (e.g., JEE), it is better to use the external version. The requirement is that there should be a single, self-executable uber/fat jar for the SUT (e.g., Wildfly Swarm). It can be possible to handle WAR files (e.g., by using Payara), but currently we have not tried it out yet.

Once a class is written that extends either EmbeddedSutController or ExternalSutController, there are a few abstract methods that need to be implemented. For example, those methods specify how to start the SUT, how it should be stopped, and how to reset its state. The EvoMaster Java client library also provides further utility classes to help writing those controllers/drivers. For example, helps in resetting the state of a database (if any is used by the SUT).

Until better documentation and tutorials are provided, to implement a SutController class check the JavaDocs of the extended super class, and the existing examples in EMB.

Once a class X that is a descendant of SutController is written, you need to be able to start the EvoMaster driver, by using the class. For example, in the source code of the class X, you could add:

public static void main(String[] args){

   SutController controller = new X();
   InstrumentedSutStarter starter = new InstrumentedSutStarter(controller);


At this point, once this driver is started (e.g., by right-clicking on it in an IDE to run it as a Java process), then you can use evomaster.jar to finally generate test cases.

How to Contribute

There are many ways in which you can contribute. If you found EvoMaster of any use, the easiest way to show appreciation is to star it. Issues and feature requests can be reported on the issues page:

  • Bugs: as for any bug report, the more detailed you can be the better. If you are using EvoMaster on an open source project, please provide links to it, as then it is much easier to reproduce the bugs.

  • If you are trying to use EvoMaster, but the instructions in these notes are not enough to get you started, then it means it is a "bug" in the documentation, which then would need to be clarified.

  • Feature Requests: to improve EvoMaster, we are very keen to receive feature requests, although of course we cannot guarantee when they are going to be implemented.

  • Pull Requests: we are very keen to receive PRs, as long as you agree with the license of EvoMaster. However, before making a PR, should read the notes for developers.

Publications and Further Resources

The development of EvoMaster is rooted in academia. Academic publications based on EvoMaster and slides of presentations can be found here. These can be useful if you want to know more on how EvoMaster works internally, e.g., details on the Many Independent Objective (MIO) algorithm.


EvoMaster's source code is released under the LGPL (v3) license. For a list of the used third-party libraries, you can directly see the root pom.xml file. For a list of code directly imported (and then possibly modified/updated) from other open-source projects, see here.

YourKit supports open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of YourKit Java Profiler and YourKit .NET Profiler, innovative and intelligent tools for profiling Java and .NET applications.

You can’t perform that action at this time.