Skip to content
Permalink
master
Switch branches/tags
Go to file
 
 
Cannot retrieve contributors at this time

NOTES FOR DEVELOPERS

These notes are meant for developers working on EvoMaster, and for people making a pull request. There are several rules of thumb regarding how to write "good code", but often rules are either too generic and not tailored for a given particular piece of software (e.g., different kinds of architectures).

The rules of thumb described here in this document are not meant to be either exhaustive nor absolute. Rigid rules are not substitute for common sense, as they are rather guidelines that can be ignored in some special cases. Furthermore, the guidelines need to be realistic and easy to use: there would be no point to ask for detailed comments on each single method/field and 100% coverage test suites...

These notes also include some explanations and motivations for some of the architectural choices made in the development of EvoMaster.

Kotlin vs. Java

The core process of EvoMaster is built in Kotlin, as we strongly prefer it over Java. However, the client libraries for JDK SUTs (e.g., not just Java, but also all other languages that do compile to JDK bytecode) are written in Java instead of Kotlin. The main reason is that, being libraries, we do not want to also have to ship the Kotlin runtime libraries with them.

AVOID System.out AND System.err

EvoMaster uses a logging framework. For debugging and logging errors in a class Foo, create a logger in the following way.

  • for Java: private static Logger log = LoggerFactory.getLogger(Foo.class);
  • for Kotlin: companion object { private val log: Logger = LoggerFactory.getLogger(Foo::class.java)}

It is important to keep the same name log to make things consistent among different classes. If the logging should be part the actual output for the console user, then rather use:

LoggingUtil.getInfoLogger()

AVOID String CONCATENATION IN LOGGERS

Writing something like:

log.debug("this is not "+ foo + " very " + bar +" efficient")

is not efficient, as most of the time debug logs are deactivated, and concatenating strings is expensive. Recall String is immutable, and each + does create a new String object. The above logging can be rewritten into:

log.debug("this is not {} very {} efficient", foo, bar)

Note: not a big deal for warn/error, as those are/should be rare... but it can become quite an overhead for trace/debug/info.

DO NOT USE System.exit

Better to throw an exception, as the entry point of EvoMaster does some logging when ends. Furthermore, System.exit becomes problematic when unit testing EvoMaster.

STATIC VARIABLES ARE YOUR ENEMY

Static variables should be either constant or representing transient data (e.g., cache information whose presence/missing has only effect on performance, not on functionality). Having "classes with static state" is usually a poor OO design (an exception to this rule is ExecutionTracer). If those are really needed, then you should rather use an injectable singleton service (see next point). This is not just to be pedantic, but, really, non-constant static variables make unit testing far much harder and lead to code that is more difficult to understand and maintain.

Guice and Governator

To avoid issues with mutable static variables, we use a dependency injection framework. In particular, we use Guice, extended with Governator to handle post-construct events. All injectable services should be singletons, and declared under a package called *.service (this is to make it easy to find out which services are available).

There is no auto-discovery of beans. This is done manually. The reason is that, depending on configurations, we can have many different context initializations. For example, the beans used for testing REST APIs would not be needed when testing GraphQL ones.

HOW TO WRITE UNIT TEST CASES

Unit tests should be put in the src/test/java and src/test/kotlin folders, following the same package structure as EvoMaster code. A unit test suite for SUT org.evomaster.somepackage.Foo MUST be called org.evomaster.somepackage.FooTest. This is important for several reasons:

  • Need to know what class the test case is supposed to unit test by just looking at its name
  • Should be easy to identify if a class has a test suite for it
  • If in same package, then the test suite can access package/protected fields/methods
  • Having Test as postfix (instead of a prefix) is useful for when searching for classes by name
  • A Test postfix is a requirement for Maven to execute the test suite during the build

HOW TO WRITE END-TO-END (E2E) TEST CASES

Besides unit tests, it is essential to have E2E ones as well. Those should be added under the e2e-tests module. Being non-deterministic, we cannot guarantee that EvoMaster can always find a valid solution (e.g., create test cases with certain properties). Furthermore, we cannot run the E2E tests for long time (otherwise the CI builds will take forever). The idea is to create artificial SUTs that should be trivial to solve when some settings (which we want to test) are on, and very difficult (if not straight-out infeasible) otherwise.

Note: current version of JUnit 5 is worse than JUnit 4 when dealing with E2E tests. E.g., there is no handling of flaky tests (in JUnit 4, this was handled by the Surefire/Failsafe plugins). This is the reason why such test executions should be wrapped inside a handleFlaky call.

AVOID TOO LONG METHODS

Too long methods (e.g., more than 100 lines) should be split, as difficult to understand. For this task, in IntelliJ, you can right-click on a code snippet and choose "Refactor -> Extract -> Function"

WRITE COMMENTS

In the ideal world, each class/method/field would have nice, detailed, appropriate code comments. But even in such a beautiful world, everything would go to hell at the first code change, as that might require manually changing most of the code comments.

Cannot really quantify how much comments one should write, but at least it would be good to have:

  • brief (1-2 sentences) description of what the class is useful for (just before the class declaration)
  • for fields that are data structures (e.g., collections and arrays) some comments would be useful, as long and detailed variable names are not practical
  • for Maps, should add a comment stating what is the key, and what is the value.

When writing a comment for a class/method/field, use JavaDoc style: /** */ In this way, your IDE can show the comments when you hover with the mouse over them.

IF CANNOT AVOID EXTERNAL SIDE-EFFECTS, DO DOCUMENT IT!!!

If a call on a object has side-effects outside the class itself (e.g., writing to disk, add a system hook thread), then this needs to be documented (see point on how to write comments), unless it is obvious from the function/class name.

PRE AND POST CONDITIONS

  • Pre-conditions of public methods should throw exceptions explicitly (e.g., IllegalArgumentException and IllegalStateException). Whenever possible, it is worth to write pre-conditions to public methods.
  • Pre-conditions of private methods and post-conditions (both public and private methods) should use the keyword assert in Java, and the function assert() in Kotlin. (An exception is when the validation of inputs of a public method is delegated/moved to a private method: in this case you could add throw.) Post-conditions are good, but often are difficult to write. Note: a post-condition does not to be complete to be useful (i.e., find bugs). For example, if we have A && B, but the writing of B is too difficult (or time consuming), still having just A as post-condition can help

Note: currently Kotlin does not have lazily evaluated assertions. If you are writing a computational expensive check, rather user Lazy.assert(predicate).

FIELDS/CONSTRUCTORS/METHODS ORDER IN A CLASS

When writing a new class (or re-factoring a current one), fields should come first, followed by class constructors and then the other methods.

NON-DETERMINISM

EvoMaster uses randomized algorithms. Running it twice on the same application can give different results. This is a problem for testing and debugging EvoMaster itself, as for example the test cases will be flaky. To avoid such issues, we must control the source of non-determinism. All randomness sources MUST come from the Randomness class. Some data-structures could lead to non-deterministic behavior (e.g., iteration over a Set does not guarantee the order). This does not seem the case for the default data-structures in Kotlin, but it is definitively a problem in Java, e.g., HashSet vs. LinkedHashSet.

In EvoMaster we do have checks for its determinism. This is achieved by running some E2E tests twice with verbose logging, and then compare the logs for an exact match. If some logs are not deterministic (e.g., printing out for how many seconds the search ran), those should be inside a check for EMConfig.avoidNonDeterministicLogs.

When running EvoMaster on an application, the seed for the random generator is taken from the CPU clock. To make a run deterministic, you will need to use the --seed option to specify a constant seed.

NAMING CONVENTION

We follow the typical naming convention used in Java: class names start in capital letter (e.g., class Foo), whereas we use camel-case for variables and methods (e.g., void fooBar() and String helloWorld;). Constants in Java (but usually not in Kotlin, unless they are global public variables in a companion object) would be typically in upper-case using snake-case (e.g., final String HELLO_WORLD). Kebab-case should be avoided for names of classes/methods/variables (e.g., no String hello-world, which anyway would not compile).

Regarding packages and modules, it is a bit more tricky. In this project, the current rules are the following (but might change if given arguments for a better approach): no dashes - and no upper-case in the package names, but - are fine (and preferable) in module names. For example, org.EvoMaster.foo-bar would be wrong for 2 reasons, which could be fixed with org.evomaster.foobar or org.evomaster.foo.bar. On the other hand, a Maven module called foo-bar would be fine, but not Foo-bar. The motivation here is that modules are mapped to folders on the operating system, and we need to avoid issues with OSs like Windows that are case insensitive, and with . treated as beginning of a file extension.

All code written for EvoMaster must be inside the package org.evomaster.*. Each module must define a subpackage, with a name somehow related to the module itself. Dashes - in the module name would be either stripped or replaced with dots .. For example, a module called controller-api under the module client-java could define a package called org.evomaster.client.java.controller.api. Note that it is imperative that no module defines the same subpackage, as to avoid class name conflicts.

All names should use ASCII letters. Non-ASCII ones like ø or Å must be avoided.

MAVEN MODULE HIERARCHY

EvoMaster is built with Maven, with a hierarchy of submodules. Given a module X declaring a submodule Y with <module> in its pom.xml file, then Y must declare X as parent with <parent>. Do no break the hierarchy by pointing to a parent outside EvoMaster (e.g., something like spring-boot-starter-parent). If you need to use such external poms, you can import them as dependency, i.e., specifying the <scope>import</scope> tag.

When creating a new module, it is also important to add it as a dependency to report, so that aggregated, transitive code coverage can be calculated.

MAVEN DEPENDENCY VERSION

All dependency <version> tags must be declared in the root pom.xml file, in the <dependencyManagement> section. Submodules must not declare a version for a library, and rather refer to the ones in the root using just <groupId> and <artifactId> (but possibly overriding some configurations, like <scope>).

Motivation: must have only a single version of a library in EvoMaster. Specifying versions in submodules can lead to duplicated <version> declarations with different version numbers. All version numbers should be easily audited, and so should be in a single file (i.e., the root pom.xml).

THIRD-PARTY LIBRARIES

Adding a new dependency is fine, but few things to consider:

  • NEVER ever add a GPL licensed library, unless it is under the so called classpath exception. Note that LGPL libraries are fine.

  • When adding a new library, check who is maintaining it, and when was its last update. No longer maintained libraries should be avoided.

THIRD-PARTY CODE

As a rule of thumb, to avoid possible issues with copyrights and license compliance, we should not include code directly from third-party sources. However, when that happens, it MUST be made clear in the files themselves (e.g., with comments in their top, with URLs of the original sources). Furthermore, this information should also be added to the reused_code.md file.

Trello

If you are among the core developers of EvoMaster, you should get an invitation to join Trello. We use it to track activities and assign tasks. Anyone can create new tasks/cards. Current usage:

  • On going: tasks that are currently under development. Those must be assigned to at least 1 person.
  • Done: tasks that are fully done. We do not delete them, e.g., just in case if need to look at them again in the future. Even when a task is completed, the moving from On going to Done should be carried out only during a developer meeting (so it can be demoed or at least discussed). Furthermore, a done task should be added on top of the Done list. In this way, by looking at the top of the list, one can see what were the most recent changes.
  • Important, to do soon: high priority tasks which have not been started yet.
  • Issues/bugs: reported bugs which are not trivial to fix. For developers, better to report them here than GitHub issue page.
  • Backlog-*: different backlogs, divided by topic.

MAKING A NEW RELEASE

Only the project manager should make a new release, as it requires a password. Instructions can be found here.

JDK VERSIONS

At this point, we only support JDK 8 and JDK 11. EvoMaster must be built with JDK 8, but still must be able to run it with JDK 11. Can be useful to setup your machine to easily switch between the 2 versions. For example, if you are using a Mac, in your ~/.profile configuration, you could have something like:

export JAVA_HOME_8=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/ 
export JAVA_HOME_11=/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/ 

export JAVA_HOME=$JAVA_HOME_11 
export PATH=$JAVA_HOME/bin:$PATH

alias java8='$JAVA_HOME_8/bin/java'
alias java11='$JAVA_HOME_11/bin/java'
alias mvn8='JAVA_HOME=$JAVA_HOME_8 && mvn'
alias mvn11='JAVA_HOME=$JAVA_HOME_11 && mvn'