Skip to content
Permalink
Browse files

adding notes to developers

  • Loading branch information...
arcuri82 committed Apr 1, 2019
1 parent 39ccc7d commit 0f6c8a4c3e6fa5bf58dc755725b28f0821ef4ba6
Showing with 204 additions and 20 deletions.
  1. +24 −20 README.md
  2. +180 −0 docs/for_developers.md
@@ -14,22 +14,22 @@ For more information and for applying, see [here]().
<br>
-->

EvoMaster ([www.evomaster.org](http://evomaster.org)) is a tool prototype
_EvoMaster_ ([www.evomaster.org](http://evomaster.org)) is a tool prototype
that automatically *generates* system-level test cases.
Internally, it uses an *evolutionary algorithm* and *bytecode analysis* to be
able to generate effective test cases.
The approach is to *evolve* test cases from an initial population of
random ones, trying to maximize measures like code coverage and fault detection.

At the moment, EvoMaster targets RESTful APIs compiled to
At the moment, _EvoMaster_ targets RESTful APIs compiled to
JVM 8 bytecode (e.g., Java and Kotlin).

This project is in early stage of development. Documentation is still under construction.
A short [video](https://youtu.be/7zTLUlH-BNI) shows the use of EvoMaster on one of the
A short [video](https://youtu.be/7zTLUlH-BNI) shows the use of _EvoMaster_ on one of the
case studies in [EMB](https://github.com/EMResearch/EMB).


EvoMaster is currently (2018-2021) funded by a 7.8 million Norwegian Kroner grant
_EvoMaster_ is currently (2018-2021) funded by a 7.8 million Norwegian Kroner grant
by the Research Council of Norway (RCN), as part of the
Frinatek project <i>Evolutionary Enterprise Testing</i>.

@@ -45,7 +45,7 @@ Note: if you get an arror from the shade-plugin, then make sure to use
### Example

The following code is an example of one test that was automatically
generated by EvoMaster for a REST service called
generated by _EvoMaster_ for a REST service called
"scout-api" (see [EMB](https://github.com/EMResearch/EMB)).
The generated test uses the [RestAssured](https://github.com/rest-assured/rest-assured) library.

@@ -147,13 +147,13 @@ package structure.

### EvoMaster Driver

Note, to generate tests, you need an EvoMaster Driver up and running before
Note, to generate tests, you need an _EvoMaster Driver_ up and running before
executing `evomaster.jar`.
These drivers have to be built manually for each system under test (SUT).
See [EMB](https://github.com/EMResearch/EMB) for a set of existing SUTs with drivers.

To build a client driver in Java (or any JVM language), you need to import the
EvoMaster Java client library. For example, in Maven:
_EvoMaster_ Java client library. For example, in Maven:

```
<dependency>
@@ -165,7 +165,7 @@ EvoMaster Java client library. For example, in Maven:

For the latest version, check [Maven Central Repository](https://mvnrepository.com/artifact/org.evomaster/evomaster-client-java-controller).
The latest version number should also appear at the top of this page.
If you are compiling directly from the EvoMaster source code, make sure to use `mvn install` to
If you are compiling directly from the _EvoMaster_ source code, make sure to use `mvn install` to
install the snapshot version of the Java client into your local Maven repository
(e.g., under *~/.m2*).

@@ -174,12 +174,12 @@ Once the client library is imported, you need to create a class that extends eit
or
`org.evomaster.client.java.controller.ExternalSutController`.
Both these classes extend `SutController`.
The difference is on whether the SUT is started in the same JVM of the EvoMaster
The difference is on whether the SUT is started in the same JVM of the _EvoMaster_
driver (*embedded*), or in a separated JVM (*external*).

The easiest approach is to use the *embedded* version, especially when dealing with
frameworks like Spring and DropWizard.
However, when the presence of the EvoMaster client library gives side-effects (although
However, when the presence of the _EvoMaster_ client library gives side-effects (although
its third-party libraries are shaded, side-effects might still happen),
or when it is not possible (or too complicate) to start the SUT directly (e.g., JEE),
it is better to use the *external* version.
@@ -193,7 +193,7 @@ Once a class is written that extends either `EmbeddedSutController` or
be implemented.
For example, those methods specify how to start the SUT, how it should be stopped,
and how to reset its state.
The EvoMaster Java client library also provides further utility classes to help
The _EvoMaster_ Java client library also provides further utility classes to help
writing those controllers/drivers.
For example, `org.evomaster.client.java.controller.db.DbCleaner` helps in resetting
the state of a database (if any is used by the SUT).
@@ -205,7 +205,7 @@ and the existing examples in


Once a class `X` that is a descendant of `SutController` is written, you need
to be able to start the EvoMaster driver, by using the
to be able to start the _EvoMaster_ driver, by using the
`org.evomaster.client.java.controller.InstrumentedSutStarter`
class.
For example, in the source code of the class `X`, you could add:
@@ -228,39 +228,43 @@ then you can use `evomaster.jar` to finally generate test cases.
### How to Contribute

There are many ways in which you can contribute.
If you found EvoMaster of any use, the easiest
If you found _EvoMaster_ of any use, the easiest
way to show appreciation is to *star* it.
Issues and feature requests can be reported on
the [issues](https://github.com/EMResearch/EvoMaster/issues) page:

* *Bugs*: as for any bug report, the more detailed
you can be the better.
If you are using EvoMaster on an open source project,
If you are using _EvoMaster_ on an open source project,
please provide links to it, as then it is much easier
to reproduce the bugs.

* If you are trying to use EvoMaster, but the instructions
* If you are trying to use _EvoMaster_, but the instructions
in these notes are not enough to get you started,
then it means it is a "bug" in the documentation, which then would need
to be clarified.

* *Feature Requests*: to improve EvoMaster,
* *Feature Requests*: to improve _EvoMaster_,
we are very keen to receive
feature requests, although of course we cannot
guarantee when they are going to be implemented.

* *Pull Requests*: we are very keen to receive PRs, as long as you agree
with the license of _EvoMaster_. However, before making a PR, should read
the [notes for developers](docs/for_developers.md).


### Publications and Further Resources

The development of EvoMaster is rooted in academia.
Academic publications based on EvoMaster and slides of presentations can be
The development of _EvoMaster_ is rooted in academia.
Academic publications based on _EvoMaster_ and slides of presentations can be
found [here](docs/publications.md).
These can be useful if you want to know more on how EvoMaster works internally,
These can be useful if you want to know more on how _EvoMaster_ works internally,
e.g., details on the Many Independent Algorithm (MIO).


### License
EvoMaster's source code is released under the LGPL (v3) license.
_EvoMaster_'s source code is released under the LGPL (v3) license.



@@ -0,0 +1,180 @@
# NOTES FOR DEVELOPERS


These notes are meant for developers working on _EvoMaster_, and for people making a pull request.
There are several rules of thumb regarding how to write "good code",
but often rules are either too generic and not tailored for a given particular
piece of software (e.g., different kinds of architectures).

The rules of thumb described here in this document are not meant to be either exhaustive nor absolute.
Rigid rules are not substitute for common sense, as they are rather guidelines that can
be ignored in some special cases.
Furthermore, the guidelines need to be _realistic_ and easy to use: there would be no point
to ask for detailed comments on each single method/field and 100% coverage test suites...

These notes also include some explanations and motivations for some of the architectural choices
made in the development of _EvoMaster_.


### Kotlin vs. Java
The core process of _EvoMaster_ is built in _Kotlin_, as we strongly prefer it over _Java_.
However, the client libraries for JDK SUTs (e.g., not just _Java_, but also all other languages that do
compile to JDK bytecode) are written in _Java_ instead of _Kotlin_.
The main reason is that, being libraries, we do not want to also have to ship the _Kotlin_ runtime
libraries with them.



### AVOID `System.out` AND `System.err`
_EvoMaster_ uses a logging framework.
For debugging and logging errors in a class `Foo`, create a logger in the following way.

* for Java: `private static Logger log = LoggerFactory.getLogger(Foo.class);`
* for Kotlin: `companion object { private val log: Logger = LoggerFactory.getLogger(Foo::class.java)}`

It is important to keep the same name `log` to make things consistent among different classes.
If the logging should be part the actual output for the console user, then rather use:

`LoggingUtil.getInfoLogger()`


### AVOID `String` CONCATENATION IN LOGGERS

Writing something like:

`log.debug("this is not "+ foo + " very " + bar +" efficient")`

is not efficient, as most of the time debug logs are deactivated, and concatenating strings is
expensive. Recall `String` is immutable, and each `+` does create a new `String` object.
The above logging can be rewritten into:

`log.debug("this is not {} very {} efficient", foo, bar)`

Note: not a big deal for _warn_/_error_, as those are/should be rare... but it can become
quite an overhead for _trace_/_debug_/_info_.




### DO NOT USE `System.exit`

Better to throw an exception, as the entry point of _EvoMaster_ does some logging when ends.
Furthermore, `System.exit` becomes problematic when unit testing _EvoMaster_.



### STATIC VARIABLES ARE YOUR ENEMY

Static variables should be either constant or representing transient data (e.g., cache information
whose presence/missing has only effect on performance, not on functionality).
Having "classes with static state" is usually a poor OO design (an exception to this rule
is `ExecutionTracer`).
If those are really needed, then you should rather use an _injectable_ singleton service (see next point).
This is not just to be pedantic, but, really, non-constant static variables make unit testing
far much harder and lead to code that is more difficult to understand and maintain.


### `Guice` and `Governator`

To avoid issues with mutable static variables, we use a dependency injection framework.
In particular, we use `Guice`, extended with `Governator` to handle post-construct events.
All injectable services should be singletons, and declared under a package called `*.service` (this
is to make it easy to find out which services are available).

There is no auto-discovery of beans. This is done manually.
The reason is that, depending on configurations, we can have many different context initializations.
For example, the beans used for testing REST APIs would not be needed when testing GraphQL ones.


### HOW TO WRITE UNIT TEST CASES

Unit tests should be put in the `src/test/java` and `src/test/kotlin` folders,
following the same package structure as _EvoMaster_ code.
A unit test suite for SUT `org.evomaster.somepackage.Foo` __MUST__ be called `org.evomaster.somepackage.FooTest`.
This is important for several reasons:
- Need to know what class the test case is supposed to unit test by just looking at its name
- Should be easy to identify if a class has a test suite for it
- If in same package, then the test suite can access package/protected fields/methods
- Having `Test` as postfix (instead of a prefix) is useful for when searching for classes by name
- A `Test` postfix is a requirement for _Maven_ to execute the test suite during the build


### HOW TO WRITE END-TO-END (E2E) TEST CASES

Besides unit tests, it is essential to have E2E ones as well.
Those should be added under the `e2e-tests` module.
Being _non-deterministic_, we cannot guarantee that _EvoMaster_ can always find a valid solution (e.g.,
create test cases with certain properties).
Furthermore, we cannot run the E2E tests for long time (otherwise the CI builds will take forever).
The idea is to create artificial SUTs that should be _trivial_ to solve when some settings (which we want
to test) are on, and very difficult (if not straight-out infeasible) otherwise.

Note: current version of JUnit 5 is worse than JUnit 4 when dealing with E2E tests.
E.g., there is no handling of _flaky_ tests (in JUnit 4, this was handled by the _Surefire_/_Failsafe_ plugins).
This is the reason why such test executions should be wrapped inside a `handleFlaky` call.


### AVOID TOO LONG METHODS

Too long methods (e.g., more than 100 lines) should be split, as difficult to understand.
For this task, in _IntelliJ_, you can right-click on a code snippet and choose
"_Refactor -> Extract -> Function_"




### WRITE COMMENTS

In the ideal world, each class/method/field would have nice, detailed, appropriate code comments.
But even in such a beautiful world, everything would go to hell at the first code change, as that might
require manually changing most of the code comments.

Cannot really quantify how much comments one should write, but at least it would be good to have:
* brief (1-2 sentences) description of what the class is useful for (just before the class declaration)
* for fields that are data structures (e.g., collections and arrays) some comments would be useful, as long and detailed
variable names are not practical
* for `Map`s, should add a comment stating what is the _key_, and what is the _value_.

When writing a comment for a class/method/field, use JavaDoc style:
/**
*/
In this way, your IDE can show the comments when you hover with the mouse over them.
### IF CANNOT AVOID EXTERNAL SIDE-EFFECTS, DO DOCUMENT IT!!!
If a call on a object has side-effects outside the class itself (e.g., writing to disk, add a system hook thread),
then this needs to be documented (see point on how to write comments),
unless it is obvious from the function/class name.
### PRE AND POST CONDITIONS
* _Pre-conditions_ of `public` methods should throw exceptions explicitly
(e.g., `IllegalArgumentException` and `IllegalStateException`).
Whenever possible, it is worth to write pre-conditions to `public` methods.
* _Pre-conditions_ of `private` methods and _post-conditions_ (both `public` and `private` methods)
should use the keyword `assert` in _Java_, and the function `assert()` in _Kotlin_.
(An exception is when the validation of inputs of a public method is delegated/moved to
a `private` method: in this case you could add `throw`.)
_Post-conditions_ are good, but often are difficult to write.
Note: a _post-condition_ does not to be complete to be useful (i.e., find bugs).
For example, if we have _A && B_, but the writing
of _B_ is too difficult (or time consuming), still having just _A_ as _post-condition_ can help
Note: currently _Kotlin_ does not have lazily evaluated assertions.
If you are writing a computational expensive check, rather user `Lazy.assert(predicate)`.
### FIELDS/CONSTRUCTORS/METHODS ORDER IN A CLASS
When writing a new class (or re-factoring a current one), fields should come first,
followed by class constructors and then the other methods.

0 comments on commit 0f6c8a4

Please sign in to comment.
You can’t perform that action at this time.