Get use of test results
If generating summaries is enabled in UnitTestBot settings, you have the generated tests clustered into suites. Analyze these suites to see failing tests, explicitly thrown exceptions, and successful executions.
These suites are marked as regions in the test source code (see cluster comments in the picture below). To view all the suites at once and navigate between them, press Alt+7 to open the Structure tool window.
The typical suites are:
- Symbolic execution engine,
- Fuzzer,
- Other
each divided into:
- Successful executions
- Explicitly thrown unchecked exceptions
- Error suite
Find the example below:
Dive into these suites to find test methods with detailed descriptions.
Each test contains:
Briefly describes a test execution path.
- See Javadoc comment style for configuring the Plain text or Structured via custom Javadoc tags view.
- Render Javadoc comments to get more clear and structured view.
- Jump to the method-under-test right from the comment.
Provide a super short test method description inside:
-
For JUnit 5:
@DisplayName("<short description>") -
For TestNG:
@Test(description = "<short description>")
(For JUnit 4 there are no descriptions right in the annotations.)Disabled (or sandboxed) tests
Sometimes the tests with the @Disabled annotation and a stack trace appear in the output:
@Disabled("Disabled due to sandbox")Keep calm: it is a feature, not a bug. These tests were sandboxed, i.e. interrupted, because they tried to execute potentially unsafe code.
The most commonly forbidden actions are:
- working with files (read, write, create, delete),
- connecting to sockets,
- invoking
System.exit(), - accessing system properties or JVM properties,
- using Reflection.
You can add permissions in
~\.utbot\sandbox.policy. Create this file manually.You can refer to the Full list of permissions.
If you are sure you want the code to be executed as is (including the unsafe operations!) you can turn sandboxing off: add
AllPermissionto~\.utbot\sandbox.policy. Be careful!If you remove the @Disabled annotation and run the test, it will be run without creating a sandbox.
Usually looks like testMethodUnderTestName. Sometimes it is accompanied by explanatory supplements: e.g.,
ByFuzzer, ReturnsOneWithCornerCase, or others.
There are multiple ways to run tests and view coverage in IntelliJ IDEA. All of them apply to UnitTestBot results — feel free to choose any.
If you are not satisfied with coverage, try to Experiment with test generation.
It is the Static Analysis Results Interchange Format — a standard, JSON-based format for the output of static analysis tools.
Every static analysis tool has its own output format, and these formats usually have little in common. SARIF provides a universal output format, which makes it possible to create common tooling, such as viewers, bug filers, metrics calculators, etc.
For more information, please refer to SARIF Tutorials.
By default, UnitTestBot creates a SARIF report for each test generation.
Note: if you enable Parameterized tests in the Generate Tests with UnitTestBot window, SARIF report is unavailable for now.
You can find SARIF reports in the resources directory near Test sources root.
If test generation reveals errors in your code, they are registered in the SARIF report. IntelliJ IDEA automatically opens the Problems tool window showing the detected errors.
In Problems:
- click each error and press the buttons with a yellow light bulb to View generated test or Analyze stack trace,
- double-click the error to navigate to a corresponding fragment in your source code.
