From 84dc8eef3af4c9d141de012bb1ef70f7eb448328 Mon Sep 17 00:00:00 2001 From: Olga Naumenko <64418523+olganaumenko@users.noreply.github.com> Date: Fri, 17 Feb 2023 13:10:46 +0300 Subject: [PATCH 1/3] UnitTestBot Go README and Developer guide reviewed --- utbot-go/README.md | 221 ++++++++++++------------------ utbot-go/docs/DEVELOPERS_GUIDE.md | 67 --------- utbot-go/docs/DEVELOPER_GUIDE.md | 57 ++++++++ 3 files changed, 148 insertions(+), 197 deletions(-) delete mode 100644 utbot-go/docs/DEVELOPERS_GUIDE.md create mode 100644 utbot-go/docs/DEVELOPER_GUIDE.md diff --git a/utbot-go/README.md b/utbot-go/README.md index 90645ebe2a..75776920d9 100644 --- a/utbot-go/README.md +++ b/utbot-go/README.md @@ -1,180 +1,141 @@ -# UTBot Go +# UnitTestBot Go -## About project +UnitTestBot Go automatically generates ready-to-use unit tests for Go programs. -UTBot Go _**automatically generates unit tests for Go programs**_. Generated tests: +With UnitTestBot Go, you can find bugs in your code and fixate the desired program behavior with less effort. -* provide _high code coverage_ and, as a result, its reliability; -* fixate the current behavior of the code as _regression tests_. +The project is under development, +so feel free to [contribute](https://github.com/UnitTestBot/UTBotJava/blob/main/utbot-go/docs/DEVELOPERS_GUIDE.md). -The core principles of UTBot Go are _**ease of use**_ and _**maximizing code coverage**_. +## Features and details -*** +UnitTestBot Go now implements the _basic fuzzing technique_. +It generates input values with respect to parameter types, +inserts these values into the user functions, and executes the resulting test cases. -_The project is currently under development._ +### Supported types for function parameters -## Features +At the moment, UnitTestBot Go is able to generate values for _primitive types_, _arrays_ and _structs_. -At the moment, only the _basic fuzzing technique_ is supported: namely, the execution of functions on predefined values, -depending on the type of parameter. +For _floating point types_, UnitTestBot Go supports working with _infinity_ and _NaN_. -At the moment, functions are supported, the parameters of which have _any primitive types_, _arrays_ and _structs_. +### Supported types for the returned results -For floating point types, _correct work with infinities and NaNs_ is also supported. +For the returned function results, +UnitTestBot Go supports the `error` type in addition to all the types supported for the function parameters. -Function result types are supported the same as for parameters, but with _support for types that implement `error`_. +It also captures `panic` cases correctly. -In addition, UTBot Go correctly captures not only errors returned by functions, but also _`panic` cases_. +Check the examples of [supported functions](https://github.com/UnitTestBot/UTBotJava/blob/main/utbot-go/go-samples/simple/samples.go). -Examples of supported functions can be found [here](go-samples/simple/samples.go). +### Keeping tests near the source code -## Important notes +[Testing code typically +lives in the same package as the code it tests](https://gobyexample.com/testing). +By default, UnitTestBot Go generates tests into the `[name of source file]_go_ut_test.go` file located in the same +directory and Go package as the corresponding source file. -### Where are tests generated? +If you need to change the location for the generated tests, +use the `generateGo` CLI command and set the generated test output mode to +`StdOut` (`-p, --print-test` flag). +Then, with bash primitives, redirect the output to an arbitrary file. -It is true that in the described API it is currently almost impossible to customize the file in which the tests are -generated. By default, test generation results in the file `[name of source file]_go_ut_test.go` _located in the same -directory and Go package_ as the source file. +### Requirements to source code -In other words, tests are generated right next to the source code. But why? +To simplify handling dependencies and generating tests, UnitTestBot Go requires the code under test _to +be a part of a Go project_ consisting of a module and packages. -* Go was created for convenient and fast development, therefore it has appropriate guidelines: `Testing code typically - lives in the same package as the code it tests` ([source](https://gobyexample.com/testing)). For example, this - approach provides a clear file structure and allows you to run tests as simply and quickly as possible. -* Placing tests in the same package with the source code allows you to test private functions. Yes, this is not good - practice in programming in general: but, again, it allows you to develop in Go faster by automatically checking even - the internal implementation of the public API of the package via unit testing. -* This approach avoids problems with dependencies from imported packages etc. It's always nice not to have them, if - possible. +To create a simple project, refer to the [starter tutorial](https://go.dev/doc/tutorial/getting-started) if necessary. -Of course, Go has the ability to store tests away from the source code. In the future, it is planned to support this -functionality in the UTBot Go. +For larger projects, try the [Create a Go module](https://go.dev/doc/tutorial/create-module) +and [Call your code from another module](https://go.dev/doc/tutorial/call-module-code) tutorials. -However, the word `almost` in the first sentence of this section is not redundant at all, there is _a small hack_. When -using the `generateGo` CLI command, you can set the generated tests output mode to StdOut (`-p, --print-test` flag). -Then using, for example, bash primitives, you can redirect the output to an arbitrary file. Such a solution will not -solve possible problems with dependencies, but will automatically save the result of the generation in the right place. +To create a new module rooted in the current directory, use the `go mod init` command. -### Is there any specific structure of Go source files required? +To add missing module requirements necessary to build the current module’s packages and dependencies, +use the `go mod tidy` command. For editing and formatting `go.mod` files, use the `go mod edit` command. -Yes, unfortunately or fortunately, it is required. Namely, the source code file for which the tests are generated _must -be in a Go project_ consisting of a module and packages. +In the future, we plan to make UnitTestBot Go working with arbitrary code as input and generate the simplest +Go projects automatically. -But do not be afraid! Go is designed for convenient and fast development, so _it's easy to start a Go -project_. For example, the [starter tutorial](https://go.dev/doc/tutorial/getting-started) of the language just -tells how to create the simplest project in Go. For larger projects, it is recommended to read a couple of sections of -the tutorial further: [Create a Go module](https://go.dev/doc/tutorial/create-module) -and [Call your code from another module](https://go.dev/doc/tutorial/call-module-code). - -To put it simply and briefly, in the simplest case, it is enough to use one call to the `go mod init` command. For more -complex ones, `go mod tidy` and `go mod edit` may come in handy. Finally, when developing in IntelliJ IDEA, you almost -don’t have to think about setting up a project: it will set everything up by itself. - -But _why does UTBot Go need a Go project_ and not enough files in a vacuum? The answer is simple — -dependencies. Go modules are designed to conveniently support project dependencies, which are simply listed in -the `go.mod` file. Thanks to it, modern Go projects are easy to reproduce and, importantly for UTBot Go, to test. - -In the future, it is planned to add the ability to accept arbitrary code as input to UTBot Go and generate the simplest -Go project automatically. - -## Install and use easily +## Installation and usage ### IntelliJ IDEA plugin -_Requirements:_ +#### Requirements -* `IntelliJ IDEA (Ultimate Edition)`, compatible with version `2022.2`; -* installed `Go SDK` version later than `1.18`; -* installed in IntelliJ IDEA [Go plugin](https://plugins.jetbrains.com/plugin/9568-go), compatible with the IDE - version (it is for this that the `Ultimate` edition of the IDE is needed); -* properly configured Go module for source code file (i.e. for file to generate tests for): corresponding `go.mod` file - must exist; -* installed Go modules `github.com/stretchr/testify/assert` and `golang.org/x/tools@v0.4.0`: fortunately, IDEA will automatically highlight - it and offer to install the first time the tests are generated. +* IntelliJ IDEA (Ultimate Edition) — versions from 2022.2 to 2022.2.4 +* Go SDK 1.18 or later +* Compatible [Go plugin](https://plugins.jetbrains.com/plugin/9568-go) for IntelliJ IDEA +* Properly configured `go.mod` file for the code under test +* `github.com/stretchr/testify/assert` and `golang.org/x/tools@v0.4.0` Go modules installed (IntelliJ IDEA automatically offers to install them as soon as the tests are generated) -Most likely, if you are already developing Go project in IntelliJ IDEA, then you have already met all the requirements. +#### Installation -_To install the UTBot Go plugin in IntelliJ IDEA:_ +To install the UnitTestBot Go plugin in IntelliJ IDEA, refer to [UnitTestBot user guide](https://github.com/UnitTestBot/UTBotJava/actions/runs/3012565900). -* just find the latest version of [UnitTestBot](https://plugins.jetbrains.com/plugin/19445-unittestbot) in the plugin - market; -* or download zip archive with `utbot-intellij JAR` - from [here](https://github.com/UnitTestBot/UTBotJava/actions/runs/3012565900) and install it in IntelliJ IDEA as - follows from plugins section (yes, you need to select the entire downloaded zip archive, it does not need to be - unpacked). - ![](docs/images/install-intellij-plugin-from-disk.png) +#### Usage -Finally, you can _start using UTBot Go_: open any `.go` file in the IDE and press `alt + u, alt + t`. After -that, a window will appear in which you can configure the test generation settings and start running it in a couple -of clicks. +1. In your IntelliJ IDEA, go to **File** > **Settings** > **Tools** > **UnitTestBot** and enable **Experimental languages support**. +2. Open a `.go` file and press **Alt+Shift+U**. +3. In the **Generate Tests with UnitTestBot** window, you can configure the settings. +### CLI -### CLI application +#### Requirements -_Requirements:_ +* Java SDK 11 or later +* Go SDK 1.18 or later +* Properly configured `go.mod` file for the code under test +* GCC as well as `github.com/stretchr/testify/assert` and `golang.org/x/tools@v0.4.0` Go modules installed -* installed `Java SDK` version `11` or higher; -* installed `Go SDK` version later than `1.18`; -* properly configured Go module for source code file (i.e. for file to generate tests for): corresponding `go.mod` file - must exist; -* installed `gcc` and Go modules `github.com/stretchr/testify/assert` and `golang.org/x/tools@v0.4.0` to run tests. +#### Installation -_To install the UTBot Go CLI application:_ download zip archive containing `utbot-cli JAR` -from [here](https://github.com/UnitTestBot/UTBotJava/actions/runs/3012565900), then extract its content (JAR file) to a -convenient location. +To install the UnitTestBot Go CLI application, go to GitHub, scroll through the release notes and click **Assets**. +Download the zip-archive named like **utbot-cli-VERSION**. +Extract the JAR file from the archive. -Finally, you can _start using UTBot Go_ by running the extracted JAR on the command line. Two actions are -currently supported: `generateGo` and `runGo` for generating and running tests, respectively. +#### Usage -For example, to find out about all options for actions, run the commands as follows -(`utbot-cli-2022.8-beta.jar` here is the path to the extracted JAR): +Run the extracted JAR file using a command line: `generateGo` and `runGo` actions are supported for now. +To find info about the options for these actions, +insert the necessary JAR file name instead of `utbot-cli-2022.8-beta.jar` in the example and run the following commands: ```bash java -jar utbot-cli-2022.8-beta.jar generateGo --help ``` - or - ```bash java -jar utbot-cli-2022.8-beta.jar runGo --help ``` -respectively. - -_Action `generateGo` options:_ - -* `-s, --source TEXT`, _required_: specifies Go source file to generate tests for. -* `-f, --function TEXT`, _required_: specifies function name to generate tests for. Can be used multiple times to select multiple - functions at the same time. -* `-go, --go-path TEXT`, _required_: specifies path to Go executable. For example, it could be `/usr/local/go/bin/go` - for some systems. -* `-et, --each-execution-timeout INT`: specifies a timeout in milliseconds for each fuzzed function execution. Default is - `1000` ms. -* `-at, --all-execution-timeout INT`: specifies a timeout in milliseconds for all fuzzed function execution. Default is - `60000` ms. -* `-p, --print-test`: specifies whether a test should be printed out to StdOut. Is disabled by default. -* `-w, --overwrite`: specifies whether to overwrite the output test file if it already exists. Is disabled by default. -* `-h, --help`: show help message and exit. - -_Action `runGo` options:_ - -* `-p, --package TEXT`, _required_: specifies Go package to run tests for. -* `-go, --go-path TEXT`, _required_: specifies path to Go executable. For example, it could be `/usr/local/go/bin/go` - for some systems. -* `-v, --verbose`: specifies whether an output should be verbose. Is disabled by default. -* `-j, --json`: specifies whether an output should be in JSON format. Is disabled by default. -* `-o, --output TEXT`: specifies output file for tests run report. Prints to StdOut by default. -* `-cov-mode, --coverage-mode [html|func|json]`: specifies whether a test coverage report should be generated and - defines its mode. Coverage report generation is disabled by default. Examples of different coverage reports modes can - be found [here](go-samples/simple/reports). -* `-cov-out, --coverage-output TEXT`: specifies output file for test coverage report. Required if `[--coverage-mode]` is +`generateGo` options: + +* `-s, --source TEXT`, _required_: specifies a Go source file to generate tests for. +* `-f, --function TEXT`, _required_: specifies a function name to generate tests for. Can be used multiple times to select multiple + functions. +* `-go, --go-path TEXT`, _required_: specifies a path to a Go executable. For example, `/usr/local/go/bin/go`. +* `-et, --each-execution-timeout INT`: specifies a timeout in milliseconds for each target function execution. + The default timeout is 1,000 ms. +* `-at, --all-execution-timeout INT`: specifies a timeout in milliseconds for all target function executions. + The default timeout is 60,000 ms. +* `-p, --print-test`: specifies whether a test should be printed out to `StdOut`. Disabled by default. +* `-w, --overwrite`: specifies whether to overwrite the output test file if it already exists. Disabled by default. +* `-h, --help`: shows a help message and exits. + +`runGo` options: + +* `-p, --package TEXT`, _required_: specifies a Go package to run tests for. +* `-go, --go-path TEXT`, _required_: specifies a path to a Go executable. For example, `/usr/local/go/bin/go`. +* `-v, --verbose`: specifies whether an output should be verbose. Disabled by default. +* `-j, --json`: specifies whether an output should be in JSON format. Disabled by default. +* `-o, --output TEXT`: specifies an output file for a test run report. Prints to `StdOut` by default. +* `-cov-mode, --coverage-mode [html|func|json]`: specifies whether a test coverage report should be generated and defines the report format. + Coverage report generation is disabled by default. +* `-cov-out, --coverage-output TEXT`: specifies the output file for a test coverage report. Required if `[--coverage-mode]` is set. -## Contribute to UTBot Go - -If you want to _take part in the development_ of the project or _learn more_ about how it works, check -out [DEVELOPERS_GUIDE.md](docs/DEVELOPERS_GUIDE.md). - -For the current list of tasks, check out [FUTURE_PLANS.md](docs/FUTURE_PLANS.md). +## Contributing to UnitTestBot Go -Your help and interest is greatly appreciated! +To take part in project development or learn more about UnitTestBot Go, check +out the [Developer guide](docs/DEVELOPER_GUIDE.md) and our [future plans](docs/FUTURE_PLANS.md). \ No newline at end of file diff --git a/utbot-go/docs/DEVELOPERS_GUIDE.md b/utbot-go/docs/DEVELOPERS_GUIDE.md deleted file mode 100644 index f52693b896..0000000000 --- a/utbot-go/docs/DEVELOPERS_GUIDE.md +++ /dev/null @@ -1,67 +0,0 @@ -# UTBot Go: Developers Guide - -# How UTBot Go works in general - -```mermaid -flowchart TB - A["Targets selection and configuration (IDEA plugin or CLI)"]:::someclass --> B(Go source code analysis) - classDef someclass fill:#b810 - - B --> C(Fuzzing) - C --> D(Fuzzed function execution) - D --> E(Getting results) - E --> C - C ---> F(Test file generation) -``` - - -In the diagram above, you can see _the main stages of the UTBot Go test generation pipeline_. Let's take a look at each -in more detail! - -### Targets selection and configuration - -This block in the diagram is highlighted in a separate color, since the action is mainly performed by the user. Namely, -the user selects the target source file and functions for which tests will be generated and configure generation -parameters (for example, fuzzed function execution timeout, path to Go executable, etc.). - -If UTBot Go is built as a plugin for IntelliJ IDEA, then the user opens the target Go source file and, using a keyboard -shortcut, calls up a dialog window where it's possible to select functions and configure settings. - -If UTBot Go is build as a CLI application, then the user sets the functions and parameters using command line flags. - -### Go source code analysis - -Once the user has chosen the target functions, it is necessary to collect information about them. Namely: their -signatures and information about types (in the basic version); constants in the body of functions and more (in the -future). This is exactly what happens at the stage of code analysis. As a result, UTBot Go gets an internal -representation of the target functions, sufficient for the subsequent generation of tests. - -### Fuzzing - -Fuzzing is the first part of the test cases generation process. At this stage, values, that will be used -to test the target functions, are generated. Namely, to be passed to functions as arguments in the next steps. -Then the result of function execution is analyzed and the generation of new values continues or stops. - -### Fuzzed function execution - -In the previous step, UTBot Go generated the values that the functions need to be tested with — now the task is to -do this. Namely, execute the functions with the values generated for them. - -Essentially, the target function, the values generated for it, and the result of its execution form a test case. In -fact, that is why this stage ends the global process of generating test cases. - -### Getting results - -Saving results of fuzzed function execution and sending them to fuzzing for analysis. - -### Test code generation - -Finally, the last stage: the test cases are ready, UTBot Go needs only to generate code for them. Need to carefully consider the features of the Go language (for example, the necessity to cast -constants to the desired type). - -_That's how the world (UTBot Go) works!_ - -## How to test UTBot Go - -_**TODO:**_ Gradle `runIde` task or building CLI application JAR locally. To build CLI version the `build` on `utbot-cli-go` should be called. - diff --git a/utbot-go/docs/DEVELOPER_GUIDE.md b/utbot-go/docs/DEVELOPER_GUIDE.md new file mode 100644 index 0000000000..3aaa19c1d6 --- /dev/null +++ b/utbot-go/docs/DEVELOPER_GUIDE.md @@ -0,0 +1,57 @@ +# UnitTestBot Go Developer Guide + +## Overview + +Here are the main stages of UnitTestBot Go test generation pipeline. + +```mermaid +flowchart TB + A["Target selection and configuration (IntelliJ IDEA plugin or CLI)"]:::someclass --> B(Go source code analysis) + classDef someclass fill:#b810 + + B --> C(Fuzzing) + C --> D(Executing functions) + D --> E(Getting results) + E --> C + C ---> F(Test file generation) +``` + +### Target selection and configuration + +A user manually selects the target source file and functions to generate the tests for. +Test generation settings are also configured manually. + +### Go source code analysis + +UnitTestBot Go collects information about the target functions: +* signatures and type information (in the basic version); +* constants in the function bodies (to be implemented). + +As a result, UnitTestBot Go gets an internal +representation of the target functions. + +### Fuzzing + +During the fuzzing stage, UnitTestBot Go generates the input values for the source code. + +### Executing target functions + +UnitTestBot Go executes the target functions with the generated values as arguments and analyses the result +to continue or stop the generation process. + +### Getting results + +The result of target function execution is sent to the fuzzer for analysis: +coverage rate information guides the following value generation process. +Based on this feedback, +the fuzzer decides whether it should continue or stop generating input values. + +### Test code generation + +Generating test source code requires support for Go language features (for example, the necessity to cast +constants to the desired type). +Extended support for various Go constructs is a future plan for us. + +## How to test UnitTestBot Go + +_**TODO**_ From 9cf7665f892f7e65a51130452acc82d9a296e50e Mon Sep 17 00:00:00 2001 From: Olga Naumenko <64418523+olganaumenko@users.noreply.github.com> Date: Fri, 17 Feb 2023 13:36:49 +0300 Subject: [PATCH 2/3] Link to user guide fixed --- utbot-go/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/utbot-go/README.md b/utbot-go/README.md index 75776920d9..b44e407332 100644 --- a/utbot-go/README.md +++ b/utbot-go/README.md @@ -72,7 +72,7 @@ Go projects automatically. #### Installation -To install the UnitTestBot Go plugin in IntelliJ IDEA, refer to [UnitTestBot user guide](https://github.com/UnitTestBot/UTBotJava/actions/runs/3012565900). +To install the UnitTestBot Go plugin in IntelliJ IDEA, refer to [UnitTestBot user guide](https://github.com/UnitTestBot/UTBotJava/wiki/Install-or-update-plugin). #### Usage From b865922812c27c19da4ec44425c598277bb18f28 Mon Sep 17 00:00:00 2001 From: Olga Naumenko <64418523+olganaumenko@users.noreply.github.com> Date: Mon, 20 Feb 2023 18:03:15 +0300 Subject: [PATCH 3/3] Requirements fixed --- utbot-go/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/utbot-go/README.md b/utbot-go/README.md index b44e407332..3faca20f00 100644 --- a/utbot-go/README.md +++ b/utbot-go/README.md @@ -68,7 +68,7 @@ Go projects automatically. * Go SDK 1.18 or later * Compatible [Go plugin](https://plugins.jetbrains.com/plugin/9568-go) for IntelliJ IDEA * Properly configured `go.mod` file for the code under test -* `github.com/stretchr/testify/assert` and `golang.org/x/tools@v0.4.0` Go modules installed (IntelliJ IDEA automatically offers to install them as soon as the tests are generated) +* `github.com/stretchr/testify/assert` Go module installed (IntelliJ IDEA automatically offers to install it as soon as the tests are generated) #### Installation