Skip to content

Update UNITTESTS and Greentea test docs #1364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 8, 2020
Merged

Update UNITTESTS and Greentea test docs #1364

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
508e4ce
Update UNITTESTS and Greentea test docs
rajkan01 Sep 3, 2020
4911ada
Editorial changes
Elise-Kaminski Sep 8, 2020
c48edce
Editorial changes
Elise-Kaminski Sep 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 28 additions & 27 deletions docs/debugging-testing/testing/testing_greentea.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ This document will help you start using Greentea. Please see the [`htrun` docume

### Test code structure

You can run tests throughout Mbed OS and for your project's code. They are located under a special directory called `TESTS`.
Greentea tests can be created anywhere in the Mbed OS directory structure or in your project's code. They are located under a special directory called `TESTS`. By convention, greentea tests for a specific Mbed OS library should be placed in a `TESTS` folder under that library. The `mbed-os/TESTS` folder is dedicated to integration or system tests.

The fact that the code is located under this directory means that it is ignored when building applications and libraries. It is only used when building tests. This is important because all tests require a `main()` function, and building them with your application would cause multiple `main()` functions to be defined.

Expand Down Expand Up @@ -43,7 +43,7 @@ Test discovery also obeys the same rules that are used when building your projec
For example, if you place a test under the directory `FEATURE_BLE` with the following path:

```
myproject/mbed-os/features/FEATURE_BLE/TESTS/ble_tests/unit_test
myproject/mbed-os/connectivity/FEATURE_BLE/tests/TESTS/ble_tests/unit_test
```

This test case is only discovered if the target being tested supports the BLE feature. Otherwise, the test is ignored.
Expand Down Expand Up @@ -77,7 +77,7 @@ The full build process is:

When building an Mbed application, the presence of an `mbed_app.json` file allows you to set or override different configuration settings from libraries and targets. However, because the tests share a common build, this can cause issues when tests have different configurations that affect the OS.

The build system looks for an `mbed_app.json` file in your shared project files (any directory not inside of a `TESTS` folder). If the system finds it, then this configuration file is used for both the non-test code and each test case inside your project's source tree. If there is more than one `mbed_app.json` file in the source tree, then the configuration system will error.
The build system looks for an `mbed_app.json` file in your shared project files (any directory not inside of a `TESTS` folder). If the system finds it, then this configuration file is used for both the non-test code and each test case inside your project's source tree. If there is more than one `mbed_app.json` file in the source tree, then the configuration system will throw an error.

If you need to test with multiple configurations, then you can use the `--app-config` option. This overrides the search for an `mbed_app.json` file and uses the configuration file that you specify for the build.

Expand Down Expand Up @@ -149,7 +149,8 @@ To interact with the host test from the device, you can use two functions: `gree

### Creating the host test

This example writes an integration test that sends `hello` to the host and waits until it receives `world`. Create a file called `hello_world_tests.py` in the `TESTS/host_tests` folder, and fill it with:
This example writes an integration test that sends `hello` to the host and waits until it receives `world`.
Create an `host_tests` directory under `mbed-os/TESTS/`, if it does not exist and create a file called `hello_world_tests.py` in the `mbed-os/TESTS/host_tests` folder, and fill it with:

```py
from mbed_host_tests import BaseHostTest
Expand Down Expand Up @@ -189,7 +190,7 @@ This registers one function you can call from the device: `init`. The function c

### Creating the Greentea test

This example writes the embedded part of this test. Create a new file `main.cpp` in `TESTS/tests/integration-test`, and fill it with:
This example writes the embedded part of this test. Create a new file `main.cpp` under the `TESTS/integration/test_case` directory, and fill it with:

[![View code](https://www.mbed.com/embed/?url=https://github.com/ARMmbed/mbed-os-snippet-Greentea_Ex_2/tree/v6.0)](https://github.com/ARMmbed/mbed-os-snippet-Greentea_Ex_2/blob/v6.0/main.cpp)

Expand All @@ -198,7 +199,7 @@ You see the calls to and from the host through the `greentea_send_kv` and `green
Run the test:

```
$ mbed test -v -n tests-tests-integration-test
$ mbed test -v -n tests-integration-test_case
```

## Debugging tests
Expand Down Expand Up @@ -269,41 +270,41 @@ You can use the `--compile-list` argument to list all available tests:

```
$ mbed test --compile-list
[mbed] Working path "/Users/janjon01/repos/first-greentea-test" (program)
[mbed] Working path "/Users/janjon01/mbed-os" (program)
Test Case:
Name: mbed-os-components-storage-blockdevice-component_flashiap-tests-filesystem-fopen
Path: ./mbed-os/components/storage/blockdevice/COMPONENT_FLASHIAP/TESTS/filesystem/fopen
Name: storage-blockdevice-component_sd-tests-tests-filesystem-fopen
Path: ./storage/blockdevice/COMPONENT_SD/tests/TESTS/filesystem/fopen
Test Case:
Name: mbed-os-features-cellular-tests-api-cellular_device
Path: ./mbed-os/features/cellular/TESTS/api/cellular_device
Name: connectivity-lorawan-tests-tests-lorawan-loraradio
Path: ./connectivity/lorawan/tests/TESTS/lorawan/loraradio

...
```

After compilation, you can use the `--run-list` argument to list all tests that are ready to be ran.
After compilation, you can use the `--run-list` argument to list all tests that are ready to be run.

### Executing all tests

The default action of Greentea using `mbed test` is to execute all tests found. You can also add `-v` to make the output more verbose.

### Limiting tests

You can select test cases by name using the `-n` argument. This command executes all tests named `tests-mbedmicro-rtos-mbed-mail` from all builds in the test specification:
You can select test cases by name using the `-n` argument. This command executes all tests named `platform-tests-tests-mbed_platform-atomic` from all builds in the test specification:

```
$ mbed test -n tests-mbedmicro-rtos-mbed-mail
$ mbed test -n platform-tests-tests-mbed_platform-atomic
```

When using the `-n` argument, you can use the `*` character as a wildcard. This command executes all tests that start with `tests-` and have `-rtos-` in them.
When using the `-n` argument, you can use the `*` character as a wildcard. This command executes all tests that start with `tests-` and have `-mbed_platform-` in them.

```
$ mbed test -n tests-*-rtos-*
$ mbed test -n tests-*-mbed_platform-*
```

You can use a comma (`,`) to separate test names (argument `-n`) and build names (argument `-t`). This command executes the tests `tests-mbedmicro-rtos-mbed-mail` and `tests-mbed_drivers-c_strings` for the `K64F-ARM` and `K64F-GCC_ARM` builds in the test specification:
You can use a comma (`,`) to separate test names (argument `-n`) and build names (argument `-t`). This command executes the tests `platform-tests-tests-mbed_platform-atomic` and `drivers-tests-tests-mbed_drivers-c_strings` for the `K64F-ARM` and `K64F-GCC_ARM` builds in the test specification:

```
$ mbed test -n tests-mbedmicro-rtos-mbed-mail,tests-mbed_drivers-c_strings -t K64F-ARM,K64F-GCC_ARM
$ mbed test -n platform-tests-tests-mbed_platform-atomic,drivers-tests-tests-mbed_drivers-c_strings -t K64F-ARM,K64F-GCC_ARM
```

### Selecting platforms
Expand Down Expand Up @@ -393,19 +394,19 @@ Place this file in your root folder, and name it `test_spec.json`.
"base_path": "./BUILD/K64F/ARM",
"baud_rate": 9600,
"tests": {
"tests-mbedmicro-rtos-mbed-mail": {
"platform-tests-tests-mbed_platform-atomic": {
"binaries": [
{
"binary_type": "bootable",
"path": "./BUILD/K64F/ARM/tests-mbedmicro-rtos-mbed-mail.bin"
"path": "./BUILD/K64F/ARM/platform/tests/tests/mbed_platform/atomic/atomic.bin"
}
]
},
"tests-mbed_drivers-c_strings": {
"drivers-tests-tests-mbed_drivers-c_strings": {
"binaries": [
{
"binary_type": "bootable",
"path": "./BUILD/K64F/ARM/tests-mbed_drivers-c_strings.bin"
"path": "./BUILD/K64F/ARM/drivers/tests/tests/mbed_drivers/c_strings/c_strings.bin"
}
]
}
Expand All @@ -417,11 +418,11 @@ Place this file in your root folder, and name it `test_spec.json`.
"base_path": "./BUILD/K64F/GCC_ARM",
"baud_rate": 9600,
"tests": {
"tests-mbedmicro-rtos-mbed-mail": {
"platform-tests-tests-mbed_platform-atomic": {
"binaries": [
{
"binary_type": "bootable",
"path": "./BUILD/K64F/GCC_ARM/tests-mbedmicro-rtos-mbed-mail.bin"
"path": "./BUILD/K64F/GCC_ARM/platform/tests/tests/mbed_platform/atomic/atomic.bin"
}
]
}
Expand All @@ -439,10 +440,10 @@ mbedgt: using multiple test specifications from current directory!
using 'BUILD\tests\K64F\ARM\test_spec.json'
using 'BUILD\tests\K64F\GCC_ARM\test_spec.json'
mbedgt: available tests for built 'K64F-GCC_ARM', location 'BUILD/tests/K64F/GCC_ARM'
test 'tests-mbedmicro-rtos-mbed-mail'
test 'platform-tests-tests-mbed_platform-atomic'
mbedgt: available tests for built 'K64F-ARM', location 'BUILD/tests/K64F/ARM'
test 'tests-mbed_drivers-c_strings'
test 'tests-mbedmicro-rtos-mbed-mail'
test 'drivers-tests-tests-mbed_drivers-c_strings'
test 'platform-tests-tests-mbed_platform-atomic'
```


Expand Down
12 changes: 6 additions & 6 deletions docs/debugging-testing/testing/unit_testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,9 +61,9 @@ In a terminal window:

## Test code structure

Find unit tests in the Mbed OS repository under the `UNITTESTS` folder. We recommend unit test files use an identical directory path as the file under test. This makes it easier to find unit tests for a particular class or a module. For example, if the file you're testing is `some/example/path/ClassName.cpp`, then all the test files are in the `UNITTESTS/some/example/path/ClassName` directory. Each test suite needs to have its own `unittest.cmake` file for test configuration.
Find unit tests in the Mbed OS repository under the `UNITTESTS` folder of each library. We recommend unit test files use an identical directory path as the file under test. This makes it easier to find unit tests for a particular class or a module. For example, if the file you're testing is `some/example/path/ClassName.cpp`, then all the test files are in the `UNITTESTS/some/example/path/ClassName` directory. Each test suite needs to have its own `unittest.cmake` file for test configuration.

All the class stubs should be located in the `UNITTESTS/stubs` directory. Multiple test suites can use a single stub class, which should follow the naming convention `ClassName_stub.cpp` for the source file and `ClassName_stub.h` for the header file. Use the actual header files for the unit tests, and don't stub headers if possible. The stubbed headers reside in the `UNITTESTS/target_h` directory.
All the class stubs should be located in the Mbed OS root directory `UNITTESTS/stubs`. Multiple test suites can use a single stub class, which should follow the naming convention `ClassName_stub.cpp` for the source file and `ClassName_stub.h` for the header file. Use the actual header files for the unit tests, and don't stub headers if possible. The stubbed headers reside in the `UNITTESTS/target_h` directory.

### Test discovery

Expand All @@ -77,7 +77,7 @@ The build system automatically generates names of test suites. The name is const

## Unit testing with Mbed CLI

Mbed CLI supports unit tests through `mbed test --unittests` command. To learn how to use unit tests with Mbed CLI, please see the [unit testing documentation section](../build-tools/test-and-debug.html). For other information on using Mbed CLI, please see the [CLI documentation in handbook](../build-tools/mbed-cli.html).
Mbed CLI supports unit tests through the `mbed test --unittests` command. To learn how to use unit tests with Mbed CLI, please see the [unit testing documentation section](../build-tools/test-and-debug.html). For other information on using Mbed CLI, please see the [CLI documentation in handbook](../build-tools/mbed-cli.html).

### Writing unit tests

Expand Down Expand Up @@ -190,7 +190,7 @@ With the following steps, you can write a unit test. This example creates dummy

## Add here test classes and stubs
set(unittest-test-sources
example/MyClass/test_MyClass.cpp
${CMAKE_CURRENT_LIST_DIR}/test_MyClass.cpp
stubs/OtherClass_stub.cpp
)
```
Expand Down Expand Up @@ -227,7 +227,7 @@ With the following steps, you can write a unit test. This example creates dummy
}
```

1. Stub all external dependencies. Create the following stub in `UNITTESTS/stubs`:
1. Stub all external dependencies. Create the following stub in the Mbed OS root directory `UNITTESTS/stubs`:

**OtherClass_stub.cpp**

Expand All @@ -253,7 +253,7 @@ Use Mbed CLI to build and run unit tests. For advanced use, you can run CMake an

1. Create a build directory: `mkdir UNITTESTS/build`.
1. Move to the build directory: `cd UNITTESTS/build`.
1. Run CMake using a relative path to `UNITTESTS` folder as the argument. So from `UNITTESTS/build` use `cmake ..`:
1. Run CMake using a relative path to the `UNITTESTS` folder as the argument. So from `UNITTESTS/build` use `cmake ..`:
- Add `-g [generator]` if generating files other than Unix Makefiles. For example, for MinGW, use `-g "MinGW Makefiles"`.
- Add `-DCMAKE_MAKE_PROGRAM=<value>`, `-DCMAKE_CXX_COMPILER=<value>` and `-DCMAKE_C_COMPILER=<value>` to use a specific Make program and compilers.
- Add `-DCMAKE_BUILD_TYPE=Debug` for a debug build.
Expand Down