Skip to content

Document enhancements in IBM i Testing v1.3.2 #91

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SanjulaGanepola merged 5 commits into from
Jan 28, 2026
Merged

Document enhancements in IBM i Testing v1.3.2 #91

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
60f6c4c
Document new default params
SanjulaGanepola Jan 12, 2026
2b43f2f
Update installation requirements
SanjulaGanepola Jan 12, 2026
009a6d7
Document onFailure
SanjulaGanepola Jan 12, 2026
8c1cfc3
Add docs on new test cancellation and inline assertion errors
SanjulaGanepola Jan 28, 2026
3887379
Document advanced configuration
SanjulaGanepola Jan 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added src/content/docs/developing/testing/Testing_CancelledTestResult.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/content/docs/developing/testing/Testing_CancellingTest.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/content/docs/developing/testing/Testing_InlineAssertionError.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
95 changes: 85 additions & 10 deletions src/content/docs/developing/testing/configuring.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,15 @@ Use the following steps to create a default configuration for your project:
"cOption": [
"*EVENTF"
]
},
"rucalltst": {
"order": "*API",
"libl": "*CURRENT",
"jobD": "*DFT",
"detail": "*BASIC",
"output": "*ALLWAYS",
"rclRsc": "*NO",
"onFailure": "*ABORT"
}
},
"codecov": {
Expand Down Expand Up @@ -120,7 +129,8 @@ Here is an example with all execution parameters set in the `testing.json` file:
"jobD": "*DFT",
"detail": "*BASIC",
"output": "*ALLWAYS",
"rclRsc": "*NO"
"rclRsc": "*NO",
"onFailure": "*ABORT"
}
}
}
Expand All @@ -132,6 +142,10 @@ Here is an example with all execution parameters set in the `testing.json` file:
<TabItem label="RUCRTRPG" >
|Parameter |Name |Description |Default |
|-------------|------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------|
|`tstPgm` |Test program |Specifies the name of the test program (`*SRVPGM`) to create. By default, this will be set to `&OPENLIB/&OPENMBR` for source members and `&CURLIB/&SHORTNAME` for local files. | |
|`srcFile` |Source file |Specifies the name of the source file containing a member with the same name as `TSTPGM`. By default, this will be set to `&OPENLIB/&OPENSPF` for source members. | |
|`srcMbr` |Source member |Specifies the name of the member containing the source code to be compiled. By default, this will be set to `&OPENMBR` for source members. | |
|`srcStmf` |Source stream file |Specifies the path name of the stream file containing the source code to be compiled. By default, this will be set to `&FULLPATH` for local files. | |
|`text` |Text description |Allows you to enter text that briefly describes the RPGUnit test case and its function. | |
|`cOption` |Compile options |Refer to the `OPTION` parameter in `CRTRPGMOD` command help. |*EVENTF |
|`dbgView` |Debugging views |Refer to the `DBGVIEW` parameter in `CRTRPGMOD` command help. |*SOURCE |
Expand All @@ -153,6 +167,10 @@ Here is an example with all execution parameters set in the `testing.json` file:
<TabItem label="RUCRTCBL" >
|Parameter |Name |Description |Default |
|-------------|------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------|
|`tstPgm` |Test program |Specifies the name of the test program (`*SRVPGM`) to create. By default, this will be set to `&OPENLIB/&OPENMBR` for source members and `&CURLIB/&SHORTNAME` for local files. | |
|`srcFile` |Source file |Specifies the name of the source file containing a member with the same name as `TSTPGM`. By default, this will be set to `&OPENLIB/&OPENSPF` for source members. | |
|`srcMbr` |Source member |Specifies the name of the member containing the source code to be compiled. By default, this will be set to `&OPENMBR` for source members. | |
|`srcStmf` |Source stream file |Specifies the path name of the stream file containing the source code to be compiled. By default, this will be set to `&FULLPATH` for local files. | |
|`text` |Text description |Allows you to enter text that briefly describes the RPGUnit test case and its function. | |
|`cOption` |Compile options |Refer to the `OPTION` parameter in `CRTCBLMOD` command help. |*EVENTF |
|`dbgView` |Debugging views |Refer to the `DBGVIEW` parameter in `CRTCBLMOD` command help. |*SOURCE |
Expand All @@ -178,18 +196,75 @@ Here is an example with all execution parameters set in the `testing.json` file:
|`exclude` |Exclude |Objects to exclude | |
</TabItem>
<TabItem label="RUCALLTST" >
|Setting |Description |Default |
|------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|
|Run Order |Specifies the order for running the test procedures. Useful to check that there is no dependencies between test procedures. |*API |
|Library List |Specifies the library list for executing the specified unit test. |*CURRENT |
|Job Description |Specifies the name of the job description that is used to set the library list, when the `IBM i Testing: Library List` setting is set to `*JOBD`. `*DFT` can be used here to indicate the library of the unit test suite (service program) is searched for job description `RPGUNIT`. |*DFT |
|Report Detail |Specifies how detailed the test run report should be. |*BASIC |
|Create Report |Specifies whether a report is created. |*ALLWAYS |
|Reclaim Resources |Specifies when to reclaim resources. Resources, such as open files, can be reclaimed after each test case or at the end of the test suite. This option is useful if the test suite calls OPM programs, which do not set the `*INLR` indicator. |*NO |
|Parameter |Name |Description |Default |
|-------------|------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|
|`tstPgm` |Test program |Specifies the name of the test program (`*SRVPGM`). By default, this will be set to `&OPENLIB/&OPENMBR` for source members and `&CURLIB/&SHORTNAME` for local files. | |
|`order` |Run Order |Specifies the order for running the test procedures. Useful to check that there is no dependencies between test procedures. |*API |
|`libl` |Library List |Specifies the library list for executing the specified unit test. |*CURRENT |
|`jobD` |Job Description |Specifies the name of the job description that is used to set the library list, when the `IBM i Testing: Library List` setting is set to `*JOBD`. `*DFT` can be used here to indicate the library of the unit test suite (service program) is searched for job description `RPGUNIT`. |*DFT |
|`detail` |Report Detail |Specifies how detailed the test run report should be. |*BASIC |
|`output` |Create Report |Specifies whether a report is created. |*ALLWAYS |
|`rclRsc` |Reclaim Resources |Specifies when to reclaim resources. Resources, such as open files, can be reclaimed after each test case or at the end of the test suite. This option is useful if the test suite calls OPM programs, which do not set the `*INLR` indicator. |*NO |
|`onFailure` |On Failure |Specifies test execution behavior after a failing assertion. |*ABORT |
|`wrapperCmd` |Wrapper Command |Specifies a custom command to wrap the `RUCALLTST` command. | |
</TabItem>
</Tabs>

### Vendor Integration
## Advanced Configuration

Some compile and execution configuration parameters (`TSTPGM`, `SRCFILE`, `SRCMBR`, `SRCSTMF`) are special because the extension typically manages and sets them for you. This is because they can be determined by the extension based on what tests are being compiled and executed. However, to provide maximum flexibility in configuring even these parameters, users can still do so via the `testing.json` file. A simple use case could be that you want to build your source member tests into your current library instead of the library containing the source.

Under the covers, the extension has default values for these special parameters along with supported variables based on whether you are working with local files or source members. The defaults below can be overwritten in the `testing.json` file:

<Tabs>
<TabItem label="Source Members">
```json
{
"rpgunit": {
"rucrtrpg": {
"tstPgm": "&OPENLIB/&OPENMBR",
"srcFile": "&OPENLIB/&OPENSPF",
"srcMbr": "&OPENMBR"
},
"rucalltst": {
"tstPgm": "&OPENLIB/&OPENMBR"
}
}
}
```
Supported Variables:
* `&CURLIB`: Current library, changeable in the **Library List**
* `&OPENLIB`: Library name where the test source member lives
* `&OPENSPF`: Source file name where the test source member lives
* `&OPENMBR`: Name of the test source member

</TabItem>

<TabItem label="Local Files">
```json
{
"rpgunit": {
"rucrtrpg": {
"tstPgm": "&CURLIB/&SHORTNAME"
"srcStmf": "&FULLPATH",
},
"rucalltst": {
"tstPgm": "&CURLIB/&SHORTNAME"
}
}
}
```

Supported Variables:
* `&CURLIB`: Current library, changeable in the **Library List**
* `&NAME`: Name of the test file
* `&SHORTNAME`: Shortened name of the test file (based on rules [here](https://ibm.github.io/sourceorbit/#/./pages/general/rules?id=long-file-names))
* `&FULLPATH`: Full path of the test file on the remote system
</TabItem>
</Tabs>


## Vendor Integration
The `wrapperCmd` parameter in the `RUCRTRPG`/`RUCRTCBL`/`RUCALLTST` specifications are not actual parameters of the RPGUnit commands, but rather are provided by the extension to allow wrappering them. This is primarily used when integrating RPGUnit with other vendor tools.

**Example `testing.json` configuration**:
Expand Down
9 changes: 8 additions & 1 deletion src/content/docs/developing/testing/overview.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,14 @@ The following extensions can be installed from the Visual Studio Code Marketplac

### 2. RPGUnit

The `RPGUNIT` library ***(`5.1.0-beta.005` minimum)*** must be be installed on your IBM i in order to leverage the framework and assertions that it provides. It can be installed easily via the extension using the steps below which is the recommended approach or manually using the instructions [here](https://irpgunit.sourceforge.io/help/).
The `RPGUNIT` library must be installed on your IBM i in order to leverage the framework and assertions that it provides. It can be installed easily via the extension using the steps below which is the recommended approach or manually using the instructions [here](https://irpgunit.sourceforge.io/help/).

The required RPGUnit version depends on which version of the **IBM i Testing** extension you have installed:

| Extension Version | Required RPGUnit Version |
|-------------------|----------------------------|
| ≤ 1.2.3 | 5.1.0 |
| 1.3.2 | ≥ 6.0.0 |

#### Preconditions
RPGUnit requires OS400 7.5 for the latest features. It can also be installed on 7.4 and 7.3 if the following PTFs have been installed:
Expand Down
18 changes: 18 additions & 0 deletions src/content/docs/developing/testing/running.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ For tests to be discovered by the extension, you must first [connect to an IBM i

![](Testing_TestExplorer.png)

### Running Tests

From here you can hover on any test case, test suite, folder, library, or source file and use the inline **Run Test** button. Using this action will start the following process:

<Steps>
Expand All @@ -32,6 +34,16 @@ From here you can hover on any test case, test suite, folder, library, or source
![](Testing_GutterActions.png)
</Aside>

### Cancelling Tests

The current test deployment, compilation, and execution can be cancelled at anytime by navigating to the **Test Explorer** view and using the **Cancel Test Run** button.

![](Testing_CancellingTest.png)

Any test suites which were executed before cancellation will still have their test results shown in the **Test Results** view. Cancelled test suites will be marked accordingly and a total count of cancelled test suites and cases will be included in the test summary.

![](Testing_CancelledTestResult.png)

## Test Results

After running any test, the results will be displayed in detail in the **Test Results** view. This will include the number of passed, failed, and errored test cases along with any relevant messages. The results can also be viewed at a high level in the `Testing` view itself where the icon will be updated to reflect the status of the test. Failures associated with a specific line will also be displayed in the editor.
Expand All @@ -42,6 +54,12 @@ After running any test, the results will be displayed in detail in the **Test Re

![](Testing_TestResults.png)

### Diagnosing Errors

When there is an assertion failure, you can view the actual and expected results (depending on the type of assertion) along with the failure message in the **Test Results** view. In addition, you can scroll through the editor and see the same message inline right next to the failing assertion. Upon clicking on this message, you will get an expanded inline view of the error which also includes a diff of the actual and expected result along with the stack trace.

![](Testing_InlineAssertionError.png)

## Code Coverage

You can also run the tests with code coverage to see which lines of code were executed during the test run. This is useful for identifying untested code paths and ensuring that your tests cover all critical areas of your application.
Expand Down