For all items in MUST / SHOULD / COULD / WOULD BE NICE, the following legends are defined.
| Legend | Means |
|---|---|
| [d] | Done - implemented |
| [o] | Ongoing - currently working on |
| [t] | To do - waiting to be started |
A test - written on one line, with the expected value on the left hand side (LHS) of a special unit test equality operator and the function to be called on the right hand side (RHS).
first_TEST ← { LHS UT.eq RHS }
This single line Unit Test can then be executed
UT.run first_TEST
Upon execution and evaluation of RHS, if the LHS does not match the returned value from the RHS: the test is to be considered as failed, appropriate text is to be displayed - clearly outlining the Expected value and the Actual value - along with the textual representation of the test.
FAILED: first_TEST ← { LHS UT.eq RHS }
Expected
┌→────────┐
│ ┌→──┐ │
│ 1 │2 3│ │
│ └~──┘ │
└∊────────┘
Got
┌→────┐
│1 2 3│
└~────┘
If the LHS matches the RHS, the test is to be considered as successfull and no output is generated. The execution of the test must return 1 on success and 0 on failure.
| Execution Result | Side effect | Return value |
|---|---|---|
| LHS ≡ RHS | none | 1 |
| LHS ≢ RHS | Print expected and actual values | 0 |
It can be wise to set and follow a fixed naming convention. For example, all variables with the suffix ‘_TEST’ are a test.
My_TEST
A test written across several lines. The match operator is used within the function definition.
∇ Z ← my_TEST ... Z ← LHS UT.eq RHS ∇
Failure and Success behavior identical to the single line Unit Test behavior, except that for the failure case, the whole function is not shown. It will truncate the displayed function, showing the first and last lines.
FAILED: ∇ Z ← my_TEST ... Z ← LHS UT.eq RHS ∇ Expected ┌→────────┐ │ ┌→──┐ │ │ 1 │2 3│ │ │ └~──┘ │ └∊────────┘ Got ┌→────┐ │1 2 3│ └~────┘
Having multiple Single line tests in a file, it must be possible to execute them all in one go.
(Passing Failing) ← UT.run_file File.dyalog
During the execution of the test cases, every failed test is displayed as in the single line Unit Test execution.
FAILED: my_TEST ← { LHS UT.eq RHS }
Expected
┌→────────┐
│ ┌→──┐ │
│ 1 │2 3│ │
│ └~──┘ │
└∊────────┘
Got
┌→────┐
│1 2 3│
└~────┘
At the end of the execution, an aggregated result is returned, with the amount of Failed and the amount of Passed testcases in a vector. This shall also be printed for quick readability.
File.dyalog unit tests ⍋ 152 PASSED ⍒ 2 FAILED (Passing Failing) ← UT.run_file File.dyalog
It shall be possible to assert that the executed function in the TEST will result in an exception with a given ⎕EN.
This is achieved by setting the UT.EN variable at the beginning of the test (or before the exception is expected).
exception_TEST ← { UT.EN ← 2 ⋄ ⍳ }
the multi line version would be similar
∇ multi_line_exception_TEST
UT.EN ← 6
some_function_call
∇
There shall be a setting indicating that a coverage report is to be generated as a result of the Unit Test execution. Recommended is that the output directory can be specified.
UT.coverage 1 UT.out 'coverage'
but that the coverage report is as follows: An index.html file, showing the agregated total coverage, and the specific coverage for each dyalog module, with links to the specific dyalog module cover reports.
coverage/ |- index.html |- a.html +- b.html
a.html and b.html are dyalog module specific coverage reports. The information displayed on the pages shall be at least as follows
- Total project (dyalog files) coverage in percentage
- Total project covered lines
- Total project uncovered lines
- Link to each dyalog specific coverage page
- For each link
- name of the file
- coverage in percentage
- covered lines
- uncovered lines
The X.dyalog file specific coverage page will show the source code in a page, and the covered lines are marked as black, while the uncovered lines are red.
Additionally, there will be a rightmost column with numbering showing the amount of times every line is executed.
The page itself shall have at the top
- The total coverage %
- The amount of covered lines
- The amount of uncovered lines
Exceptions in one test shall not disrupt the execution of other tests. As such, the tests shall be isolated entities.
When running multiple tests, if a test fails to execute due to an exception, it shall be marked as ‘exception’ this result shall also be shown in the returned array.
File.dyalog unit tests ⍋ 152 PASSED ⋄ 3 EXCEPTION ⍒ 2 FAILED (Passing Exception Failing) ← UT.run_file File.dyalog
Every exception:d test execution shall display the exception as
EXCEPTION: first_TEST ← { LHS UT.eq RHS }
- some additional information here -
The return value of a test failing due to exception is the event number.
A test group is an array of test functions. It shall be possible to create test groups and execute only a certain group. It can be wise to set and follow a fixed naming convention. For example, all variables with the suffix ‘_GROUP’ are a group.
My_GROUP
Test_GROUP ← (first_TEST ... last_TEST) UT.run_group Test_GROUP
It shall also be possible to target a specific group within a File
Test_GROUP UT.run_group_file File.dyalog
Output result shall for both cases show that this was a group
Group Test_GROUP in File.dyalog ⍋ 10 PASSED ⋄ 0 EXCEPTION ⍒ 0 FAILED (Passing Exception Failing) ← Test_GROUP UT.run_grup File.dyalog
A test suite is an array of test groups. It shall be possible to organize tests into Test Suites and execute only a certain suite. It can be wise to set and follow a fixed naming convention. For example, all variables with the suffix ‘_SUITE’ are a suite.
My_SUITE
Test_SUITE ← (first_GROUP .. last_GROUP) UT.run_suite Test_SUITE
It shall also be possible to target a specific suite within a File
Test_SUITE UT.run_suite File.dyalog
Output result shall for both cases show thart this was a suite being executed that contains groups.
Suite Test_SUITE in File.dyalog Group first_GROUP ⍋ 10 PASSED ⋄ 0 EXCEPTION ⍒ 0 FAILED Group second_GROUP ⍋ 13 PASSED ⋄ 0 EXCEPTION ⍒ 1 FAILED -------------------------- ⍋ 23 PASSED ⋄ 0 EXCEPTION ⍒ 1 FAILED (Passing Exception Failing) ← Test_SUITE UT.run_suite File.dyalog
For Tests, Groups and Suites, it shall be possible to specify an Initialization function, and an End function that is executed before, and after the Test / Group / Suite.
The Init and End functions are niladic and dyadic, such that the result of the Init evaluation is passed onto the left argument of the End function and the test result of the Single test / Group / Suite is passed as the right argument of the End function.
X_init | ├-→ Test/Group/Suite execution │ │ │ result │ │ ∇ X_end ∇
The relation between Test object and Initialization and End functions shall be as follows.
| Unit Test Object | Name | Init function name | End function name |
|---|---|---|---|
| Single Line | X_TEST | X_TEST_init | X_TEST_end |
| Multi line | X_TEST | X_TEST_init | X_TEST_end |
| Group | X_GROUP | X_GROUP_init | X_GROUP_end |
| Suite | X_SUITE | X_SUITE_init | X_SUITE_end |
If defined, the Init and End functions must be able to execute, invariably of test success / fail or skip.
It shall be possible to configure if the execution time of each TEST/GROUP/SUITE is to be collected and reported.
UT.runtime 1
Example of wanted output for single TEST execution
RunTime ← UT.run first_TEST PASSED - 0.01 Seconds
Example of wanted output for single GROUP execution
Group Test_GROUP in File.dyalog ⍋ 10 PASSED ⋄ 0 EXCEPTION ⍒ 0 FAILED ○ 0.02 Seconds (Passing Exception Failing RunTime) ← Test_GROUP UT.run_group File.dyalog
Example of wanted output for single SUITE execution
Suite Test_SUITE in File.dyalog Group first_GROUP ⍋ 10 PASSED ⋄ 0 EXCEPTION ⍒ 0 FAILED ○ 0.01 Seconds Group second_GROUP ⍋ 13 PASSED ⋄ 0 EXCEPTION ⍒ 1 FAILED ○ 0.02 Seconds -------------------------- ⍋ 23 PASSED ⋄ 0 EXCEPTION ⍒ 1 FAILED ○ 0.03 Seconds (Passing Exception Failing RunTime) ← Test_SUITE UT.run_suite File.dyalog