Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Tree: ccce1a654d
Fetching contributors…

Cannot retrieve contributors at this time

409 lines (318 sloc) 11.029 kB

Legends

For all items in MUST / SHOULD / COULD / WOULD BE NICE, the following legends are defined.

LegendMeans
[d]Done - implemented
[o]Ongoing - currently working on
[t]To do - waiting to be started

MUST

[d] Automatic execution of single line Unit Test

A test - written on one line, with the expected value on the left hand side (LHS) of a special unit test equality operator and the function to be called on the right hand side (RHS).

first_TEST ← { LHS UT.eq RHS }

This single line Unit Test can then be executed

UT.run first_TEST

Upon execution and evaluation of RHS, if the LHS does not match the returned value from the RHS: the test is to be considered as failed, appropriate text is to be displayed - clearly outlining the Expected value and the Actual value - along with the textual representation of the test.

FAILED: first_TEST ← { LHS UT.eq RHS }
Expected 
  ┌→────────┐
  │   ┌→──┐ │
  │ 1 │2 3│ │
  │   └~──┘ │
  └∊────────┘
Got
  ┌→────┐                                                               
  │1 2 3│
  └~────┘

If the LHS matches the RHS, the test is to be considered as successfull and no output is generated. The execution of the test must return 1 on success and 0 on failure.

Execution ResultSide effectReturn value
LHS ≡ RHSnone1
LHS ≢ RHSPrint expected and actual values0

It can be wise to set and follow a fixed naming convention. For example, all variables with the suffix ‘_TEST’ are a test.

My_TEST

[d] Automatic execution of multi-line Unit Test

A test written across several lines. The match operator is used within the function definition.

∇ Z ← my_TEST
  ...
  Z ← LHS UT.eq RHS
∇

Failure and Success behavior identical to the single line Unit Test behavior, except that for the failure case, the whole function is not shown. It will truncate the displayed function, showing the first and last lines.

FAILED: ∇ Z ← my_TEST ... Z ← LHS UT.eq RHS ∇
Expected 
  ┌→────────┐
  │   ┌→──┐ │
  │ 1 │2 3│ │
  │   └~──┘ │
  └∊────────┘
Got
  ┌→────┐                                                               
  │1 2 3│
  └~────┘

[d] Automatic execution of all Unit Tests from file

Having multiple Single line tests in a file, it must be possible to execute them all in one go.

(Passing Failing) ← UT.run_file File.dyalog

During the execution of the test cases, every failed test is displayed as in the single line Unit Test execution.

FAILED: my_TEST ← { LHS UT.eq RHS }
Expected 
  ┌→────────┐
  │   ┌→──┐ │
  │ 1 │2 3│ │
  │   └~──┘ │
  └∊────────┘
Got
  ┌→────┐                                                               
  │1 2 3│
  └~────┘

At the end of the execution, an aggregated result is returned, with the amount of Failed and the amount of Passed testcases in a vector. This shall also be printed for quick readability.

File.dyalog unit tests
⍋ 152 PASSED  
⍒ 2   FAILED 
(Passing Failing) ← UT.run_file File.dyalog

SHOULD

[d] Assertion operator to expect events with specific ⎕EN

It shall be possible to assert that the executed function in the TEST will result in an exception with a given ⎕EN.

This is achieved by setting the UT.EN variable at the beginning of the test (or before the exception is expected).

exception_TEST ← { UT.EN ← 2 ⋄ ⍳ }

the multi line version would be similar

∇ multi_line_exception_TEST
      UT.EN ← 6
      some_function_call
∇

[t] Coverage report from Unit Test execution

There shall be a setting indicating that a coverage report is to be generated as a result of the Unit Test execution. Recommended is that the output directory can be specified.

UT.coverage 1
UT.out 'coverage'

but that the coverage report is as follows: An index.html file, showing the agregated total coverage, and the specific coverage for each dyalog module, with links to the specific dyalog module cover reports.

coverage/
 |- index.html
 |- a.html
 +- b.html

a.html and b.html are dyalog module specific coverage reports. The information displayed on the pages shall be at least as follows

index.html

  1. Total project (dyalog files) coverage in percentage
  2. Total project covered lines
  3. Total project uncovered lines
  4. Link to each dyalog specific coverage page
  5. For each link
    • name of the file
    • coverage in percentage
    • covered lines
    • uncovered lines

dyalog specific coverage page

The X.dyalog file specific coverage page will show the source code in a page, and the covered lines are marked as black, while the uncovered lines are red.

Additionally, there will be a rightmost column with numbering showing the amount of times every line is executed.

The page itself shall have at the top

  1. The total coverage %
  2. The amount of covered lines
  3. The amount of uncovered lines

[d] Test exception robustness

Exceptions in one test shall not disrupt the execution of other tests. As such, the tests shall be isolated entities.

When running multiple tests, if a test fails to execute due to an exception, it shall be marked as ‘exception’ this result shall also be shown in the returned array.

File.dyalog unit tests
⍋ 152 PASSED  
⋄ 3   EXCEPTION
⍒ 2   FAILED 

(Passing Exception Failing) ← UT.run_file File.dyalog 

Every exception:d test execution shall display the exception as

EXCEPTION: first_TEST ← { LHS UT.eq RHS }
- some additional information here -

The return value of a test failing due to exception is the event number.

[o] Test Groups

A test group is an array of test function names. It shall be possible to create test groups and execute only a certain group. All variables with the suffix ‘_GROUP’ are to be interpreted as a group definition.

my_GROUP ← (first_TEST ... last_TEST)

Groups are executed with the UT.run_group function

UT.run_group test_GROUP
Group test_GROUP
⍋ 10 PASSED  
⋄ 0  EXCEPTION
⍒ 0  FAILED 

It shall also be possible to execute a specific group within a File using the UT.run_group_file function by naming the group in the left argument.

target_GROUP UT.run_group_file File.dyalog

Output result shall for both cases show that this was a group

(Passing Exception Failing) ← Test_GROUP UT.run_grup File.dyalog
Group Test_GROUP in File.dyalog
⍋ 10 PASSED  
⋄ 0  EXCEPTION
⍒ 0  FAILED  

[t] Test Suites

A test suite is an array of test groups. It shall be possible to organize tests into Test Suites and execute only a certain suite. It can be wise to set and follow a fixed naming convention. For example, all variables with the suffix ‘_SUITE’ are a suite.

My_SUITE
Test_SUITE ← (first_GROUP .. last_GROUP)
UT.run_suite Test_SUITE

It shall also be possible to target a specific suite within a File

Test_SUITE UT.run_suite File.dyalog

Output result shall for both cases show thart this was a suite being executed that contains groups.

Suite Test_SUITE in File.dyalog
  Group first_GROUP
   ⍋ 10 PASSED  
   ⋄ 0  EXCEPTION
   ⍒ 0  FAILED 
  Group second_GROUP
   ⍋ 13 PASSED  
   ⋄ 0  EXCEPTION
   ⍒ 1  FAILED 
--------------------------
 ⍋ 23 PASSED
 ⋄ 0  EXCEPTION
 ⍒ 1  FAILED
(Passing Exception Failing) ← Test_SUITE UT.run_suite File.dyalog

[t] Init and End per Test / Group / Suite

For Tests, Groups and Suites, it shall be possible to specify an Initialization function, and an End function that is executed before, and after the Test / Group / Suite.

The Init and End functions are niladic and dyadic, such that the result of the Init evaluation is passed onto the left argument of the End function and the test result of the Single test / Group / Suite is passed as the right argument of the End function.

X_init 
  |
  ├-→ Test/Group/Suite execution
  │       │
  │     result
  │       │
  ∇ X_end ∇

The relation between Test object and Initialization and End functions shall be as follows.

Unit Test ObjectNameInit function nameEnd function name
Single LineX_TESTX_TEST_initX_TEST_end
Multi lineX_TESTX_TEST_initX_TEST_end
GroupX_GROUPX_GROUP_initX_GROUP_end
SuiteX_SUITEX_SUITE_initX_SUITE_end

If defined, the Init and End functions must be able to execute, invariably of test success / fail or skip.

COULD

[t] Collect and show execution time

It shall be possible to configure if the execution time of each TEST/GROUP/SUITE is to be collected and reported.

UT.runtime 1

Example of wanted output for single TEST execution

RunTime ← UT.run first_TEST
PASSED - 0.01 Seconds

Example of wanted output for single GROUP execution

Group Test_GROUP in File.dyalog
⍋ 10 PASSED  
⋄ 0  EXCEPTION
⍒ 0  FAILED 
○ 0.02 Seconds
(Passing Exception Failing RunTime) ← Test_GROUP UT.run_group File.dyalog

Example of wanted output for single SUITE execution

Suite Test_SUITE in File.dyalog
  Group first_GROUP
   ⍋ 10 PASSED  
   ⋄ 0  EXCEPTION
   ⍒ 0  FAILED 
   ○ 0.01 Seconds
  Group second_GROUP
   ⍋ 13 PASSED  
   ⋄ 0  EXCEPTION
   ⍒ 1  FAILED 
   ○ 0.02 Seconds
--------------------------
 ⍋ 23 PASSED
 ⋄ 0  EXCEPTION
 ⍒ 1  FAILED
 ○ 0.03 Seconds
(Passing Exception Failing RunTime) ← Test_SUITE UT.run_suite File.dyalog

WOULD BE NICE

Jump to Line
Something went wrong with that request. Please try again.