Skip to content
Find file
Fetching contributors…
Cannot retrieve contributors at this time
375 lines (282 sloc) 10.4 KB

Legends

For all items in MUST / SHOULD / COULD / WOULD BE NICE, the following legends are defined.

LegendMeans
[d]Done - implemented
[o]Ongoing - currently working on
[t]To do - waiting to be started

MUST

[d] Execution of unit test

A test function, defined by the function name ending in ‘_TEST’ contains a variable assignment to the variable ‘expect’ in the UT namespace (#.UT.expect) which defines the expected return value of the function.

∇ Z ← one_plus_one_TEST
  #.UT.expect ← 2
  Z ← 1 * 1
∇

Test functions are executed using UT.run with the test function name as the argument

#.UT.run 'one_plus_one_TEST'

Upon execution of the test function (one_plus_one_TEST), the returned value (in Z) is checked against the expecte value (in UT_expected).

If the returned values does not match the expected value, the test is considered as failed, appropriate text is displayed, clearly outlining the Expected value and the Actual value, along with the name of the test.

FAILED: one_plus_one_TEST
Expected
   2
Got
   1

If the returned value of the function matches the expected value, the test is considered as passed and a passed message is printed

UT.run 'one_plus_one_TEST'
Passed

[d] Execution of multiple unit tests

It shall be possible to execute multiple unit tests from an array of unit tests by name

Tests ← 'a_TEST' 'b_TEST' 'c_TEST' 'd_TEST'
#.UT.run Tests

The result of the execution of tests shall display the result of each test as if executd by #.UT.run There shall also be an agregated result displayed on the screen at the end of the execution.

Test execution result
  ⍋ Passed: 20
  ⍒ Failed: 3

[d] Test Execution Robustness

A crashing test must not cause subsequent tests not to be executed. Crashing test cases are reported in the final result as ‘Crashed’ when executing a list of testcases.

⍋  Passed: 4
⍟ Crashed: 2
⍒  Failed: 1

On single test case execution, the test will be reported as failed with the following output, where ${DIAGNOSTIC_MESSAGE} is replaced with the ⎕DM of the crash

CRASHED: one_plus_one_TEST
Expected
   2
Got
${DIAGNOSTIC_MESSAGE}

[d] Expect Exception in Unit Test

It shall be possible to epxress that the test will result in an exception and to set what Diagnostic Message is to be expected

∇ exception_generating_TEST
    #.UT.exception ← 'DOMAIN_ERROR'
    Z ← 1 ÷ 0
∇

Upon execution of such a test, the test is considered successfull if the test function generates an exception which has a ⎕DM matching the #.UT.exception text.

On failure, a message is displayed, clearly indicating the expected and actual diagnostic messages.

FAILED: failing_error_TEST
Expected
    ┌→───────────┐
    │DOMAIN ERROR│
    └────────────┘
Got
    ┌→─────────┐
    │RANK ERROR│
    └──────────┘

On success, the output is ‘Passed’.

SHOULD

[d] Automatic execution of all Test Functions from file

Having multiple Single line tests in a file, it must be possible to execute them all in one go.

UT.run '/path/to/File.dyalog'

During the execution of the test cases, every passed/failed/crashed test is displayed as in the single test/multiple test execution.

At the end of the execution, an aggregated result is printed to the screen. The amount of Passed/Crashed/Failed testcases displayed.

/path/to/File.dyalog tests
⍋   Passed: 10
⍟  Crashed: 3
⍒   Failed: 0

Executed tests include all _TEST functions (functions whose name ends in _TEST). Such as ‘this_TEST’.

Arrays containing testcases will not be executed.

[d] Generate HTML page with coverage result of Unit Tests

It shall be possible to generate a coverage report of selected functions as a result of unit test execution.

Coverage report generation is requested by creating an instance of the UTcover class

Conf ← ⎕NEW UTcover

Setting the cover page generation output directory property

Conf.pages ← '/home/APL/coverage'

And setting the array of functions to cover property, the functions to cover are targeted through the corresponding namespace and function. In the example below, we are targeting the plus_function in the Example namespace.

Conf.cover ← ⊂ '#.Example.plus_function'

This instance is then given as left argument to the ordinary UT.run function.

Conf ← ⎕NEW UTcover
Conf.pages ← '/home/APL/coverage'
Conf.cover ← ⊂ '#.Example.plus_function'
Conf UT.run 'one_plus_one_TEST'

The resulting HTML page will be generated to a file in the path defined by the user through the parameter ‘UT.pages’. The name of the page will be determined by the type of Tests being executed (the right argument of UT.run).

Coverage onPage Output inPage Name
Conf #.UT.run ‘a_TEST’Conf.pagesa_TEST_coverage.html
Conf #.UT.run ‘a_TEST’ ‘b_TEST’Conf.pageslist_coverage.html
Conf #.UT.run ‘/t/File.dyalog’Conf.pagesFile_coverage.html

Thus, the example

Conf ← ⎕NEW UTcover
Conf.pages ← '/home/APL/coverage'
Conf.cover ← ⊂ '#.Example.timetable_selector'
Conf UT.run 'out_of_bound_TEST'

Passed

Will generate a coverage result for the function timetable_selector, the result coverage page is written to the file ‘/home/APL/coverage/out_of_bound_TEST.html’

[d] Cover Result Page Time Stamp

The Coverage result page will have a timestamp indicating when the page was generated.

The timestam will have the format

YYYY-MM-DD HH:mm:SS

and be located at the bottom of the page, together with the text

Page generated: 

So, an example page will have the following text at the bottom of the page.

Page generated: 2013-2-19 | 19:11:36

The reason to put this info at the bottom of the page, is that the most interesting part is the coverage.

COULD

[d] Run all unit test files in the directory passed to run

When executing #.UT.run ${PATH} if ${PATH} is a directory, then find all APLUnit test files and execute them in order.

UT.run '/path/to/directory/'

APLUnit test files are SALT scripts whone name ends in ‘_tests.dyalog’.

Currently this only works for *nix as it used the OS level test -d

[d] Cover all functions in a namespace by only passing the namespace

When setting the coverage configuration, it should be possible to only passing the namespace to the cover property of the Cover conf object.

Conf ← ⎕NEW UTcover
Conf.pages ← '/home/APL/coverage'
Conf.cover ← ⊂ '#.Example'

Running the Unit Tests from the Example_tests.dyalog file should now generate a full coverage report of all functions defined in #.Example

[d] It shall be possible to express non equality in expected result

When writing Unit Tests, it shall be possible to express a non-equality for the result. This is done by exchanging the expect with nexpect.

∇ Z ← dyadic_iota_will_not_give_array_TEST
   Z ← 1 2 ⍳ 1
   #.UT.nexpect ← 1
∇

The non equality shall behave the same as the equality, except that the condition for success is the actual argument NOT being equal to the result.

[d] Result from #.UT.run should be shy

When executing multiple tests with #.UT.run, the resulting output on the screen should be suppressed

.....
Passed
Passed
Passed
-----------------------------------------
../numbername_tests.dyalog tests
    ⍋  Passed:  45
    ⍟ Crashed:  0
    ⍒  Failed:  0

Without the currently following

#.UT.[UTresult]  #.UT.[UTresult]  #.UT.[UTresult]  #.UT.[UTresult]  #.UT.[U
     Tresult]  #.UT.[UTresult]  #.UT.[UTresult]  #.UT.[UTresult]  #.UT.[UTr
     esult]  #.UT.[UTresult]  #.UT.[UTresult]  #.UT.[UTresult]  #.UT.[UTres
     ult]  #.UT.[UTresult]  #.UT.[UTresult]  #.UT.[UTresult]  #.UT.[UTresul
     ....

[d] Generic configuration array passed to UT.run

Instead of using a specific Coverage configuration as the left argument to UT.run, it should accept an unordered keyed list (proplist) carrying the configuration parameters. This allows extension to be performed easier.

This change will break backwards compatibility with the currently existing tests. It is important that the merge point labels this clearly.

C ← ⍬
C,← ⊂('cover_out' '/home/APL/coverage/')
C,← ⊂('cover_target' ('#.Demo.this_TEST' '#.Demo.something_TEST'))
C,← ⊂('skip' ⊂'#.Demo.count_zero_comments_from_no_input_TEST')
C UT.run 'out_of_bound_TEST'

[d] Fast reload of dyalog SALT scripts from within APL Environment

It shall be possible to quickly reload all dyalog SALT scripts from within the Dyalog environment without too much ceremony. For this to work, a global configuration parameter is set, pointing to the root path of the Dyalog Application.

#.UT.appdir ← './path/to/app'

There is an implicit assumption here, that all Dyalog applications following this style has two directories, one ‘src/’ and one ‘test/’, found under ‘./path/to/app’

./path/to/app/src/
./path/to/app/test/

within these directories are one or more dyalog SALT scripts which can be loaded using ⎕SE.SALT.Load. All tests are supposed to be under test/ and all implementation source is assumed to be under src/

Upon having APLUnit already loaded in the system, running the usual test commands will always start by first loading all the Dyalog salt scripts into the current namespace first, and then proceeding as usual.

The path can be relative and absolute, just as passed to SALT. To disable this behavior, ⍬ can be assigned.

#.UT.appdir ← ⍬
Something went wrong with that request. Please try again.