CLIApp Testrunner Mk2 #120
Conversation
…ite)
This is a combination of TestRunner and FunSuiteLike as DirSuite trait.
This lets you to write testrunner tests normally as scalatest's
Funsuite tests, ignore them, and most importantly, use normal FunSuite
asserts. Especially this is handy with assertThrows, which lets us
to test error paths and constraints with specific asserts.
DirSuite trait provides following extra testrunner-methods:
- testDirSuite
- ignoreDirSuite
DirSuite tests can be defined as:
testDirSuite("../tests", ".*/sclT[0-9]+(-.*)$") { args: Array[String] =>
assertResult(CLIApp.SUCCEEDED){
CLIApp.mainStatus(args)
}
}
testDirSuite("../tests", ".*/SettingsError$") { args: Array[String] =>
assertThrows[SettingsError]{
CLIApp.run(args)
}
}
There are also changes to main app outside test-scope to better support
testability and fix few warts.
These tests found one corner case input handling bug (bugsInputError/pe01.ledger), which is currently ignored, and one AssertionError (AssertionError/ae01.ledger) with invalid input.
|
Hey, I wonder why this error happens now: (I am travelling so I am not able to review in detail) |
|
=), I wondered same thing when I saw it first time, but everything is good. It is not filter-test which is producing AssertError, but the next test, which is testing that CLIApp errors (and producing AssertionError). So it happens because funtest prints first test output, and then the name of current test. So filter test is passing, it's result is printed (in green) then is started AssertError test, which produce that line, and lastly name of that AssertError test (it is supposed to error, only problem is that it is AssertError). The actual test-result line for that AssertionError is on line Travis-166#L922 So everything is good and all tests are passing. Scalatest prints test output that way, because it has to first execute test, so it can report correctly test output with single line. It is kind of confusing, especially with those error tests. Maybe there is an option to print Starting test ... Ending test - Passing or something similar, dunno. On the other hand, when test is falling there is big red text which is saying that test is falling and all tests arguments are also printed on output, so test failure is really obvious when there is one. Error path output code: |
|
Cool; thanks for the explanation! Merged. By the way, what does |
|
Thanks, Mk2 is kind of pun - all fine products have mark II variant. =D |
This PR implements cli-testrunner Mk2 which makes TestRunner behave truly like Scalatest's FunSuite (all asserts available).
This is done by extending FunSuiteLike with TestRunner as trait. It is now possible to write testrunner tests normally as scalatest's Funsuite tests, ignore them, and most importantly, use normal FunSuite asserts. Especially this is handy with assertThrows, which lets us to test error paths and constraints with specific asserts. DirSuite implementation: 7caad6b
Example usage of DirSuite: 7f5c91d
These added error path tests found one corner case input handling bug (bugsInputError/pe01.ledger), which is currently ignored, and one AssertionError (AssertionError/ae01.ledger) with invalid input.
Travis has nice example of test run's output: Travis-166