External DirSuite Mk3 #127
Conversation
This replace DirSuite Mk2 with external DirSuite Mk3. See hrj#125 for background information. Major changes between Mk2 and Mk3: - Test are discovered based on exec file, not by conf files. - Args-files are gone. If there are args for test case, then those are defined in exec-files. - Mk3 scanning is recursive by default. - Mk3 supports also Glob's. With glob `something/**.exec` includes all subdirectories under `something/` - Mk3 is more strict against human errors (e.g. Suite is aborted if pattern doesn't find anything, or if basedir is wrong, etc.). - It is possible to run tests without any agruments, even without conf-file - It is possible to run multiple steps per test - It is possible to run multistep dual assertions (success + failure assertion) Changes: - tests/**.exec: Added exec-file for every conf-file (previous "test case") - tests/**.args: removed args and content was moved to exec, if there was an args-file for test case - cli/src/test/../DirSuite.scala: removed Mk2 DirSuite - cli/src/tets/../TestComparator.scala: removed Mk2 DirSuite - cli/src/test/../DefaultArgsDirSuite.scala: Default Argument provider for Abandon test cases. - cli/src/test/../CliTest.scala: Use new Mk3 DirSuite, and change all old Regex to Glob - project/Dependencies.scala: Add library fi.sn127.utils.testing (rel:0.6.0 - DirSuite Mk3) - cli/build.sbt: Add test time dependency to DirSuite
|
One important thing to validate / review, is that test counts are same. I have double check conf vs. exec count (they are same, small.conf and complete.conf are not part of testsuite), and also sbt reported test counts are same. But this still worth to check, and in future to remember that tests are triggered by exec-file, not by conf-file. |
| @@ -0,0 +1,2 @@ | |||
| # format: exec | |||
| exec:--filter;asdf=2016-02-01; | |||
There was a problem hiding this comment.
Does the library prepend a hyphen automatically? If so, that seems inflexible. What if an option needs to be added which doesn't start with a hyphen?
There was a problem hiding this comment.
No it doesn't. It feeds arguments excatly as they are between ';' chars.
It is testing filter, which has command line in general form:
"--filter" "date=2017-01-01", but in this case, because it's an error test, it is using incorrect filter name. It has excatly same args than what it had with *.args file. So it is ok. =)
Here is link to documentation of exec-file:
https://github.com/sn127/utils/blob/master/docs/dirsuite.md#exec-file-format
There was a problem hiding this comment.
Oh, I finally understood. I was counting the - shown by the diff as part of the argument 🤦♂️
Looks correct now.
This replace DirSuite Mk2 with external DirSuite Mk3. See #125
for background information.
Major changes between Mk2 and Mk3:
then those are defined in exec-files.
something/**.execincludesall subdirectories under
something/pattern doesn't find anything, or if basedir is wrong, etc.).
conf-file
(success + failure assertion)
Changes:
if there was an args-file for test case
for Abandon test cases.
and change all old Regex to Glob
fi.sn127.utils.testing (rel:0.6.0 - DirSuite Mk3)