Some tests are not referenced in a test suite

[divdavem]

This pull request adds the following tests in the corresponding suite and fixes them (so that the build is successful):

• test.aria.templates.issue142.HtmlStyleTemplateTestCase
• test.aria.templates.issue279.ButtonSpacingTestCase

[piuccio]

I've integrated the changes with commit 9956682 but we should have a task that detects orphan tests.
I'm thinking about a grunt task...

[jakub-g]

I'm wondering what would be the best solution for this.

We can't rely on file names because the majority of the test cases doesn't have any Test part in their name.

We could list all .js files in test dir, and filter only those which contain the magic words to be the test case (because some 30% are tpl scripts and other util JS files used by tests etc. so we need to exclude them).

The best way to check for this would be to look at $extends because each test case/suite extends, directly or not, the base test case/suite file.

The problems are with indirect inheritance.

I've quickly analyzed the contents of the folder and we have test cases/suites that extend from:

$extends : 'aria.jsunit.TestCase',
$extends : 'aria.jsunit.RobotTestCase',
$extends : 'aria.jsunit.TemplateTestCase',
$extends : "aria.jsunit.ModuleCtrlTestCase",
$extends : "aria.jsunit.WidgetTestCase",

$extends : "test.aria.widgets.container.issue80.shared.BindableSizeTestCase",
$extends : "test.aria.widgets.form.multiselect.issue312.Common",

and all test suites extend from

$extends : "aria.jsunit.TestSuite",

Regarding test cases, the last two are families of very specialized per-issue test cases inheriting from the base test cases and differing only in widgets under tests.

So if we renamed the last file say to Issue312TestCase then we could analyze the contents of the files looking for match for $extends : *TestCase match (either with regex or AST).

Then we need to obtain all the tests added by this.addTests( ) in the suites and compare the two arrays.

[jakub-g]

I'm in favor of wildcard match, because we have e.g. aria.jsunit.MultiSelectTemplateTestCase in CC, and we may potentially add new in the future, so a bulletproof solution will be preferred.

BTW we should add some marker to BindableSizeTestCase and issue312.Common saying that those classes are abstract test cases ($abstract : true ?) and exclude them.

[jakub-g]

We can also go the other way and rename all test files to end with Test / TestCase - this will at least not need to read 300 files from disk to do the check...

[piuccio]

The best would be to remove the TestSuite altogether, they are just a duplication of information.
The test runner should be able to understand which files are test cases and run them all.

[jakub-g]

I'm closing this as this is an old specific pull request, I opened a new issue #924 for tracking the general issue (I don't like pull request to be visible as open while they were merged).