Add command line option to run only one test/testsuite

[dpreussner]

It would be great if a command line option could be added to tell attester to run only one test/testsuit instead of only being able to run the whole test suite.

Something like this:
attester -test mytest/test.js
attester -test mytest/testsuite.js

[jakub-g]

Hi Daniel,

Thanks for reporting that, I agree it would be good to simplify that indeed somehow.

However, there are a few things to consider:

1. attester is able to run different types of tests, not only AT tests (https://github.com/attester/attester/tree/master/spec/test-type),
2. in case of running AT tests, to start a campaign you need a bunch of config info

• config for builtin webserver
• path to AT bootstrap file
• extrascripts to include a skin etc.

Actually

• everything that can be passed via attester.json/attester.yml file, can be also passed via command line using --config.some.key somevalue
• you already can pass the AT classpath to be included via a long --config.tests.aria-templates.classpaths.includes param (this param can be passed multiple times: --config...includes foo --config...includes bar`)

I'm now working around the thing by having an bash file with aliases, which specifies all the common args via command line, and reads the input classpath/path from to specify the proper classpath to include for testing:

https://github.com/jakub-g/dotfiles/blob/master/ariatemplates.profile#L22-31

# Usage:
# attest test.aria.widgets.form.autocomplete.issue315.OpenDropDownFromButtonTest
# attest test/aria/widgets/form/autocomplete/issue315/OpenDropDownFromButtonTest
# attest test/aria/widgets/form/autocomplete/issue315/OpenDropDownFromButtonTest.js

This is a bit more powerful since apart from classpaths, it accepts a disk path (and transforms it accordingly) so you can take advantage of the standard bash filename autocompletion on pressing TAB key. However it needs an installation in bash prompt, and basically duplication of settings in the alias.

What we could do to is to make it possible to accept both

• a config file name,
• and a command line switch with test-to-execute path, which when passed overrides (discards) paths to run from config file name, if any (I mean config["tests"]["aria-templates"]["classpaths"]["includes"])

Currently, you can pass both the config file name and --config.tests.aria-templates.classpaths.includes switch, but it would merge the two; i.e. if you do sth like this in AT repo (using 1.5.4 branch):

node_modules/attester/bin/attester.js --config.tests.aria-templates.classpaths.includes test.aria.core.BrowserTest test/attester.yml

then you'll end up with 607 tests from YAML + additionally one test passed via command line.

So we could have a new switch that could read like that:

• attester --config.tests.aria-templates.grep my.test.classpath attester.yml
• attester --config.tests.aria-templates.grep my/test/path.js attester.yml

(or we can simply keep the --config.tests.aria-templates.includes and do not merge the ones passed from command line and ones included in the cfg file...)

This could be quite ok though it looks ugly due to the long switch name, but to shorten it to --grep we'd have to infer the test type from path/classpath or sth along those lines...

@divdavem do you have some better suggestions?