Skip to content

Commit

Permalink
Rewrite commandline test spec docs
Browse files Browse the repository at this point in the history
Closes #2738
  • Loading branch information
horenmar committed Aug 30, 2023
1 parent f24d39e commit 2a19ae1
Showing 1 changed file with 84 additions and 25 deletions.
109 changes: 84 additions & 25 deletions docs/command-line.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,43 +85,102 @@ Click one of the following links to take you straight to that option - or scroll

<pre>&lt;test-spec> ...</pre>

Test cases, wildcarded test cases, tags and tag expressions are all passed directly as arguments. Tags are distinguished by being enclosed in square brackets.
By providing a test spec, you filter which tests will be run. If you call
Catch2 without any test spec, then it will run all non-hidden test
cases. A test case is hidden if it has the `[!benchmark]` tag, any tag
with a dot at the start, e.g. `[.]` or `[.foo]`.

If no test specs are supplied then all test cases, except "hidden" tests, are run.
A test is hidden by giving it any tag starting with (or just) a period (```.```) - or, in the deprecated case, tagged ```[hide]``` or given name starting with `'./'`. To specify hidden tests from the command line ```[.]``` or ```[hide]``` can be used *regardless of how they were declared*.
There are three basic test specs that can then be combined into more
complex specs:

Specs must be enclosed in quotes if they contain spaces. If they do not contain spaces the quotes are optional.
* Full test name, e.g. `"Test 1"`.

Wildcards consist of the `*` character at the beginning and/or end of test case names and can substitute for any number of any characters (including none).
This allows only test cases whose name is "Test 1".

Test specs are case insensitive.
* Wildcarded test name, e.g. `"*Test"`, or `"Test*"`, or `"*Test*"`.

If a spec is prefixed with `exclude:` or the `~` character then the pattern matches an exclusion. This means that tests matching the pattern are excluded from the set - even if a prior inclusion spec included them. Subsequent inclusion specs will take precedence, however.
Inclusions and exclusions are evaluated in left-to-right order.
This allows any test case whose name ends with, starts with, or contains
in the middle the string "Test". Note that the wildcard can only be at
the start or end.

Test case examples:
* Tag name, e.g. `[some-tag]`.

This allows any test case tagged with "[some-tag]". Remember that some
tags are special, e.g. those that start with "." or with "!".


You can also combine the basic test specs to create more complex test
specs. You can

* Concatenate specs to apply all of them, e.g. `[some-tag][other-tag]`.

This allows test cases that are tagged with **both** "[some-tag]" **and**
"[other-tag]". A test case with just "[some-tag]" will not pass the filter,
nor will test case with just "[other-tag]".

* Comma-join specs to apply any of them, e.g. `[some-tag],[other-tag]`.

This allows test cases that are tagged with **either** "[some-tag]" **or**
"[other-tag]". A test case with both will obviously also pass the filter.

Note that commas take precendence over simple concatenation. This means
that `[a][b],[c]` accepts tests that are tagged with either both "[a]" and
"[b]", or tests that are tagged with just "[c]".

* Negate the spec by prepending it with `~`, e.g. `~[some-tag]`.

This rejects any test case that is tagged with "[some-tag]". Note that
rejection takes precedence over other filters.

Note that negations always binds to the following _basic_ test spec.
This means that `~[foo][bar]` negates only the "[foo]" tag and not the
"[bar]" tag.

Note that when Catch2 is deciding whether to include a test, first it
checks whether the test matches any negative filters. If it does,
the test is rejected. After that, the behaviour depends on whether there
are positive filters as well. If there are no positive filters, all
remaining non-hidden tests are included. If there are positive filters,
only tests that match the positive filters are included.

You can also match test names with special characters by escaping them
with a backslash (`"\"`), e.g. a test named `"Do A, then B"` is matched
by "Do A\, then B" test spec. Backslash also escapes itself.


### Examples

Given these TEST_CASEs,
```
thisTestOnly Matches the test case called, 'thisTestOnly'
"this test only" Matches the test case called, 'this test only'
these* Matches all cases starting with 'these'
exclude:notThis Matches all tests except, 'notThis'
~notThis Matches all tests except, 'notThis'
~*private* Matches all tests except those that contain 'private'
a* ~ab* abc Matches all tests that start with 'a', except those that
start with 'ab', except 'abc', which is included
~[tag1] Matches all tests except those tagged with '[tag1]'
-# [#somefile] Matches all tests from the file 'somefile.cpp'
TEST_CASE("Test 1") {}
TEST_CASE("Test 2", "[.foo]") {}
TEST_CASE("Test 3", "[.bar]") {}
TEST_CASE("Test 4", "[.][foo][bar]") {}
```

Names within square brackets are interpreted as tags.
A series of tags form an AND expression whereas a comma-separated sequence forms an OR expression. e.g.:
this is the result of these filters
```
./tests # Selects only the first test, others are hidden
./tests "Test 1" # Selects only the first test, other do not match
./tests ~"Test 1" # Selects no tests. Test 1 is rejected, other tests are hidden
./tests "Test *" # Selects all tests.
./tests [bar] # Selects tests 3 and 4. Other tests are not tagged [bar]
./tests ~[foo] # Selects test 1, because it is the only non-hidden test without [foo] tag
./tests [foo][bar] # Selects test 4.
./tests [foo],[bar] # Selects tests 2, 3, 4.
./tests ~[foo][bar] # Selects test 3. 2 and 4 are rejected due to having [foo] tag
./tests ~"Test 2"[foo] # Selects test 4, because test 2 is explicitly rejected
./tests [foo][bar],"Test 1" # Selects tests 1 and 4.
./tests "Test 1*" # Selects test 1, wildcard can match zero characters
```

<pre>[one][two],[three]</pre>
This matches all tests tagged `[one]` and `[two]`, as well as all tests tagged `[three]`
_Note: Using plain asterisk on a command line can cause issues with shell
expansion. Make sure that the asterisk is passed to Catch2 and is not
interpreted by the shell._

Test names containing special characters, such as `,` or `[` can specify them on the command line using `\`.
`\` also escapes itself.

<a id="choosing-a-reporter-to-use"></a>
## Choosing a reporter to use
Expand Down

0 comments on commit 2a19ae1

Please sign in to comment.