Test case grouping

[m-mcgowan]

Unless I've missed it, it doesn't seem possible to group test cases into larger structures.

With UnitTest++, you can group tests into suites:

  SUITE(MySuite)
  {
    TEST_CASE(...)
  }

It would be great to have this in Catch - it would make test organization possible (and simple). Nesting suites, or groups, would be a natural way to organize a large body of tests.

This would also make the junit reporter make more sense which presently creates a single <testsuite> element (with no name). With SUITE() support, it could output multiple suites, giving nicer organization in HTML test reports.

Sweet!

[philsquared]

It's easy to group test cases into groups.
Simply give tests you want to group together the same tag, then specify that tag on the command line.
See: https://github.com/philsquared/Catch/blob/master/docs/test-cases-and-sections.md#tags

I did have a plan at one point to have named groups (where you associate a tag or set of tags with a name) and that name would appear as the suite name in the Junit report. I never followed through on that and, since nobody asked about it, eventually deprecated it. It would be possible to resurrect it, though.

[philsquared]

If you have a complicated tag spec that you use a lot, btw (e.g. specifying several tags, including some exclusions) I recently added the ability to define tag aliases using a macro. It's not (yet) documented, but you just write something like:

CATCH_REGISTER_TAG_ALIAS( "[@myAlias]", "[this]~[notthis]" )

Then you can run all tests tagged, [this], except those also tagged, [notthis], by specifying [@myAlias] on the command line (note the @ is a special character, denoting a tag alias, and is required).

[m-mcgowan]

Thanks for the information. Will the tags appear in the junit test report? (Ah, I see you covered that - missed it on first reading.)

If you're sure tags are the right way to go to gain structure to the test reports, then it would be great to have that feature resurrected. To my mind, tags are orthogonal to a more rigid hierarchical test-case organization, so having a GROUP() or SUITE() macro still seems more fitting, at least from my limited knowledge of this project.

[philsquared]

IIRC it used the tags specified as the group name. I think that's still the case - but I can't check right now - try it out

[wichert]

For what it's worth I would love to have that test suite grouping. Especially when writing BDD-style tests it is very practical to allow grouping all tests for a specific module or class.

[philsquared]

@m-mcgowan: Over in issue #5 you said:

"I'm finding tags a bit of a pain, and I often forget. For example, I just made unit tests for a PRNG, in random.cpp. Each test then has as a minimum one tag [random]. Would be great to simply wrap the whole lot in a big group."

• I thought it would be better to respond to that here.

The trouble with the SUITE() idea is that it would have to be a construct that exists at a scope above global functions.
I have a half-formed idea where a SUITE() macro would open a namespace and create a self-registered object that associates a suite name with the namespace - the TEST_CASE macro could then find this and associate itself with the suite name.
It feels a bit hacky but I think I could get something to work.
Let me think about that some more.

[philsquared]

This ticket is quite old - and seems at least partially addressed.
In addition to the suggestions I already made, you can also now run with -# to cause Catch to automatically tag every test case with a tag based on the filename (see: https://github.com/philsquared/Catch/blob/master/docs/test-cases-and-sections.md#special-tags)

Is anyone still dissatisfied with these options? (@m-mcgowan, @wichert)?

[tsondergaard]

The tagging mechanism is excellent for controlling which tests to run, but I am having problems creating adequately structured junit xml reports for display in Jenkins. The junit plugin in Jenkins is a little strange. It doesn't care about the hierarchy in the xml, instead it displays the test results in a hierarchy based on java-packages, which is forms by splitting the classname attribute of the element by ".".

The Catch::JunitReporter would serve the needs of my project well if the classname attribute of the element was formed using the .<[#filename]>.

We run the same test code compiled as 32-bit and 64-bit code and the with the included as above, it would be possible to run the test as "test-executable -n win32.x86 -# -r junit -o junit-report-x86.xml" and " test-executable -n win32.x64 -# -r junit -o junit-report-x64.xml" and get two xml reports that jenkins will group nicely by architecture (x86 and x64).

[horenmar]

I believe the last comment was addressed in the linked issue, so I am going to close this.

[LecrisUT]

If I may revive this issue, I would say tags are not groups, and the main advantage of grouping would be in displaying the reports, and finding the failing errors, while tags are meant for filtering. For example:

Group(BackEnd){
  Group(Database){
    TEST_CASE(..., "[NeedsInternet, MySQL]")
  }
  Group(API){
    TEST_CASE(...)
  }
}
Group(FrontEnd){
  TEST_CASE(...)
}

On the backend it could be made that each top-level group is its own executable/cmake target, unless a custom implementation is requested.

---

Benefits

• More readable tests. I would like it that ctest and catch2 test executable display the results of these tests as a tree, i.e.:

  Backend: 58 Tests Enabled, 34 Tests Skipped
    |
    |-- Database: ...
    |
    |-- API: ...
  Frontend: ...
    |
  

• Instead of being bombarded with hundreds of TestCases, it would be clearer to expand the tree where the specific tests failed. That way the developer can navigate back to either or both the test source and main source to make necessary changes
• Simpler more descriptive test names
• Having each group as separate executable/cmake target, it is easier to enable building and conditionally linking the tests, e.g. depending on compiler or available libraries