Document How to make Catch tests in a library be found

[martinmoene]

See Catch forum messages

• Getting catch to work across multiple compilation units by John Byrd
• CATCH_CONFIG_RUNNER usage by Maxim

[johnwbyrd]

On this topic... it would be useful to have a clearly documented method for adding test cases manually rather than depending on AutoReg to do so for you by instancing objects before main() is called. Test fixtures as written depend on the AutoReg behavior, which I expect would not work for the reasons already given in this issue.

[snortadmin]

Catch v1.2.1

• Generated: 2015-06-30 18:23:27.961086

Seems I'm running into the same or similar issue on both Linux and OSX building with g++ (not clang++) and autotools. Here is the layout:

program
    lib1
        mod1a.cc
        mod1a_test.cc
        mod1b.cc
        ...
    lib2
        mod2a.cc
        ...

TEST_CASEs for xxx.cc are in xxx_test.cc. The lib# are convenience libs (built directly into the executable).

The above doesn't work because auto registration is apparently failing. The work around is to add

#include "mod1a_test.cc"

to the bottom of mod1a.cc. Then the tests run just fine.

mod1a_test.cc is being compiled into lib1 but there is a difference in the symbols therein. With the include there are 60 Catch symbols, w/o there are only 30. The missing symbols are given below.

Note that this is with optimization disabled ( -O0 ).

I'm not sure how else to convince the linker this stuff is needed. 2nd the suggestion that explicit registration be supported as a fallback.

1,30d0
< 0000000000002fae S __ZN5Catch11NameAndDescC1EPKcS2_
< 0000000000002fe4 S __ZN5Catch13AssertionInfoD1Ev
< 000000000000316c S __ZN5Catch13ExpressionLhsIRKiE17captureExpressionILNS_8Internal8OperatorE0EiEERNS_13ResultBuilderERKT0_
< 0000000000003142 S __ZN5Catch13ExpressionLhsIRKiEC1ERNS_13ResultBuilderES2_
< 000000000000311c S __ZN5Catch13ExpressionLhsIRKiEeqIiEERNS_13ResultBuilderERKT_
<                  U __ZN5Catch13ResultBuilder13endExpressionEv
<                  U __ZN5Catch13ResultBuilder13setResultTypeEb
< 0000000000003048 S __ZN5Catch13ResultBuilder14ExprComponentsD1Ev
<                  U __ZN5Catch13ResultBuilder18useActiveExceptionENS_17ResultDisposition5FlagsE
<                  U __ZN5Catch13ResultBuilder5reactEv
<                  U __ZN5Catch13ResultBuilder5setOpERKSs
<                  U __ZN5Catch13ResultBuilder6setLhsERKSs
<                  U __ZN5Catch13ResultBuilder6setRhsERKSs
<                  U __ZN5Catch13ResultBuilderC1EPKcRKNS_14SourceLineInfoES2_NS_17ResultDisposition5FlagsE
< 00000000000030a0 S __ZN5Catch13ResultBuilderD1Ev
< 00000000000030ea S __ZN5Catch13ResultBuilderleIiEENS_13ExpressionLhsIRKT_EES5_
< 0000000000003086 S __ZN5Catch14CopyableStreamD1Ev
<                  U __ZN5Catch14SourceLineInfoC1EPKcm
< 0000000000002f94 S __ZN5Catch14SourceLineInfoD1Ev
<                  U __ZN5Catch16isDebuggerActiveEv
< 000000000000301e S __ZN5Catch19AssertionResultDataD1Ev
< 0000000000002f84 S __ZN5Catch6isTrueEb
<                  U __ZN5Catch7AutoRegC1EPFvvERKNS_14SourceLineInfoERKNS_11NameAndDescE
<                  U __ZN5Catch7AutoRegD1Ev
< 0000000000002fd7 S __ZN5Catch8Internal14OperatorTraitsILNS0_8OperatorE0EE7getNameEv
< 000000000000331b S __ZN5Catch8Internal6opCastIiEERT_RKS2_
< 00000000000032bd S __ZN5Catch8Internal7compareILNS0_8OperatorE0EiiEEbRKT0_RKT1_
< 00000000000032e2 S __ZN5Catch8Internal9EvaluatorIiiLNS0_8OperatorE0EE8evaluateERKiS5_
<                  U __ZN5Catch8toStringEi
<                  U __ZNK5Catch13ResultBuilder16shouldDebugBreakEv

[johnwbyrd]

Furthermore... my leak detector in Windows is detecting all these pre-main allocated objects as memory leaks on shutdown. I've not investigated this enough to blame Catch, as there might be some incantation I should be using to ask Catch to release all these pre-main allocated structures.

[philsquared]

Manual test registration is now in on the develop branch - as of v1.3.0-develop.4.
Use REGISTER_TEST_CASE( <function>, <name> [,<tags>] ).

@johnwbyrd Are you using CATCH_CONFIG_MAIN or CATCH_CONFIG_RUNNER - and if the latter are you using Catch::Session?

[snortadmin]

CATCH_CONFIG_RUNNER with Catch::Session.

Thanks
Russ

On 11/20/15 12:04 PM, Phil Nash wrote:

Manual test registration is now in on the develop branch - as of
v1.3.0-develop.4.
Use |REGISTER_TEST_CASE( , [,] )|.

@johnwbyrd https://github.com/johnwbyrd Are you using
|CATCH_CONFIG_MAIN| or |CATCH_CONFIG_RUNNER| - and if the latter are
you using |Catch::Session|?

—
Reply to this email directly or view it on GitHub
#421 (comment).

[philsquared]

The destructor of Session should be calling Catch::cleanUp(), which, in turn, should be clearing down all those static structures. If you get a chance to see what it thinks is leaking more specifically I'll take a look.

I run it with leak detection on Windows and this does it for me but I've noticed in the past that doesn't catch everything for some reason.

[johnwbyrd]

@philsquared I am also currently using Catch::Session but will probably switch to the new argc/argv interference method you implemented. I was getting a couple hundred leaks in my Catch-related code.

Putting Catch::cleanUp() after all Catch calls causes all those memory leaks to go away.

Please make the Catch::cleanUp() function call part of the official documentation example for Catch::Session(). Also, please document Catch::cleanUp(). This will save a lot of people a lot of pain.

[philsquared]

Just a thought, but is your Catch::Session instance a global, or in a function scope?

[johnwbyrd]

Following the documentation at https://github.com/philsquared/Catch/blob/master/docs/own-main.md, I simply call

int result = Catch::Session().run( argc, argv );

at the appropriate time and the self-registered test cases run. Since I don't instance a Session, I assume it exists in global scope.

[pthom]

(This answer is outdated, see better alternative below, using add_library( OBJECT)

I have an alternative suggestion, using auto-generated code.
Please look at https://github.com/pthom/catch_registerstaticlibrary

This repo provides some cmake/python scripts with which adding catch to a static library becomes a one-liner: simply add this line to the CMakeLists.txt file
catch_registerstaticlibrary(MyLibrary MyLibraryTest)

Although, many developers may not like it, as it slightly modifies the code,

It was tested under OSX (clang) Windows (MSVC 2015 and 2010), and linux. The registration problem was present on all this plaforms and can be solved with such a solution.

[pthom]

I have found a more reliable solution, that does

• does not require code modifications
• does not use fancy compiler switches
• does not require to use REGISTER_TEST_CASE

The idea is to first add an "object" library through cmake:
add_library(MyLibrary_Object OBJECT lib1.cpp lib2.cpp)

Then create a static or shared lib from the object library, as well as a test target that uses this object library.

Full working example here : https://github.com/pthom/cmake_registertest/blob/master/examples/example_catch_raw/MyLibrary/CMakeLists.txt

include_directories("../../thirdparty_testlibs/catch")
add_library(MyLibrary_Object OBJECT lib1.cpp lib2.cpp)

add_library(MyLibrary_Static STATIC $<TARGET_OBJECTS:MyLibrary_Object>)
# or
add_library(MyLibrary_Shared SHARED $<TARGET_OBJECTS:MyLibrary_Object> ../catch_dynamic.cpp)

add_executable(MyLibraryTest $<TARGET_OBJECTS:MyLibrary_Object> ../catch_main.cpp)
add_test(NAME MyLibraryTest COMMAND MyLibraryTest)

This example is an extract of a github project where I provide a cmake scripts that helps test registration for catch, googletest and doctest.

Example for a library that does have test code in its source (as well as in external files) :
https://github.com/pthom/cmake_registertest/blob/master/examples/example_catch/MyLibrary/CMakeLists.txt

add_library(MyLibrary_Object OBJECT lib1.cpp lib2.cpp)
crt_registertest(
  TEST_INPLACE
  INPUT_OBJECT_LIBRARY MyLibrary_Object
  OUTPUT_LIBRARY_NAME MyLibrary_Static
  OUTPUT_LIBRARY_TYPE STATIC
  OUTPUT_TEST_TARGET MyLibraryTest
  TEST_SOURCES lib3_test.cpp lib4_test.cpp 
)

[Karry]

Another solution is to tell linker to NOT strip not referenced symbols. In CMake:

target_link_libraries (unittests
    -Wl,--whole-archive $<TARGET_FILE:mylib> -Wl,--no-whole-archive
)

where unittests is Catch executable and mylib is static library

[trashuj]

Hi, i just ran into this issue. Can you please add some info in the main docs because it's really confusing.
Broke both msvc and ubuntu/clang.
Besides that i'm happy so far :) thanks!

[boinst]

I'm still confused by this. Currently compiling all of my tests into one executable, I could not figure out how to get this to work.

[ferdnyc]

I'm still confused by this. Currently compiling all of my tests into one executable, I could not figure out how to get this to work.

That's not necessarily a bad thing. If you're using CMake and catch_discover_tests(), it will extract the names of all your TEST_CASEs from the compiled test runner and CTest will execute each test as a separate process (by running the binary with the appropriate arguments to execute that specific test).

Still, if you want to compile multiple test executables, because you have the tests in separate source files, that's easily doable. Here's the code I use, in CMake:

include(Catch)
include(CTest)

# Create object library for test executable main(),
# to avoid recompiling for every test
add_library(catch-main OBJECT catch_main.cpp)

foreach(tname ${UNIT_TESTS})
  add_executable(${tname}-test ${tname}.cpp $<TARGET_OBJECTS:catch-main>)
  target_link_libraries(${tname}-test Catch2::Catch2 mylib)
  # Automatically configure CTest targets from Catch2 test cases
  catch_discover_tests(
    ${tname}-test
    TEST_PREFIX ${tname}:
  )
endforeach()

The OBJECT library for catch_main.cpp is there, as the comment says, to avoid recompiling it for each test executable. I have something like 19 of them, and each of those compiles takes a relatively long time even on Linux (and ages on Windows) despite consisting, in its entirety, of nothing but:

#define CATCH_CONFIG_MAIN
#include <catch2/catch.hpp>

So, cutting out 18/19 of those compiles is a significant win.

Now, this all still assumes that your test cases are in external source files. (In my case, they're in files named ${tname}.cpp, and I have a list of test names in ${UNIT_TESTS}.)

If you're trying to compile the tests into the library you're testing, like in the OP for this issue, then I just have to echo Martin in the CATCH_CONFIG_RUNNER forum thread, who quotes the googletest documentation:

The general conclusion here is: make your life easier - do not write your tests in libraries!

Is there any reason at all that would actually be desirable?