Discoverer fails to find any tests #19

ScottHutchinson · 2019-01-30T22:08:27Z

Why can't it find the debug symbols when the .pdb files are in the same folder as the .exe file?

[1/30/2019 1:57:34 PM Warning] Could not locate debug symbols for 'C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\NG-DART.exe'. To make use of '--list_content' discovery, ensure that debug symbols are available or make use of '<ForceListContent>' via a .runsettings file.

    <DebugBreak>on</DebugBreak>
    <DiscoverCommandLine>--discover *</DiscoverCommandLine><!-- Note this custom discoverer is not part of the Catch2 framework -->
    <DiscoverTimeout>10000</DiscoverTimeout>
    <FilenameFilter>.*</FilenameFilter>
    <IncludeHidden>true</IncludeHidden>
    <Logging>Verbose</Logging>
    <MessageFormat>StatsOnly</MessageFormat>
    <StackTraceFormat>ShortInfo</StackTraceFormat>
    <StackTracePointReplacement>,</StackTracePointReplacement>
    <TestCaseTimeout>10000</TestCaseTimeout><!-- Milliseconds -->
    <WorkingDirectory></WorkingDirectory>
    <WorkingDirectoryRoot>Executable</WorkingDirectoryRoot>
    <ForceListContent>true</ForceListContent>
  </Catch2Adapter>

The text was updated successfully, but these errors were encountered:

JohnnyHendriks · 2019-01-30T23:09:40Z

Looks like the warning is generated by the Boost test adapter. It may be interfering with Catch2 test discovery. One solution is to disable the Boost test adapter extension, or change the settings for the boost test adapter so it does not try to run discovery on your Catch2 unit tests. The Test Adapter for Catch2 does not need the debug symbols.

I have no experience with the Boost test adapter so I'm afraid I cannot help any further there.

ScottHutchinson · 2019-01-30T23:38:19Z

Thank you for the prompt reply. Indeed you were correct. Disabling the Boost test adapter got rid of the debug symbols error. However, the discoverer is still not finding our tests. We have over 100 Catch2 tests in the NG-DART.exe file, but it does not find any of them. Also I can find no FilenameFilter setting to prevent it from searching every DLL in the project.

[1/30/2019 3:35:25 PM Informational] Started Catch2Adapter test discovery...
[1/30/2019 3:35:25 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver060602.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 3:35:25 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\NG-DART Base.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 3:35:26 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\NG-DART CLR.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 3:35:27 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver090700.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 3:35:27 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver090701.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 3:35:27 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver090703.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 3:35:28 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver090801.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 3:35:29 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver090800.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 3:35:29 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver090806.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 3:35:30 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver100900.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 3:35:30 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver101002.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 3:35:31 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver101004.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 3:35:32 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver101004_MK41VLS.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 3:35:32 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver101005.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 3:35:33 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver101006.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 3:35:34 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver101100.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 3:35:34 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver101101.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 3:35:38 PM Informational] Discover log:
Source: C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\NG-DART.exe
  Testcase count: 0

[1/30/2019 3:35:38 PM Informational] Finished Catch2Adapter test discovery.
[1/30/2019 3:35:38 PM Informational] Started Catch2Adapter test execution...
[1/30/2019 3:35:38 PM Informational] Finished Catch2Adapter test execution.
[1/30/2019 3:35:38 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\NG-DART.exe. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 3:35:38 PM Informational] ========== Run test finished: 0 run (0:00:16.2802547) ==========

ScottHutchinson · 2019-01-30T23:55:30Z

Setting <FilenameFilter>NG-DART.exe</FilenameFilter> yielded this:

[1/30/2019 3:51:32 PM Informational] Discover log:
Source: C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\NG-DART.exe
  Invalid source.

ScottHutchinson · 2019-01-31T00:19:35Z

With <FilenameFilter>^NG</FilenameFilter> yields this:

[1/30/2019 4:14:54 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\NG-DART CLR.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 4:14:54 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\NG-DART Base.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 4:14:54 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver060602.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 4:14:54 PM Informational] Started Catch2Adapter test discovery...
[1/30/2019 4:14:56 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver090700.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 4:14:56 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver090701.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 4:14:56 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver090703.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 4:14:58 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver090800.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 4:14:58 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver090801.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 4:14:58 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver090806.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 4:15:00 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver100900.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 4:15:00 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver101002.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 4:15:00 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver101004.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 4:15:02 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver101004_MK41VLS.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 4:15:02 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver101005.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 4:15:02 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver101006.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 4:15:04 PM Informational] Discover log:
Source: C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\NG-DART.exe
CheckSource name: NG-DART
  XML Discovery:
--discover*doDiscover == true
<?xml version="1.0" encoding="UTF-8"?>
<Catch name="NG-DART.exe">
  <Group name="NG-DART.exe">
    <TestCase name="Test output of v101100 0x1962" tags="[0x1962][2.2.3 Data Processing][v101100]" filename="c:\users\scott\source\dart\ng-dart\source\vs2012\ng-dart\tests\ver101100gmltests.cpp" line="13">
      <OverallResult success="true"/>
    </TestCase>
...
   <TestCase name="Test output of v101101 0x2350 DR_Synchronization_t" tags="[0x2350][2.2.3 Data Processing][v101101]" filename="c:\users\scott\source\dart\ng-dart\source\vs2012\ng-dart\tests\ver101101gmltests.cpp" line="13">
      <OverallResult success="true"/>
    </TestCase>
    <OverallResults successes="0" failures="0" expectedFailures="0"/>
  </Group>
  <OverallResults successes="0" failures="0" expectedFailures="0"/>
</Catch>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NG-DART.exe is a Catch v2.4.2 host application.
Run with -? for options

-------------------------------------------------------------------------------
Test output of v101100 0x1962
-------------------------------------------------------------------------------
c:\users\scott\source\dart\ng-dart\source\vs2012\ng-dart\tests\ver101100gmltests.cpp(13)
...............................................................................

c:\users\scott\source\dart\ng-dart\source\vs2012\ng-dart\tests\dxfilestestsfixture.cpp(42): FAILED:
  REQUIRE( resultsOfLoadFilterFile )
with expansion:
  0

c:\users\scott\source\dart\ng-dart\source\vs2012\ng-dart\tests\dxfilestestsfixture.cpp(31): FAILED:
  REQUIRE( fileRemoved == 0 )
with expansion:
  -1 == 0

  Testcase count: 0
  Accumulated Testcase count: 0

[1/30/2019 4:15:04 PM Informational] Testcase count: 0
[1/30/2019 4:15:04 PM Informational] Finished Catch2Adapter test discovery.
[1/30/2019 4:15:04 PM Informational] Started Catch2Adapter test execution...
[1/30/2019 4:15:04 PM Informational] Finished Catch2Adapter test execution.
[1/30/2019 4:15:04 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\NG-DART.exe. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 4:15:04 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver101100.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 4:15:04 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\Ver101101.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 4:15:04 PM Informational] ========== Run test finished: 0 run (0:00:12.7479446) ==========

ScottHutchinson · 2019-01-31T00:56:42Z

With <DiscoverCommandLine>--list-tests *</DiscoverCommandLine> it yields:

Source: C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\NG-DART.exe
CheckSource name: NG-DART
  Default Discovery:
--list-tests*  Testcase count: 0
  Accumulated Testcase count: 0

[1/30/2019 4:54:42 PM Informational] Testcase count: 0
[1/30/2019 4:54:42 PM Informational] Finished Catch2Adapter test discovery.
[1/30/2019 4:54:42 PM Informational] Started Catch2Adapter test execution...
[1/30/2019 4:54:42 PM Informational] Finished Catch2Adapter test execution.
[1/30/2019 4:54:42 PM Warning] No test is available in C:\Users\Scott\Source\DART\NG-DART\Source\VS2012\NG-DART\Debug\NG-DART.exe. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[1/30/2019 4:54:43 PM Informational] ========== Run test finished: 0 run (0:00:30.3666974) ==========

JohnnyHendriks · 2019-01-31T06:37:13Z

To start, the <FilenameFilter>, uses the filename without the .exe to filter the files, so providing the .exe in the filename filter will cause it NOT to discover those files.

I'm a bit puzzled that you get no tests available in *.dll, as the Test Adapter for Catch2 should only be passed exe-files. It might be a different test adapter that is interfering in this case (e.g., Google test adapter).

But it looks like that the real problem is that during discovery there are some extra unexpected lines being send to the output that probably are messing with the discovery process. Notably in:

  XML Discovery:
--discover*doDiscover == true
<?xml version="1.0" encoding="UTF-8"?>

the --discover*doDiscover == true should not be there. Not sure what is causing that line to be there.

Also in:

CheckSource name: NG-DART
  Default Discovery:
--list-tests*  Testcase count: 0
  Accumulated Testcase count: 0

the --list-tests* in front of Testcase count: 0 should not be there. I suspect these extra lines are caused by the way the custom main is implemented for the NG-DART executable you want to discover tests from.

ScottHutchinson · 2019-01-31T15:29:58Z

I have disabled both the Google Test Adapter and the Boost Test Adapter. AFAIK there are no other adapters for C++ installed.

I added the "doDiscover == true" to cout for debugging.

If it's not supposed to echo the command line (for Verbose and Debug output), then maybe our main is messing up the command line. But the output shows that the command line is effectively switching between XML Discovery and Default Discovery. So I think maybe the command line is ok, and the problem is something else. I have no idea what yet. Thank you very much for your help!

ScottHutchinson · 2019-02-05T18:05:18Z

It seems like the only way we might get this to work is if we could debug the adapter. I was able to clone and build it, but I don't know how to debug it. I'll try to learn how when I have time.

JohnnyHendriks · 2019-02-05T23:20:20Z

I have found debugging the adapter directly is troublesome, in the sense that I haven't been able to get it to work. What I've done is that I use unittests to test the main components of the test adapter. Here I call a Catch2 executable, from the unittest. See the ExecuterTests and DiscovererTests for examples.

Note that I've just submitted some changes that improve test discovery using the default discover mechanism that comes with Catch2 (works with Catch 2.0.1 and up). Here you now also get source file and line number information, so you no longer need the custom XML discovery mechanism for this. I expect to release the new version of the adapter somewhere around the weekend. I still need to update the documentation, and do some last sanity checks before I release the new version. But you can of course already get the latest features if you compile the adapter yourself.

ScottHutchinson · 2019-04-13T17:50:40Z

Now I am trying to get this to work with the exercism hello-world folder. I just installed VS 2019 and the Catch2 test adapter. Even with that, the adapter cannot find any tests. I just don't get it.

Here is the test.runnsettings file, which I selected:

<RunSettings>

  <Catch2Adapter>
    <FilenameFilter>.*</FilenameFilter>
  </Catch2Adapter>

</RunSettings>

ScottHutchinson · 2019-04-17T15:56:32Z

I finally got it to work on a new console project. Apparently, the test adapter requires a solution file to discover tests.

JohnnyHendriks · 2019-04-18T04:54:04Z

Good to hear you got it working. I was out of ideas of what could be causing your problems. Thank you for mentioning your solution. It is typically one of those subtleties that is easy to take for granted.

allan-bommer · 2021-08-11T22:32:44Z

I have been having trouble with discovery, but think I have diagnosed the problem.
The "output directory" (in vs proj file) contains the exe, but not in runnable form. Our build system symlinks it into another directory with all its dependencies for running/debugging. This is handled with a custom Debug Command.

So.. "MyTests.exe --list-tests" will crash instead of returning anything useful.

Is there a way to control where the discoverer looks for the exe to discover the tests?

StefanOlin · 2022-01-10T15:05:14Z

@ScottHutchinson I have the same problem as you are describing, " the test adapter requires a solution file to discover tests.": what do you mean? Does it need its own *.sln file or its own subproject in the master sln-file?

Stefan

allan-bommer · 2022-01-10T16:00:11Z

I didn't do the fix (for my build) myself, but it seemed to consist of:

Ensuring Catch2Interface.dll and Catch2TestAdapter.dll were accessible (where the tests are run)
Adding a runsettings file (see below)
Setting an environment variable (named SAETests_DEPLOYMENT_PATH for us)

Runsettings looks like this:

%SAETests_DEPLOYMENT_PATH% x64 true .* 60000 on --verbosity high --list-tests * true debug ShortInfo 60 , 20000 Executable

StefanOlin · 2022-01-10T16:29:08Z

@allan-bommer, then I guess you have built the interface on your own and not used the version downloaded from within VS.

ScottHutchinson · 2022-01-10T20:05:15Z

@StefanOlin The tests just have to be in a project that is part of a solution. I believe initially I was trying to get it to work on a stand-alone project with no .sln file at all.

ScottHutchinson changed the title ~~Could not locate debug symbols~~ Discoverer fails to find any tests Jan 31, 2019

JohnnyHendriks closed this as completed Apr 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Discoverer fails to find any tests #19

Discoverer fails to find any tests #19

ScottHutchinson commented Jan 30, 2019 •

edited

JohnnyHendriks commented Jan 30, 2019

ScottHutchinson commented Jan 30, 2019 •

edited

ScottHutchinson commented Jan 30, 2019

ScottHutchinson commented Jan 31, 2019

ScottHutchinson commented Jan 31, 2019

JohnnyHendriks commented Jan 31, 2019

ScottHutchinson commented Jan 31, 2019

ScottHutchinson commented Feb 5, 2019

JohnnyHendriks commented Feb 5, 2019

ScottHutchinson commented Apr 13, 2019

ScottHutchinson commented Apr 17, 2019

JohnnyHendriks commented Apr 18, 2019

allan-bommer commented Aug 11, 2021

StefanOlin commented Jan 10, 2022

allan-bommer commented Jan 10, 2022

StefanOlin commented Jan 10, 2022

ScottHutchinson commented Jan 10, 2022

Discoverer fails to find any tests #19

Discoverer fails to find any tests #19

Comments

ScottHutchinson commented Jan 30, 2019 • edited

JohnnyHendriks commented Jan 30, 2019

ScottHutchinson commented Jan 30, 2019 • edited

ScottHutchinson commented Jan 30, 2019

ScottHutchinson commented Jan 31, 2019

ScottHutchinson commented Jan 31, 2019

JohnnyHendriks commented Jan 31, 2019

ScottHutchinson commented Jan 31, 2019

ScottHutchinson commented Feb 5, 2019

JohnnyHendriks commented Feb 5, 2019

ScottHutchinson commented Apr 13, 2019

ScottHutchinson commented Apr 17, 2019

JohnnyHendriks commented Apr 18, 2019

allan-bommer commented Aug 11, 2021

StefanOlin commented Jan 10, 2022

allan-bommer commented Jan 10, 2022

StefanOlin commented Jan 10, 2022

ScottHutchinson commented Jan 10, 2022

ScottHutchinson commented Jan 30, 2019 •

edited

ScottHutchinson commented Jan 30, 2019 •

edited