Add TestSourcePathRoots to settings for better discovery of source locations

[vvuk]

My project results in catch test containers that report relative filenames, e.g. modules/foo/bar.cpp. They're relative to the solution root, but they seem to attempt to be resolved relative to the test executable itself (or.. who knows where the test runner resolves them against, in any case it's not the solution root). This PR adds a TestSourcePathRoots to the config where paths can be explicitly specified.

I would ideally like to have those paths be relative to the solution root (right now they have to be explicit, which is less than ideal), and include the solution root in the search list, but I wasn't sure if it was possible to get that info during the discovery process.

[xkbeyer]

That's interesting, could you provide a small example project which shows the relative path problem? I would like to see how it works. I agree, that it would be better to resolve it w/o having to configure anything.

[vvuk]

Hmm, I don't know if I can easily -- the project is generated from our own internal build system, but I can describe the structure. The project that contains the test structure, FooTest, is a Makefile project:

• output directory and intermediate directory set to c:\toplevel\artifacts (where c:\toplevel is the solution location)
• Under "NMake", the output location is c:\toplevel\build\win64\Debug\FooTest.exe
• Under "Debugging", the working directory is set to $(ProjectDir) (changing this to $(SolutionDir) doesn't help)
• The project file itself is generated in c:\toplevel\projects.

Each catch runner has embedded filenames that are relative to c:\toplevel. I'm not sure why they're coming out as relative paths though; in the reference catch project they're embedded as full absolute paths, so maybe that's the difference.

[xkbeyer]

If you run FooTest.exe --list-tests -v high what is the output?
Do you use the #define CATCH_CONFIG_MAIN or do you have you own test main?
If I understand you correctly, then the Solution is a make project (it is a long time that I have build a make file :( Could you strip down the make file to minimum, so that I can take it as a template and try to reproduce it. Currently I didn't found a way to have relative path in the output of the TestFoo.exe. I really want to have a way to see what happens (at least as an examples to test again).
BTW: The SolutionDirectory can be taken from DiscoverContext.RunSettings

[vvuk]

We use #define CATCH_CONFIG_MAIN (in a CatchMain.cpp that gets linked in with our test files). When I run --list-tests -v high:

$ ./build/win64/Debug/CoreTest.exe --list-tests -v high
All available test cases:
  basic creation
    modules\core\tests\TestCore.cpp(60)
    (NO DESCRIPTION)
      [core]
  test2
    modules\core\tests\TestCore.cpp(67)
    (NO DESCRIPTION)
...

The test source files are built with:

cl "modules\core\tests\TestCore.cpp" /MTd /c /bigobj /cgthreads2 /Od /RTC1 /TP /EHsc /GR- /Gy /nologo /Zi /Zf /GS- /W3 /DWINAPI_FAMILY=WINAPI_FAMILY_DESKTOP_APP /DCATCH_CONFIG_NO_POSIX_SIGNALS /D_DEBUG /I"." /I"external\catch" /I"modules\core\src" /I"C:\Program Files (x86)\Microsoft Visual Studio\Preview\Professional\VC\Tools\MSVC\14.15.26629\include" /I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.17134.0\shared" /I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.17134.0\um" /I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.17134.0\winrt" /I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.17134.0\ucrt" /DWIN32 /DWINDOWS /D_WINDOWS /D_HAS_ITERATOR_DEBUGGING=0 /D_SECURE_SCL=0 /D_CRT_SECURE_NO_DEPRECATE /D_SCL_SECURE_NO_DEPRECATE /D_WINSOCK_DEPRECATED_NO_WARNINGS /D_CRT_NONSTDC_NO_DEPRECATE /DUNICODE /D_UNICODE /D_WIN64 /D_AMD64_ /Fo"artifacts\CoreTestObjects\TestCore.obj" /Fd"artifacts\DebugInfoForAllObjectFiles.pdb" /FS

What's interesting is that the same file built on the same machine with the same build system, but a different compiler, ends up with an absolute filename. I'll try to isolate it.

[vvuk]

Ah. The issue is due to a relative path being specified on the command line. FILE expands out to whatever the command line argument is; I believe VS, when built using msbuild, always expands out the full absolute path on the command line it passes to cl.exe.

[vvuk]

I'm not sure how to pull out SolutionDirectory still -- discoveryContext.RunSettings just has a SettingsXml; am I missing an import?

[xkbeyer]

Ah Ok, I'll try to set it up that way. Will see how far I get.

Regarding the SolutionDirectory: You are right. I confused it with the IRunContext in TestExecutor.RunTests(). But SettingsXml has the information, too. It must be extracted somehow.

[xkbeyer]

I can now reproduce your case.
I would prefer to implement it by just getting the SolutionDirectory from the runsettings and then prepend it. I assume the files are allways relative to the solution directory. So it can be resolved w/o having the user to prepare a runsettings file.
I have prepared something and it would be nice if you can test if it works with your solution, too.
CatchTestAdapter.zip
Please rename zip to vsix.

[vvuk]

Yep, I prefer the SolutionDir solution as well :) Glad you found a way -- the attached zip file works great for me.

[xkbeyer]

I made a new release version 1.4.1.