Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Catch2 tests described in contribution guide fail to execute #2339

Closed
Alan-Jowett opened this issue Dec 16, 2021 · 2 comments
Closed

Catch2 tests described in contribution guide fail to execute #2339

Alan-Jowett opened this issue Dec 16, 2021 · 2 comments
Labels
3rd party bug

Comments

@Alan-Jowett
Copy link
Contributor

Describe the bug
https://github.com/catchorg/Catch2/blob/devel/docs/contributing.md#testing-your-changes describes how to validate changes when contributing. When running tests under Windows, they fail:

E:\Catch2\debug-build>ctest -j 4 --output-on-failure -C Debug
Test project E:/Catch2/debug-build
      Start 20: ApprovalTests
      Start  1: RunTests
      Start  2: List::Tests::Output
      Start  3: List::Tests::Quiet
 1/62 Test  #2: List::Tests::Output ...............................................   Passed    0.05 sec
      Start  4: List::Tests::ExitCode
 2/62 Test  #3: List::Tests::Quiet ................................................   Passed    0.06 sec
      Start  5: List::Tests::XmlOutput
 3/62 Test  #4: List::Tests::ExitCode .............................................   Passed    0.03 sec
      Start  6: List::Tags::Output
 4/62 Test  #6: List::Tags::Output ................................................   Passed    0.03 sec
      Start  7: List::Tags::ExitCode
 5/62 Test  #5: List::Tests::XmlOutput ............................................   Passed    0.06 sec
      Start  8: List::Tags::XmlOutput
 6/62 Test  #7: List::Tags::ExitCode ..............................................   Passed    0.02 sec
      Start  9: List::Reporters::Output
 7/62 Test  #9: List::Reporters::Output ...........................................   Passed    0.02 sec
      Start 10: List::Reporters::ExitCode
 8/62 Test  #8: List::Tags::XmlOutput .............................................   Passed    0.03 sec
      Start 11: List::Reporters::XmlOutput
 9/62 Test #10: List::Reporters::ExitCode .........................................   Passed    0.02 sec
      Start 12: NoAssertions
10/62 Test #11: List::Reporters::XmlOutput ........................................   Passed    0.03 sec
      Start 13: NoTest
11/62 Test #12: NoAssertions ......................................................   Passed    0.02 sec
      Start 14: WarnAboutNoTests
12/62 Test #13: NoTest ............................................................   Passed    0.03 sec
      Start 15: UnmatchedOutputFilter
13/62 Test #15: UnmatchedOutputFilter .............................................   Passed    0.02 sec
      Start 16: FilteredSection-1
14/62 Test #16: FilteredSection-1 .................................................   Passed    0.02 sec
      Start 17: FilteredSection-2
15/62 Test #17: FilteredSection-2 .................................................   Passed    0.02 sec
      Start 18: FilteredSection::GeneratorsDontCauseInfiniteLoop-1
16/62 Test #14: WarnAboutNoTests ..................................................   Passed    0.09 sec
      Start 19: FilteredSection::GeneratorsDontCauseInfiniteLoop-2
17/62 Test #18: FilteredSection::GeneratorsDontCauseInfiniteLoop-1 ................   Passed    0.02 sec
      Start 21: RegressionCheck-1670
18/62 Test #19: FilteredSection::GeneratorsDontCauseInfiniteLoop-2 ................   Passed    0.02 sec
      Start 22: VersionCheck
19/62 Test #21: RegressionCheck-1670 ..............................................   Passed    0.03 sec
      Start 23: LibIdentityTest
20/62 Test #22: VersionCheck ......................................................   Passed    0.02 sec
      Start 24: FilenameAsTagsTest
21/62 Test #23: LibIdentityTest ...................................................   Passed    0.02 sec
      Start 25: FilenameAsTagsMatching
22/62 Test #24: FilenameAsTagsTest ................................................   Passed    0.02 sec
      Start 26: EscapeSpecialCharactersInTestNames
23/62 Test #25: FilenameAsTagsMatching ............................................   Passed    0.03 sec
      Start 27: NegativeSpecNoHiddenTests
24/62 Test #26: EscapeSpecialCharactersInTestNames ................................   Passed    0.03 sec
      Start 28: TestsInFile::SimpleSpecs
25/62 Test #27: NegativeSpecNoHiddenTests .........................................   Passed    0.02 sec
      Start 29: TestsInFile::EscapeSpecialCharacters
26/62 Test #28: TestsInFile::SimpleSpecs ..........................................   Passed    0.02 sec
      Start 30: TestsInFile::InvalidTestNames-1
27/62 Test #29: TestsInFile::EscapeSpecialCharacters ..............................   Passed    0.02 sec
      Start 31: TestsInFile::InvalidTestNames-2
28/62 Test #30: TestsInFile::InvalidTestNames-1 ...................................   Passed    0.02 sec
      Start 32: TagAlias
29/62 Test #31: TestsInFile::InvalidTestNames-2 ...................................   Passed    0.02 sec
      Start 33: RandomTestOrdering
30/62 Test #32: TagAlias ..........................................................   Passed    0.03 sec
      Start 34: CheckConvenienceHeaders
31/62 Test #34: CheckConvenienceHeaders ...........................................   Passed    0.18 sec
      Start 35: Benchmarking::FailureReporting::ThrowingBenchmark
32/62 Test #33: RandomTestOrdering ................................................   Passed    0.32 sec
      Start 36: Benchmarking::FailureReporting::FailedAssertion
33/62 Test #36: Benchmarking::FailureReporting::FailedAssertion ...................   Passed    2.25 sec
      Start 37: Benchmarking::FailureReporting::FailMacro
34/62 Test #35: Benchmarking::FailureReporting::ThrowingBenchmark .................   Passed    2.39 sec
      Start 38: Benchmarking::FailureReporting::ShouldFailIsRespected
35/62 Test #20: ApprovalTests .....................................................   Passed    5.21 sec
      Start 39: TestShardingIntegration
36/62 Test #37: Benchmarking::FailureReporting::FailMacro .........................   Passed    2.31 sec
      Start 40: TestSharding::OverlyLargeShardIndex
37/62 Test #40: TestSharding::OverlyLargeShardIndex ...............................   Passed    0.02 sec
      Start 42: MinDuration::DurationOverrideYes
38/62 Test #38: Benchmarking::FailureReporting::ShouldFailIsRespected .............   Passed    2.34 sec
      Start 43: MinDuration::DurationOverrideNo
39/62 Test #39: TestShardingIntegration ...........................................   Passed    0.66 sec
      Start 44: CATCH_CONFIG_PREFIX_ALL
40/62 Test #44: CATCH_CONFIG_PREFIX_ALL ...........................................   Passed    0.02 sec
      Start 45: CATCH_CONFIG_DISABLE-1
41/62 Test #45: CATCH_CONFIG_DISABLE-1 ............................................   Passed    0.02 sec
      Start 46: CATCH_CONFIG_DISABLE-2
42/62 Test #46: CATCH_CONFIG_DISABLE-2 ............................................   Passed    0.02 sec
      Start 47: CATCH_CONFIG_DISABLE_EXCEPTIONS-1
43/62 Test #47: CATCH_CONFIG_DISABLE_EXCEPTIONS-1 .................................   Passed    0.02 sec
      Start 48: CATCH_CONFIG_DISABLE_EXCEPTIONS-3
44/62 Test #48: CATCH_CONFIG_DISABLE_EXCEPTIONS-3 .................................   Passed    0.02 sec
      Start 49: CATCH_CONFIG_DISABLE_EXCEPTIONS-4
45/62 Test #49: CATCH_CONFIG_DISABLE_EXCEPTIONS-4 .................................   Passed    0.02 sec
      Start 50: DeferredStaticChecks
46/62 Test #50: DeferredStaticChecks ..............................................   Passed    0.02 sec
      Start 51: FallbackStringifier
47/62 Test #51: FallbackStringifier ...............................................   Passed    0.02 sec
      Start 52: CATCH_CONFIG_DISABLE_STRINGIFICATION
48/62 Test #52: CATCH_CONFIG_DISABLE_STRINGIFICATION ..............................   Passed    0.02 sec
      Start 53: WindowsHeader
49/62 Test #53: WindowsHeader .....................................................   Passed    0.02 sec
      Start 54: PartialTestCaseEvents
50/62 Test #54: PartialTestCaseEvents .............................................   Passed    0.22 sec
      Start 55: BenchmarksInCumulativeReporter
51/62 Test #42: MinDuration::DurationOverrideYes ..................................   Passed    1.16 sec
      Start 56: Reporters::registration-is-case-preserving
52/62 Test #56: Reporters::registration-is-case-preserving ........................   Passed    0.02 sec
      Start 57: Reporters::selection-is-case-insensitive
53/62 Test #43: MinDuration::DurationOverrideNo ...................................   Passed    1.15 sec
      Start 58: DuplicatedTestCases::SameNameAndTags
54/62 Test #58: DuplicatedTestCases::SameNameAndTags ..............................   Passed    0.02 sec
      Start 59: DuplicatedTestCases::SameNameDifferentTags
55/62 Test #59: DuplicatedTestCases::SameNameDifferentTags ........................   Passed    0.02 sec
      Start 60: DuplicatedTestCases::DuplicatedTestCaseMethods
56/62 Test #60: DuplicatedTestCases::DuplicatedTestCaseMethods ....................   Passed    0.02 sec
      Start 61: DuplicatedTestCases::DuplicatedTestCaseMethodsDifferentFixtures
57/62 Test #61: DuplicatedTestCases::DuplicatedTestCaseMethodsDifferentFixtures ...   Passed    0.02 sec
      Start 62: AmalgamatedFileTest
58/62 Test #57: Reporters::selection-is-case-insensitive ..........................   Passed    0.12 sec
59/62 Test #62: AmalgamatedFileTest ...............................................   Passed    0.03 sec
60/62 Test  #1: RunTests ..........................................................   Passed   12.68 sec
61/62 Test #55: BenchmarksInCumulativeReporter ....................................   Passed   30.37 sec
      Start 41: MinDuration::SimpleThreshold
AssignProcessToJobObject: (87) The parameter is incorrect.

Expected behavior
Test should pass.

Reproduction steps
Steps to reproduce the bug.
ctest -j 4 --output-on-failure -C Debug

Platform information:

  • OS: Windows 11 - version 22000
  • Compiler+version: Visual Studio 2022
  • Catch version: devel branch.

Additional context
Add any other context about the problem here.

Running the test under cdb gives the following callstack when the failure occurs:
59/62 Test #43: MinDuration::DurationOverrideNo ................................... Passed 1.15 sec
60/62 Test #1: RunTests .......................................................... Passed 14.07 sec
61/62 Test #55: BenchmarksInCumulativeReporter .................................... Passed 30.99 sec
Start 41: MinDuration::SimpleThreshold
AssignProcessToJobObject: (87) The parameter is incorrect.
(11b48.1489c): Break instruction exception - code 80000003 (first chance)
KERNELBASE!DebugBreak [inlined in KERNELBASE!wil::details::DebugBreak+0x2]:
00007ffa5bbc4a22 cc int 3 0:000> k100 Child-SP RetAddr Call Site (Inline Function) ---------------- KERNELBASE!DebugBreak
000000584b78cba8 00007ff7bdb155fa KERNELBASE!wil::details::DebugBreak+0x2
000000584b78cbb0 00007ff7bdb14bcf ctest+0x7255fa
000000584b78cc00 00007ff7bd52b7c2 ctest+0x724bcf
000000584b78cd50 00007ff7bd530b2b ctest+0x13b7c2
000000584b78cfe0 00007ff7bd534e5f ctest+0x140b2b
000000584b78d2a0 00007ff7bd534fad ctest+0x144e5f
000000584b78d4f0 00007ff7bd5020b9 ctest+0x144fad
000000584b78d550 00007ff7bd500397 ctest+0x1120b9
000000584b78d7b0 00007ff7bd4fba94 ctest+0x110397
000000584b78d9c0 00007ff7bd5302d4 ctest+0x10ba94
000000584b78da60 00007ff7bdb1451e ctest+0x1402d4
000000584b78da90 00007ff7bdb09c74 ctest+0x72451e
000000584b78dac0 00007ff7bdb09d98 ctest+0x719c74
000000584b78daf0 00007ff7bd4ff14d ctest+0x719d98
000000584b78eb90 00007ff7bd4a0a36 ctest+0x10f14d
000000584b78ec20 00007ff7bd4a1046 ctest+0xb0a36
000000584b78ee80 00007ff7bd432637 ctest+0xb1046
000000584b78f170 00007ff7bd422268 ctest+0x42637
000000584b78f360 00007ff7bd435872 ctest+0x32268
000000584b78f530 00007ff7bd40f9d9 ctest+0x45872
000000584b78f750 00007ff7bdba1e30 ctest+0x1f9d9
000000584b78fa70 00007ffa5c271350 ctest+0x7b1e30
000000584b78fab0 00007ffa5e331e78 KERNEL32!BaseThreadInitThunk+0x10
000000584b78fae0 0000000000000000 ntdll!RtlUserThreadStart+0x28

I will attempt to find symbols for this version of ctest.

E:\Catch2\debug-build>ctest --version
ctest version 3.21.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).
@Alan-Jowett
Copy link
Contributor Author

Stack trace of the failure:

0:000> k100
Child-SP          RetAddr               Call Site
(Inline Function) --------`--------     KERNELBASE!DebugBreak [minkernel\kernelbase\debug.c @ 143]
0000005e`0b389638 00007ff7`1c321224     KERNELBASE!wil::details::DebugBreak+0x2 [onecore\internal\sdk\inc\wil\opensource\wil\result_macros.h @ 1835]
0000005e`0b389640 00007ff7`1c31aca0     ctest!uv_fatal_error+0xf4 [E:\cmake\Utilities\cmlibuv\src\win\error.c @ 62]
0000005e`0b3896b0 00007ff7`1b8bc2f4     ctest!uv_spawn+0x8e0 [E:\cmake\Utilities\cmlibuv\src\win\process.c @ 1200]
0000005e`0b3898f0 00007ff7`1b6e0755     ctest!cm::uv_process_ptr::spawn+0x44 [E:\cmake\Source\cmUVHandlePtr.cxx @ 221]
0000005e`0b389920 00007ff7`1b6e9b41     ctest!cmProcess::StartProcess+0x9c5 [E:\cmake\Source\CTest\cmProcess.cxx @ 136]
0000005e`0b38a170 00007ff7`1b6e6433     ctest!cmCTestRunTest::ForkProcess+0x841 [E:\cmake\Source\CTest\cmCTestRunTest.cxx @ 769]
0000005e`0b38a670 00007ff7`1b6e4d93     ctest!cmCTestRunTest::StartTest+0x12d3 [E:\cmake\Source\CTest\cmCTestRunTest.cxx @ 627]
0000005e`0b38b330 00007ff7`1b6657f0     ctest!cmCTestRunTest::StartTest+0x73 [E:\cmake\Source\CTest\cmCTestRunTest.cxx @ 485]
0000005e`0b38b380 00007ff7`1b6659da     ctest!cmCTestMultiProcessHandler::StartTestProcess+0x1050 [E:\cmake\Source\CTest\cmCTestMultiProcessHandler.cxx @ 249]
0000005e`0b38bb90 00007ff7`1b664020     ctest!cmCTestMultiProcessHandler::StartTest+0x18a [E:\cmake\Source\CTest\cmCTestMultiProcessHandler.cxx @ 466]
0000005e`0b38bc30 00007ff7`1b667bdd     ctest!cmCTestMultiProcessHandler::StartNextTests+0x640 [E:\cmake\Source\CTest\cmCTestMultiProcessHandler.cxx @ 574]
0000005e`0b38c6f0 00007ff7`1b6e9042     ctest!cmCTestMultiProcessHandler::FinishTestProcess+0x4bd [E:\cmake\Source\CTest\cmCTestMultiProcessHandler.cxx @ 672]
0000005e`0b38c890 00007ff7`1b6e19ec     ctest!cmCTestRunTest::FinalizeTest+0x72 [E:\cmake\Source\CTest\cmCTestRunTest.cxx @ 899]
0000005e`0b38c8f0 00007ff7`1b6e131b     ctest!cmProcess::Finish+0xac [E:\cmake\Source\CTest\cmProcess.cxx @ 343]
0000005e`0b38c960 00007ff7`1b6e1185     ctest!cmProcess::OnExit+0xab [E:\cmake\Source\CTest\cmProcess.cxx @ 330]
0000005e`0b38c990 00007ff7`1c31b171     ctest!cmProcess::OnExitCB+0x35 [E:\cmake\Source\CTest\cmProcess.cxx @ 303]
0000005e`0b38c9d0 00007ff7`1c2f01c6     ctest!uv_process_proc_exit+0x171 [E:\cmake\Utilities\cmlibuv\src\win\process.c @ 900]
0000005e`0b38ca30 00007ff7`1c2ef075     ctest!uv_process_reqs+0x446 [E:\cmake\Utilities\cmlibuv\src\win\req-inl.h @ 207]
0000005e`0b38caa0 00007ff7`1b661d88     ctest!uv_run+0x65 [E:\cmake\Utilities\cmlibuv\src\win\core.c @ 609]
0000005e`0b38cae0 00007ff7`1b5bfc3f     ctest!cmCTestMultiProcessHandler::RunTests+0x88 [E:\cmake\Source\CTest\cmCTestMultiProcessHandler.cxx @ 138]
0000005e`0b38cb20 00007ff7`1b5b2c87     ctest!cmCTestTestHandler::ProcessDirectory+0xb6f [E:\cmake\Source\CTest\cmCTestTestHandler.cxx @ 1389]
0000005e`0b38d0b0 00007ff7`1b4f20c9     ctest!cmCTestTestHandler::ProcessHandler+0x347 [E:\cmake\Source\CTest\cmCTestTestHandler.cxx @ 416]
0000005e`0b38dab0 00007ff7`1b50c03c     ctest!cmCTest::ProcessSteps+0x4e9 [E:\cmake\Source\cmCTest.cxx @ 971]
0000005e`0b38e1c0 00007ff7`1b4f12ed     ctest!cmCTest::ExecuteTests+0x76c [E:\cmake\Source\cmCTest.cxx @ 2897]
0000005e`0b38e940 00007ff7`1b4e9247     ctest!cmCTest::Run+0xf3d [E:\cmake\Source\cmCTest.cxx @ 2782]
0000005e`0b38f100 00007ff7`1c43a019     ctest!main+0x627 [E:\cmake\Source\ctest.cxx @ 228]
0000005e`0b38f7b0 00007ff7`1c439efe     ctest!invoke_main+0x39 [d:\a01\_work\20\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 79]
0000005e`0b38f800 00007ff7`1c439dbe     ctest!__scrt_common_main_seh+0x12e [d:\a01\_work\20\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288]
0000005e`0b38f870 00007ff7`1c43a0ae     ctest!__scrt_common_main+0xe [d:\a01\_work\20\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 331]
0000005e`0b38f8a0 00007ffa`5c271350     ctest!mainCRTStartup+0xe [d:\a01\_work\20\s\src\vctools\crt\vcstartup\src\startup\exe_main.cpp @ 17]
0000005e`0b38f8d0 00007ffa`5e331e78     KERNEL32!BaseThreadInitThunk+0x10 [clientcore\base\win32\client\thread.c @ 75]
0000005e`0b38f900 00000000`00000000     ntdll!RtlUserThreadStart+0x28 [minkernel\ntdll\rtlstrt.c @ 1166]
0:000>

@horenmar
Copy link
Member

Upstream issue: https://gitlab.kitware.com/cmake/cmake/-/issues/23029

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3rd party bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@horenmar @Alan-Jowett