Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kernel mode multi-threaded stress test (native_invoke_v4_v6_programs_restart_extension_test) failure. #3191

Closed
dv-msft opened this issue Jan 22, 2024 · 1 comment
Assignees
Labels
bug Something isn't working triaged Discussed in a triage meeting
Milestone

Comments

@dv-msft
Copy link
Collaborator

dv-msft commented Jan 22, 2024

Describe the bug

The test native_invoke_v4_v6_programs_restart_extension_test is failing with a EBPF_FILE_NOT_FOUND error when it tries to open its ebpf test programs, i.e. cgroup_count_connect4.sys and cgroup_count_connect6.sys.

The etl traces also show the same error for a prior test (for the cgroup_sock_addr.sys ebpf program) that for reason, does not trigger this check.

Notes:

  1. This error is seen in the CI/CD runs only. Local repros attempts have not been successful.
  2. The failures is seen in both flavors (Debug and Release).
  3. This failure terminates the test run prematurely but the test run is still marked as successful.

OS information

No response

Steps taken to reproduce bug

Seen in a draft PR.

Expected behavior

This failure should not be seen.

Actual outcome

The test terminates prematurely. The premature termination produces a crash dump.

Stack dump (debug build): problem frame is 20:

00 000000f6`804f3af8 00007ffa`1b8cc01e     ntdll!ZwWaitForMultipleObjects+0x14 [minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 907] 
01 000000f6`804f3b00 00007ffa`1b8cbf0e     KERNELBASE!WaitForMultipleObjectsEx+0xfe [minkernel\kernelbase\synch.c @ 1551] 
02 000000f6`804f3e00 00007ffa`1de57b1f     KERNELBASE!WaitForMultipleObjects+0xe [minkernel\kernelbase\synch.c @ 1403] 
03 000000f6`804f3e40 00007ffa`1de575cc     kernel32!WerpReportFaultInternal+0x51b [onecore\windows\feedback\core\faultrep\lib\faultrep.cpp @ 1006] 
04 000000f6`804f3f60 00007ffa`1b8f554e     kernel32!WerpReportFault+0xac [onecore\windows\feedback\core\faultrep\lib\faultrep.cpp @ 1313] 
05 000000f6`804f3fa0 00007ffa`1e987100     KERNELBASE!UnhandledExceptionFilter+0x35e [minkernel\kernelbase\xcpt.c @ 685] 
06 (Inline Function) --------`--------     ntdll!RtlpThreadExceptionFilter+0x99 [minkernel\ntdll\rtlstrt.c @ 967] 
07 000000f6`804f40b0 00007ffa`1e96f306     ntdll!RtlUserThreadStart$filt$0+0xaa [minkernel\ntdll\rtlstrt.c @ 1164] 
08 000000f6`804f40f0 00007ffa`1e9835af     ntdll!__C_specific_handler+0x96 [minkernel\crts\crtw32\misc\riscchandler.c @ 352] 
09 000000f6`804f4160 00007ffa`1e8e4aaf     ntdll!RtlpExecuteHandlerForException+0xf [minkernel\ntos\rtl\amd64\xcptmisc.asm @ 132] 
0a 000000f6`804f4190 00007ffa`1e8e88a6     ntdll!RtlDispatchException+0x40f [minkernel\ntos\rtl\amd64\exdsptch.c @ 628] 
0b 000000f6`804f48c0 00007ffa`1b8c5299     ntdll!RtlRaiseException+0x316 [minkernel\ntos\rtl\amd64\raise.c @ 182] 
0c 000000f6`804f50c0 00007ff7`4ed573b6     KERNELBASE!RaiseException+0x69 [minkernel\kernelbase\xcpt.c @ 937] 
0d 000000f6`804f51a0 00007ff9`fcd56e31     ebpf_stress_tests_km!_wer_report::_terminate_hook+0x26 [D:\a\ebpf-for-windows\ebpf-for-windows\tests\libs\util\wer_report.hpp @ 91] 
0e 000000f6`804f51d0 00007ff9`fcd55610     ucrtbased!_VCrtDbgReportW+0x841 [minkernel\crts\ucrt\src\appcrt\misc\dbgrptt.cpp @ 613] 
0f 000000f6`804fd310 00007ff9`fcd42fa1     ucrtbased!_CrtDbgReportW+0x60 [minkernel\crts\ucrt\src\appcrt\misc\dbgrpt.cpp @ 278] 
10 000000f6`804fd370 00007ff9`fcd43153     ucrtbased!issue_debug_notification+0x31 [minkernel\crts\ucrt\src\appcrt\internal\report_runtime_error.cpp @ 25] 
11 000000f6`804fd3c0 00007ff9`fcd5ae2d     ucrtbased!__acrt_report_runtime_error+0x13 [minkernel\crts\ucrt\src\appcrt\internal\report_runtime_error.cpp @ 154] 
12 000000f6`804fd420 00007ff9`fcd59db0     ucrtbased!abort+0x1d [minkernel\crts\ucrt\src\appcrt\startup\abort.cpp @ 61] 
13 000000f6`804fd460 00007ffa`0cae223a     ucrtbased!terminate+0x40 [minkernel\crts\ucrt\src\appcrt\misc\terminate.cpp @ 59] 
14 000000f6`804fd4a0 00007ffa`0cae2ec5     VCRUNTIME140_1D!FindHandler<__FrameHandler4>+0x60a [D:\a\_work\1\s\src\vctools\crt\vcruntime\src\eh\frame.cpp @ 827] 
15 000000f6`804fd680 00007ffa`0cae2f5a     VCRUNTIME140_1D!__InternalCxxFrameHandler<__FrameHandler4>+0x475 [D:\a\_work\1\s\src\vctools\crt\vcruntime\src\eh\frame.cpp @ 401] 
16 000000f6`804fd750 00007ffa`0cae6dfb     VCRUNTIME140_1D!__InternalCxxFrameHandlerWrapper<__FrameHandler4>+0x6a [D:\a\_work\1\s\src\vctools\crt\vcruntime\src\eh\frame.cpp @ 236] 
17 000000f6`804fd7b0 00007ff7`4ee6c8b0     VCRUNTIME140_1D!__CxxFrameHandler4+0xfb [D:\a\_work\1\s\src\vctools\crt\vcruntime\src\eh\risctrnsctrl.cpp @ 304] 
18 000000f6`804fd840 00007ffa`1e9835af     ebpf_stress_tests_km!__GSHandlerCheck_EH4+0x90 [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\gshandlereh4.cpp @ 73] 
19 000000f6`804fd890 00007ffa`1e8e4aaf     ntdll!RtlpExecuteHandlerForException+0xf [minkernel\ntos\rtl\amd64\xcptmisc.asm @ 132] 
1a 000000f6`804fd8c0 00007ffa`1e8e88a6     ntdll!RtlDispatchException+0x40f [minkernel\ntos\rtl\amd64\exdsptch.c @ 628] 
1b 000000f6`804fdff0 00007ffa`1b8c5299     ntdll!RtlRaiseException+0x316 [minkernel\ntos\rtl\amd64\raise.c @ 182] 
1c 000000f6`804fe7f0 00007ffa`0c8abbf1     KERNELBASE!RaiseException+0x69 [minkernel\kernelbase\xcpt.c @ 937] 
1d 000000f6`804fe8d0 00007ff7`4ee00538     VCRUNTIME140D!_CxxThrowException+0x131 [D:\a\_work\1\s\src\vctools\crt\vcruntime\src\eh\throw.cpp @ 82] 
1e 000000f6`804fe960 00007ff7`4eda421d     ebpf_stress_tests_km!Catch::throw_test_failure_exception+0x28 [D:\a\ebpf-for-windows\ebpf-for-windows\external\Catch2\src\catch2\internal\catch_test_failure_exception.cpp @ 21] 
1f 000000f6`804fe9a0 00007ff7`4ed7cc66     ebpf_stress_tests_km!Catch::AssertionHandler::complete+0x4d [D:\a\ebpf-for-windows\ebpf-for-windows\external\Catch2\src\catch2\internal\catch_assertion_handler.cpp @ 52] 
20 000000f6`804fe9e0 00007ff7`4ed8161e     ebpf_stress_tests_km!_load_attach_program+0x2c6 [D:\a\ebpf-for-windows\ebpf-for-windows\tests\stress\km\stress_tests_km.cpp @ 754] 
21 000000f6`804ff2f0 00007ff7`4ed7bf5b     ebpf_stress_tests_km!_prep_program+0x9e [D:\a\ebpf-for-windows\ebpf-for-windows\tests\stress\km\stress_tests_km.cpp @ 834] 
22 000000f6`804ff590 00007ff7`4ed6f9be     ebpf_stress_tests_km!_invoke_test_thread_function+0x4b [D:\a\ebpf-for-windows\ebpf-for-windows\tests\stress\km\stress_tests_km.cpp @ 871] 
23 000000f6`804ffcf0 00007ff7`4ed63cff     ebpf_stress_tests_km!std::invoke<void (__cdecl*)(thread_context &),std::reference_wrapper<thread_context> >+0x3e [C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.37.32822\include\type_traits @ 1777] 
24 000000f6`804ffd30 00007ff9`fcd63010     ebpf_stress_tests_km!std::thread::_Invoke<std::tuple<void (__cdecl*)(thread_context &),std::reference_wrapper<thread_context> >,0,1>+0x8f [C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.37.32822\include\thread @ 56] 
25 000000f6`804ffdb0 00007ffa`1de07e94     ucrtbased!thread_start<unsigned int (__cdecl*)(void *),1>+0xb0 [minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 97] 
26 000000f6`804ffe10 00007ffa`1e947ad1     kernel32!BaseThreadInitThunk+0x14 [base\win32\client\thread.c @ 64] 
27 000000f6`804ffe40 00000000`00000000     ntdll!RtlUserThreadStart+0x21 [minkernel\ntdll\rtlstrt.c @ 1163]

Additional details

No response

@dv-msft dv-msft added the bug Something isn't working label Jan 22, 2024
@dv-msft dv-msft added this to the 2401 milestone Jan 22, 2024
@dv-msft dv-msft self-assigned this Jan 22, 2024
@dahavey dahavey added triaged Discussed in a triage meeting blocked Blocked on another issue that must be done first labels Jan 29, 2024
@dahavey dahavey modified the milestones: 2401, 2402 Jan 29, 2024
@dv-msft dv-msft removed the blocked Blocked on another issue that must be done first label Feb 4, 2024
@dv-msft
Copy link
Collaborator Author

dv-msft commented Feb 4, 2024

Fixed by PR #3206, #3214

@dv-msft dv-msft closed this as completed Feb 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triaged Discussed in a triage meeting
Projects
None yet
Development

No branches or pull requests

2 participants