Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kernel mode multi-threaded stress test - hang in bpf_object__load #2521

Closed
dv-msft opened this issue May 25, 2023 · 2 comments
Closed

kernel mode multi-threaded stress test - hang in bpf_object__load #2521

dv-msft opened this issue May 25, 2023 · 2 comments
Labels
bug Something isn't working
Milestone

Comments

@dv-msft
Copy link
Collaborator

dv-msft commented May 25, 2023

Describe the bug

Running the native_load_attach_detach_unload_random_v4_test from the kernel mode stress test suite causes the test application to hang.

OS information

No response

Steps taken to reproduce bug

Run the kernel mode multi-threaded stress test with the following command line (Note the 1 (one) minute runtime):
.\ebpf_stress_tests_km -tt=32 -td=1 -erd=1000 native_load_attach_detach_unload_random_v4_test

Expected behavior

The test should pass successfully and terminate after one minute.

Actual outcome

The test hangs and does not terminate after the specified test duration. While there can be some unavoidable variation in the actual runtime (generally in range of 10 - 30 seconds), the test does not terminate even after 3 minutes.

_While not 100 % reproducible, this failure is very frequent.

The hang is seen in bpf_object__load() in the stack trace of the blocked thread:

 # Child-SP          RetAddr               Call Site
00 00000030`676fdff8 00007ff8`53b3bc52     ntdll!ZwAlpcSendWaitReceivePort+0x14 [minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 1347] 
01 00000030`676fe000 00007ff8`53b38311     RPCRT4!UuidCreate+0x6ce2
02 00000030`676fe0b0 00007ff8`53b5e0e8     RPCRT4!UuidCreate+0x33a1
03 00000030`676fe100 00007ff8`53b11466     RPCRT4!RpcAsyncGetCallStatus+0x188
04 00000030`676fe130 00007ff8`53b1191f     RPCRT4!NdrpClientCall2+0x456
05 00000030`676fe950 00007ff8`54979f04     RPCRT4!NdrClientCall2+0x1f
06 00000030`676fe980 00007ff8`0875dc9f     sechost!ControlService+0x24
07 00000030`676fe9d0 00007ff8`0877fdea     EbpfApi!Platform::_stop_service+0x6f [D:\wrk\ebpf-for-windows\libs\thunk\windows\platform.cpp @ 294] 
08 00000030`676fea70 00007ff8`0878d5da     EbpfApi!_ebpf_program_load_native+0x16ea [D:\wrk\ebpf-for-windows\libs\api\ebpf_api.cpp @ 3165] 
09 00000030`676ff2c0 00007ff8`087c3c24     EbpfApi!ebpf_object_load+0x3da [D:\wrk\ebpf-for-windows\libs\api\ebpf_api.cpp @ 2727] 
0a 00000030`676ff420 00007ff6`8f494d52     EbpfApi!bpf_object__load+0x14 [D:\wrk\ebpf-for-windows\libs\api\libbpf_object.cpp @ 126] 
0b 00000030`676ff460 00007ff6`8f49a23b     ebpf_stress_tests_km!_do_creator_work+0x2d2 [D:\wrk\ebpf-for-windows\tests\stress\km\stress_tests_km.cpp @ 393] 
0c 00000030`676ff5d0 00007ff6`8f48a8ce     ebpf_stress_tests_km!_test_thread_function+0x1ab [D:\wrk\ebpf-for-windows\tests\stress\km\stress_tests_km.cpp @ 585] 
0d 00000030`676ff860 00007ff6`8f4812cf     ebpf_stress_tests_km!std::invoke<void (__cdecl*)(thread_context &),std::reference_wrapper<thread_context> >+0x3e [C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.36.32532\include\type_traits @ 1601] 
0e 00000030`676ff8a0 00007ff8`0ff03010     ebpf_stress_tests_km!std::thread::_Invoke<std::tuple<void (__cdecl*)(thread_context &),std::reference_wrapper<thread_context> >,0,1>+0x8f [C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.36.32532\include\thread @ 56] 
0f 00000030`676ff920 00007ff8`53f53db1     ucrtbased!thread_start<unsigned int (__cdecl*)(void *),1>+0xb0 [minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 97] 
10 00000030`676ff980 00007ff8`54e932a1     KERNEL32!BaseThreadInitThunk+0x21 [clientcore\base\win32\client\thread.c @ 77] 
11 00000030`676ff9b0 00000000`00000000     ntdll!RtlUserThreadStart+0x21 [minkernel\ldr\rtlstrt.c @ 1198] 

Additional details

No response

@dv-msft dv-msft added the bug Something isn't working label May 25, 2023
@dv-msft dv-msft added this to the 2306 milestone May 25, 2023
@saxena-anurag
Copy link
Contributor

This is same as #2457. Corresponding PR: #2500

@dv-msft
Copy link
Collaborator Author

dv-msft commented May 26, 2023

Closing as 'duplicate'.

@dv-msft dv-msft closed this as not planned Won't fix, can't repro, duplicate, stale May 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants