Assert in ebpfcore (_ebpf_object_tracking_list_remove)

[dv-msft]

Describe the bug

Running the kernel Multi-threaded stress test triggers an assert in ebpfcore.sys

OS information

zn_release_liof (25359.1001.230502-1642)

Steps taken to reproduce bug

Run the kernel mode multi-threaded stress test application with the following command line:
.\ebpf_stress_tests_km -tt=32 -td=10 -er=true -erd=250 native_load_attach_detach_unload_random_v4_test

Expected behavior

The test should succeed. Note that it may fail due to some known ebpf api failures, but it should not cause trigger a kernel mode assert or cause a kernel mode crash.

Actual outcome

The test run triggers a kernel mode assert:

17: kd> k
 # Child-SP          RetAddr               Call Site
00 fffff907`8529e910 fffff803`5197c065     nt!RtlAssert+0xee [minkernel\ntos\rtl\assert.c @ 132] 
01 fffff907`8529ee70 fffff803`5197ce49     ebpfcore!_ebpf_object_tracking_list_remove+0xf5 [D:\wrk\ebpf-for-windows\libs\platform\ebpf_object.c @ 147] 
02 fffff907`8529eec0 fffff803`5195be15     ebpfcore!ebpf_object_release_reference+0x1f9 [D:\wrk\ebpf-for-windows\libs\platform\ebpf_object.c @ 258] 
03 fffff907`8529efa0 fffff803`51952256     ebpfcore!ebpf_core_close_context+0x45 [D:\wrk\ebpf-for-windows\libs\execution_context\ebpf_core.c @ 2499] 
04 fffff907`8529eff0 fffff803`4d412924     ebpfcore!_ebpf_driver_file_close+0x26 [D:\wrk\ebpf-for-windows\ebpfcore\ebpf_drv.c @ 214] 
05 fffff907`8529f030 fffff803`4d414cd3     Wdf01000+0x12924
06 fffff907`8529f0c0 fffff803`4d4114ae     Wdf01000+0x14cd3
07 fffff907`8529f190 fffff803`4a429d15     Wdf01000+0x114ae
08 (Inline Function) --------`--------     nt!IopfCallDriver+0x54 [minkernel\ntos\io\iomgr\iomgr.h @ 3759] 
09 fffff907`8529f1f0 fffff803`4a95b72c     nt!IofCallDriver+0x65 [minkernel\ntos\io\iomgr\iosubs.c @ 3299] 
0a fffff907`8529f230 fffff803`4a90497e     nt!IopDeleteFile+0x13c [minkernel\ntos\io\iomgr\objsup.c @ 868] 
0b fffff907`8529f2b0 fffff803`4a42cd83     nt!ObpRemoveObjectRoutine+0x7e [minkernel\ntos\ob\obref.c @ 3314] 
0c fffff907`8529f310 fffff803`4a9093ae     nt!ObfDereferenceObjectWithTag+0xc3 [minkernel\ntos\ob\obref.c @ 659] 
0d (Inline Function) --------`--------     nt!ObCloseHandleTableEntry+0x24d [minkernel\ntos\ob\obclose.c @ 219] 
0e fffff907`8529f350 fffff803`4a909ac9     nt!ObpCloseHandle+0x31e [minkernel\ntos\ob\obclose.c @ 450] 
0f fffff907`8529f470 fffff803`4a6b6405     nt!NtClose+0x39 [minkernel\ntos\ob\obclose.c @ 517] 
10 fffff907`8529f4a0 00007ff8`9d3f0364     nt!KiSystemServiceCopyEnd+0x25 [minkernel\ntos\ke\amd64\trap.asm @ 3644] 
11 00000049`e97fedb8 00007ff8`9ac01a35     0x00007ff8`9d3f0364
12 00000049`e97fedc0 00000000`00000000     0x00007ff8`9ac01a35

Additional 'point-of-impact' data:

17: kd> dx -r2 object
object                 : 0xffff8088d57d4820 [Type: _ebpf_core_object *]
    [+0x000] base             [Type: _ebpf_base_object]
        [+0x000] marker           : 0x656f626a [Type: unsigned int]
        [+0x004] zero_fill        : 0x0 [Type: unsigned int]
        [+0x008] reference_count  : 0 [Type: __int64]
        [+0x010] acquire_reference : 0xfffff8035197c4e0 : ebpfcore!ebpf_object_acquire_reference+0x0 [Type: void (__cdecl*)(void *)]
        [+0x018] release_reference : 0xfffff8035197cc50 : ebpfcore!ebpf_object_release_reference+0x0 [Type: void (__cdecl*)(void *)]
    [+0x020] type             : EBPF_OBJECT_PROGRAM (3) [Type: _ebpf_object_type]
    [+0x028] free_function    : 0xfffff80351970360 : ebpfcore!_ebpf_program_free+0x0 [Type: void (__cdecl*)(_ebpf_core_object *)]
        ebpfcore!_ebpf_program_free+0x0 [Type: void __cdecl(_ebpf_core_object *)]
    [+0x030] get_program_type : 0xfffff80351970dc0 : ebpfcore!_ebpf_program_get_program_type+0x0 [Type: _GUID (__cdecl*)(_ebpf_core_object *)]
        ebpfcore!_ebpf_program_get_program_type+0x0 [Type: _GUID __cdecl(_ebpf_core_object *)]
    [+0x038] id               : 0x0 [Type: unsigned int]
    [+0x040] object_list_entry [Type: _LIST_ENTRY]
        [+0x000] Flink            : 0xffff8088d57d4860 [Type: _LIST_ENTRY *]
        [+0x008] Blink            : 0xffff8088d57d4860 [Type: _LIST_ENTRY *]
    [+0x050] pinned_path_count : 0 [Type: int]

	
17: kd> dx index
index            : 0x0 [Type: unsigned int]


17: kd> dx _ebpf_id_table[0]
_ebpf_id_table[0]                 [Type: _ebpf_id_entry]
    [+0x000] counter          : 0x0 [Type: unsigned short]
    [+0x008] reference_count  : -1 [Type: __int64]
    [+0x010] object           : 0x0 [Type: _ebpf_core_object *]

Additional details

Dump and symbols are here