Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

datapath-windows: BSOD cause by Driver Verifier due to memory leaks #50

Closed
svinturis opened this issue Oct 27, 2014 · 2 comments
Closed

datapath-windows: BSOD cause by Driver Verifier due to memory leaks #50

svinturis opened this issue Oct 27, 2014 · 2 comments
Assignees
@svinturis
Labels
bug

Comments

@svinturis
Copy link 

  •                                                                         *

  •                    Bugcheck Analysis                                    *

  •                                                                         *

DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
A device driver attempting to corrupt the system has been caught. This is
because the driver was specified in the registry as being suspect (by the
administrator) and the kernel has enabled substantial checking of this driver.
If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA will
be among the most commonly seen crashes.
Arguments:
Arg1: 0000000000000062, A driver has forgotten to free its pool allocations prior to unloading.
Arg2: ffffe00001b75978, name of the driver having the issue.
Arg3: ffffe00001bd56b0, verifier internal structure with driver information.
Arg4: 0000000000000005, total # of (paged+nonpaged) allocations that weren't freed.
Type !verifier 3 drivername.sys for info on the allocations
that were leaked that caused the bugcheck.

Debugging Details:

DBGHELP: C:\Windows\symbols\ntkrnlmp.exe\5215D156783000\ntkrnlmp.exe - OK
DBGHELP: C:\Windows\symbols\ntkrnlmp.exe\5215D156783000\ntkrnlmp.exe - OK

BUGCHECK_STR: 0xc4_62

IMAGE_NAME: OVSExt.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 5447adb7

MODULE_NAME: OVSExt

FAULTING_MODULE: fffff80001779000 OVSExt

VERIFIER_DRIVER_ENTRY: dt nt!_MI_VERIFIER_DRIVER_ENTRY ffffe00001bd56b0
Symbol nt!_MI_VERIFIER_DRIVER_ENTRY not found.

DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT

PROCESS_NAME: services.exe

CURRENT_IRQL: 2

ANALYSIS_VERSION: 6.3.9600.17237 (debuggers(dbg).140716-0327) amd64fre

LAST_CONTROL_TRANSFER: from fffff80212e5a7c6 to fffff80212dd7c90

STACK_TEXT:
ffffd000261e0b18 fffff80212e5a7c6 : 0000000000000000 0000000000000000 ffffd000261e0c80 fffff80212d7f654 : nt!DbgBreakPointWithStatus
ffffd000261e0b20 fffff80212e5a0d7 : 0000000000000003 0000000000000062 fffff80212ddf070 00000000000000c4 : nt!KiBugCheckDebugBreak+0x12
ffffd000261e0b80 fffff80212dd11a4 : 00000000000000c4 fffff80212d35f6c ffffe00000059870 fffff80212d38b73 : nt!KeBugCheck2+0x8ab
ffffd000261e1290 fffff802132f06a8 : 00000000000000c4 0000000000000062 ffffe00001b75978 ffffe00001bd56b0 : nt!KeBugCheckEx+0x104
ffffd000261e12d0 fffff802132f4c4a : fffff80001779000 ffffe00001b758c0 0000000000000000 00000000ffffffff : nt!VerifierBugCheckIfAppropriate+0x3c
ffffd000261e1310 fffff80212e2d2e8 : 0000000000000000 0000000000001000 0000007f00000001 ffffe00001bc6fa0 : nt!VfPoolCheckForLeaks+0x4a
ffffd000261e1350 fffff802132e2060 : fffff80212f2dcc0 fffff80212f2dcc0 ffffe00001b758c0 0000000000000002 : nt! ?? ::FNODOBFM::string'+0x4bd58 ffffd000261e13e0 fffff802131556c6 : 0000000000000000 ffffe00001b758c0 0000000000000000 ffffe00001bd5410 : nt!VfDriverUnloadImage+0x34 ffffd000261e1410 fffff80213155630 : 0000000000000000 ffffe00001b758c0 ffffe00001bd5410 ffffe00001bd53e0 : nt!MiUnloadSystemImage+0x7e ffffd000261e1490 fffff80213155578 : 0000000000000000 ffffe000001b2f20 ffffe00001bd5410 0000000000010286 : nt!MmUnloadSystemImage+0x20 ffffd000261e14c0 fffff80213032f28 : 0000000000000000 ffffe00001bd5410 ffffe000001b2f20 ffffc0000007e918 : nt!IopDeleteDriver+0x40 ffffd000261e1500 fffff80212ce905f : 0000000000000000 ffffd000261e1800 ffffe00001bd5410 00000000c0000001 : nt!ObpRemoveObjectRoutine+0x64 ffffd000261e1560 fffff80213159a2e : ffffe00001bd5410 ffffd000261e19a0 00000000c0000001 00000000c0000001 : nt!ObfDereferenceObject+0x8f ffffd000261e15a0 fffff80212ddc8b3 : 0000000000000000 ffffe000023c9080 0000008686ebe950 0000000000000000 : nt!IopUnloadDriver+0x262 ffffd000261e1780 fffff80212dd4d00 : fffff8021321de61 ffffe000023c9080 ffffd000261e1b80 0000008686ebe950 : nt!KiSystemServiceCopyEnd+0x13 ffffd000261e1918 fffff8021321de61 : ffffe000023c9080 ffffd000261e1b80 0000008686ebe950 00000000000001c8 : nt!KiServiceLinkage ffffd000261e1920 fffff80212ddc8b3 : ffffe000023c9080 ffffe000023c9080 00000086861bf450 0000000000000000 : nt! ?? ::NNGAKEGL::string'+0x70261
ffffd000261e1b00 00007ffc7e0e94fa : 00007ff7c8841c90 0000000000000078 0000000000000078 00000086861bf450 : nt!KiSystemServiceCopyEnd+0x13
0000008686ebe928 00007ff7c8841c90 : 0000000000000078 0000000000000078 00000086861bf450 00007ff7c88103f4 : ntdll!NtUnloadDriver+0xa
0000008686ebe930 00007ff7c8841b68 : 000000000000000a 0000000000000000 00007ff7c8853368 0000008686ebeaa0 : services!ScUnloadDriver+0xb4
0000008686ebe970 00007ff7c88285d6 : 0000000000000000 0000000000000000 00000086862082a0 367abb810000000d : services!ScControlDriver+0x110
0000008686ebe9a0 00007ff7c88065b8 : 0000000000000001 0000000000000001 00007ff7c880f0fc 00007ffc00000000 : services!RI_ScGetCurrentGroupStateW+0x5272
0000008686ebea60 00007ffc7db62385 : 0000000000000000 0000008686ebef88 0000008686ebf0d0 00007ffc7db62346 : services!RControlService+0x50
0000008686ebeae0 00007ffc7db6ae16 : 0000008686ebef70 00007ff7c880f0f2 00000086862338d0 0000008600000001 : RPCRT4!Invoke+0x65
0000008686ebeb40 00007ffc7db634ea : 0000000000000013 0000000000000000 00000000000002b0 00007ffcffffffff : RPCRT4!NdrStubCall2+0x38b
0000008686ebf1c0 00007ffc7db62614 : 0000008686271b01 00007ffc00000001 0000000000000031 0000000000000011 : RPCRT4!NdrServerCall2+0x1a
0000008686ebf1f0 00007ffc7db62517 : 0000000000000000 0000008686ebf2e1 0000008686ebf3a8 0000000000000001 : RPCRT4!DispatchToStubInCNoAvrf+0x14
0000008686ebf240 00007ffc7db630ad : 00000086861e0270 0000000000000000 0000000000000000 0000008686233780 : RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x177
0000008686ebf340 00007ffc7db62cc1 : 0000000000025fa6 0000008686233780 0000000000000000 00000086861e0270 : RPCRT4!LRPC_SCALL::DispatchRequest+0x1bd
0000008686ebf440 00007ffc7db62a97 : 0000001100000000 0000000000000001 0000008600000000 0000000000000000 : RPCRT4!LRPC_SCALL::HandleRequest+0x201
0000008686ebf4f0 00007ffc7db61d04 : 00000086861ddf00 0000008600000000 00007ffc7dbb9b24 00000086861ddf00 : RPCRT4!LRPC_SASSOCIATION::HandleRequest+0x237
0000008686ebf580 00007ffc7db61afe : 0000008600000001 0000000000000000 000000007ffe03c0 00007ffc7dbb9b24 : RPCRT4!LRPC_ADDRESS::ProcessIO+0x36d
0000008686ebf6c0 00007ffc7e074624 : 000000000000003e fffffffffd050f80 000000007ffe03b0 0000008686ebf778 : RPCRT4!LrpcIoComplete+0xae
0000008686ebf760 00007ffc7e072bfd : 0000000000000000 0000000000000000 0000000000000000 000000868626b3f0 : ntdll!TppAlpcpExecuteCallback+0x204
0000008686ebf7d0 00007ffc7d021611 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : ntdll!TppWorkerThread+0x3ad
0000008686ebfbc0 00007ffc7e0c64ad : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : KERNEL32!BaseThreadInitThunk+0xd
0000008686ebfbf0 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : ntdll!RtlUserThreadStart+0x1d

STACK_COMMAND: kb

FOLLOWUP_NAME: MachineOwner

FAILURE_BUCKET_ID: 0xc4_62_VRF_LEAKED_POOL_IMAGE_OVSExt.sys

BUCKET_ID: 0xc4_62_VRF_LEAKED_POOL_IMAGE_OVSExt.sys

ANALYSIS_SOURCE: KM

FAILURE_ID_HASH_STRING: km:0xc4_62_vrf_leaked_pool_image_ovsext.sys

FAILURE_ID_HASH: {4bcf06aa-3750-7f0f-72b4-698f3e1db606}

Followup: MachineOwner

0: kd> !verifier 3 OVSExt.sys

Verify Flags Level 0x01aeefbf

STANDARD FLAGS:
X Automatic Checks
X Special pool
X Force IRQL checking
X Pool tracking
X I/O verification
X Deadlock detection
X DMA checking
X Security checks
X Miscellaneous checks
X DDI compliance checking

ADDITIONAL FLAGS:
X Randomized low resources simulation
X Force pending I/O requests
X IRP logging
X Invariant MDL checking for stack
X Invariant MDL checking for driver
X Power framework delay fuzzing
X Systematic low resources simulation
X DDI compliance checking (additional)
X NDIS/WIFI verification
X Kernel synchronization delay fuzzing
X VM switch verification

[X] Indicates flag is enabled

Summary of All Verifier Statistics

RaiseIrqls 0x2aaf
AcquireSpinLocks 0x45cb
Synch Executions 0xf6
Trims 0x2240

Pool Allocations Attempted 0x10a04
Pool Allocations Succeeded 0x109b3
Pool Allocations Succeeded SpecialPool 0x109b3
Pool Allocations With NO TAG 0x0
Pool Allocations Failed 0x0

Current paged pool allocations 0x95 for 0000283A bytes
Peak paged pool allocations 0x9e for 00002B76 bytes
Current nonpaged pool allocations 0xfa6 for 00115015 bytes
Peak nonpaged pool allocations 0xfab for 00115513 bytes

Driver Verification List

MODULE: 0xffffe00000059820 ovsext.sys (Loaded)

Pool Allocation Statistics: ( NonPagedPool / PagedPool )

  Current Pool Allocations: ( 0x00000005 / 0x00000000 )
  Current Pool Bytes:       ( 0x00018050 / 0x00000000 )
  Peak Pool Allocations:    ( 0x00000009 / 0x00000000 )
  Peak Pool Bytes:          ( 0x0001e1f8 / 0x00000000 )
  Contiguous Memory Bytes:       0x00000000
  Peak Contiguous Memory Bytes:  0x00000000

Pool Allocations:

  Address             Length      Tag   Caller Address    
  ------------------  ----------  ----  ------------------
  0xffffe00002933000  0x00008010  OVST  0xfffff8000179a4ae  OVSExt!OvsAllocateMemory
  0xffffe0000292e000  0x00004010  OVST  0xfffff8000179a4ae  OVSExt!OvsAllocateMemory
  0xffffe00002929000  0x00004010  OVST  0xfffff8000179a4ae  OVSExt!OvsAllocateMemory
  0xffffe00002924000  0x00004010  OVST  0xfffff8000179a4ae  OVSExt!OvsAllocateMemory
  0xffffe0000291f000  0x00004010  OVST  0xfffff8000179a4ae  OVSExt!OvsAllocateMemory

Contiguous allocations are not displayed with public symbols.
@svinturis svinturis self-assigned this Oct 27, 2014
@svinturis
Copy link
Author

At machine boot, If the OVS extension is enabled and the Driver Verifier is set, a BSOD will be issued due to memory leaks. The PNP manager calls filter attach routine before the RPC engine is ready, which causes tunnel initialization to fail in OvsTunnelFilterInitialize. In this case, the switch context is freed without performing a cleanup first. This is the reason why the memory leaks are hapening.

The five leaks are from the datapath, portIdHashArray, portNoHashArray, pidHashArray, and ovsPortNameHashArray, members of the switch context which are not released.

@svinturis svinturis added the bug label Oct 27, 2014
blp pushed a commit to openvswitch/ovs that referenced this issue Oct 28, 2014
@svinturis @blp
datapath-windows: Fix BSOD caused by DV due to memory leaks.
3a6de11 
If the OVS extension is enabled, Driver Verifier will issue a BSOD
due to memory leaks. This issue reproduces each time and the problem
is in the filter attach routine when the switch context is initialized.

Signed-off-by: Sorin Vinturis <svinturis@cloudbasesolutions.com>
Reported-by: Sorin Vinturis <svinturis@cloudbasesolutions.com>
Reported-at: openvswitch/ovs-issues#50
Acked-by: Eitan Eliahu <eliahue@vmware.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
@svinturis
Copy link
Author

Closing this issue as it was fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@svinturis svinturis

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@svinturis