Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BSOD when loading an activated extension #67

Closed
svinturis opened this issue Jan 21, 2015 · 1 comment
Closed

BSOD when loading an activated extension #67

svinturis opened this issue Jan 21, 2015 · 1 comment
Assignees
@svinturis
Labels
bug

Comments

@svinturis
Copy link

Load and enable the OVS extension driver. Unload the enabled extension, without disabling it. When the driver is loaded again, the BSOD occurs.

Here are the PS commands to reproduce this:
PS C:> net start ovsext
PS C:> Enable-VMSwitchExtension "Open vSwitch Extension"
PS C:> net stop ovsext
PS C:> net start ovsext ----> BSOD

The BSOD is triggered because the filter attach routine tries to acquire the control lock, when the lock is not yet initialized (gOvsCtrlLock = NULL). This happens because the filter attach request comes before the driver finishes initialization, in OvsInit().

Below is the stack trace:

1: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 0000000000000000, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000001, bitfield :
    bit 0 : value 0 = read operation, 1 = write operation
    bit 3 : value 0 = not an execute operation, 1 = execute operation (only on chips which support this level of status)
Arg4: fffff802b050e28c, address which referenced memory

Debugging Details:
------------------

DBGHELP: c:\windows\symbols\NDIS.SYS\5215F825118000\NDIS.SYS - OK
DBGHELP: c:\windows\symbols\NDIS.SYS\5215F825118000\NDIS.SYS - OK
SYMSRV:  c:\windows\symbols\OVSExt.sys\54BEC21339000\OVSExt.sys not found
SYMSRV:  c:\windows\symbols\OVSExt.sys\54BEC21339000\OVSExt.sys not found
SYMSRV:  http://msdl.microsoft.com/download/symbols/OVSExt.sys/54BEC21339000/OVSExt.sys not found
SYMSRV:  http://msdl.microsoft.com/download/symbols/OVSExt.sys/54BEC21339000/OVSExt.sys not found
DBGHELP: C:\1.data\Cloudbase\Work\Git\ovs\datapath-windows\ovsext\x64\Win8.1Debug\OVSExt.sys - OK
DBGHELP: C:\1.data\Cloudbase\Work\Git\ovs\datapath-windows\ovsext\x64\Win8.1Debug\OVSExt.sys - OK
DBGHELP: c:\windows\symbols\ntkrnlmp.exe\5215D156783000\ntkrnlmp.exe - OK
DBGHELP: c:\windows\symbols\ntkrnlmp.exe\5215D156783000\ntkrnlmp.exe - OK

WRITE_ADDRESS:  0000000000000000 

CURRENT_IRQL:  2

FAULTING_IP: 
nt!KeAcquireSpinLockRaiseToDpc+1c
fffff802`b050e28c f0480fba2900    lock bts qword ptr [rcx],0

DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT

BUGCHECK_STR:  AV

PROCESS_NAME:  System

ANALYSIS_VERSION: 6.3.9600.17237 (debuggers(dbg).140716-0327) amd64fre

TRAP_FRAME:  ffffd000201e5b90 -- (.trap 0xffffd000201e5b90)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000002 rbx=0000000000000000 rcx=0000000000000000
rdx=0000000000000004 rsi=0000000000000000 rdi=0000000000000000
rip=fffff802b050e28c rsp=ffffd000201e5d20 rbp=ffffd000201e5f40
 r8=0000000000000050  r9=0000000000000001 r10=00000000ffffffff
r11=ffffd000201e5b68 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
nt!KeAcquireSpinLockRaiseToDpc+0x1c:
fffff802`b050e28c f0480fba2900    lock bts qword ptr [rcx],0 ds:00000000`00000000=????????????????
Resetting default scope

LAST_CONTROL_TRANSFER:  from fffff802b06527c6 to fffff802b05cfc90

STACK_TEXT:  
ffffd000`201e5298 fffff802`b06527c6 : 00000000`00000000 00000000`00000000 ffffd000`201e5400 fffff802`b0577654 : nt!DbgBreakPointWithStatus
ffffd000`201e52a0 fffff802`b06520d7 : 00000000`00000003 ffffd000`201e5400 fffff802`b05d7070 00000000`0000000a : nt!KiBugCheckDebugBreak+0x12
ffffd000`201e5300 fffff802`b05c91a4 : 00000000`00000000 fffff802`b05b80a5 ffffd000`20800180 00000000`00000000 : nt!KeBugCheck2+0x8ab
ffffd000`201e5a10 fffff802`b05d4be9 : 00000000`0000000a 00000000`00000000 00000000`00000002 00000000`00000001 : nt!KeBugCheckEx+0x104
ffffd000`201e5a50 fffff802`b05d343a : 00000000`00000001 00000000`00000000 00000000`00000000 ffffd000`201e5b90 : nt!KiBugCheckDispatch+0x69
ffffd000`201e5b90 fffff802`b050e28c : ffffe000`0287cdf8 fffff802`00040002 ffff554d`b9dc6ad5 ffffe000`0143aa20 : nt!KiPageFault+0x23a
ffffd000`201e5d20 fffff800`0207041c : 00000000`00000002 00000000`00000002 fffff800`02080c00 ffffc000`00000048 : nt!KeAcquireSpinLockRaiseToDpc+0x1c
ffffd000`201e5d50 fffff800`007398ca : ffffe000`0143aa20 ffffe000`023f2460 ffffd000`201e5f30 ffffe000`00f6d1a0 : OVSExt!OvsExtAttach+0x12c [c:\1.data\cloudbase\work\git\ovs\datapath-windows\ovsext\switch.c @ 86]
ffffd000`201e5dc0 fffff800`0076c391 : ffffe000`00f6e160 ffffe000`011c2d90 00000000`00000000 ffffd000`201e5f40 : NDIS!ndisFInvokeAttach+0x36
ffffd000`201e5df0 fffff800`00739f9f : ffffe000`02138000 00000000`00000000 ffffe000`00cd4548 00000000`00000001 : NDIS!ndisAttachFilterInner+0x901
ffffd000`201e61c0 fffff800`00728bcb : ffffe000`00f6e5e8 ffffe000`00f6e5e8 00000000`00000000 00000000`000013c0 : NDIS!ndisAttachFilter+0x4b
ffffd000`201e62c0 fffff800`0072a434 : ffffe000`00f6d1a0 ffffe000`00f6d1a0 ffffd000`201e6500 ffffe000`00f6e5e8 : NDIS!Ndis::BindEngine::Iterate+0x74b
ffffd000`201e64d0 fffff800`00729840 : ffffe000`00f6e5e8 fffff800`02086b90 ffffd000`201e6550 fffff800`00727a69 : NDIS!Ndis::BindEngine::UpdateBindings+0x64
ffffd000`201e6500 fffff800`0072a887 : ffffe000`00f6e5e8 ffffc000`0026c948 ffffe000`00f6d1a0 fffff800`007294e8 : NDIS!Ndis::BindEngine::DispatchPendingWork+0x50
ffffd000`201e6530 fffff800`0076385a : ffffc000`00c7ff88 ffffd000`201e65d0 ffffc000`0026c948 ffffe000`00f6d11e : NDIS!Ndis::BindEngine::ApplyBindChanges+0x33
ffffd000`201e6580 fffff800`00737766 : ffffc000`00c7ffd8 fffff800`02086b90 ffffc000`0026c948 ffffe000`00f6d1a0 : NDIS!<lambda_1ce06b2b40968439b229a98218e85867>::operator()+0x1b
ffffd000`201e65b0 fffff800`007376af : 00000000`00000000 fffff800`02086b90 ffffe000`011c2d90 ffffc000`0026c940 : NDIS!NDIS_BIND_DRIVER_BASE::ForEachLink+0x9e
ffffd000`201e6600 fffff800`00738d2a : ffffe000`011c2d90 ffffd000`201e6780 00000000`00000000 00000000`00000000 : NDIS!NDIS_BIND_DRIVER_BASE::SetRunningDriverIsReady+0x3f
ffffd000`201e6630 fffff800`006b364b : 00000000`00000000 fffff800`02086b90 ffffd000`00000000 ffffe000`011c2d90 : NDIS!NDIS_BIND_FILTER_DRIVER::SetRunningDriver+0x46
ffffd000`201e6680 fffff800`0205e157 : 00470045`0052005c ffffe000`023f2460 00000000`00000000 00000000`00000000 : NDIS!NdisFRegisterFilterDriver+0x3af
ffffd000`201e6720 fffff802`b08b901e : ffffe000`023f2460 ffffe000`01e48000 00000000`00000000 00000000`000007ff : OVSExt!DriverEntry+0x1d7 [c:\1.data\cloudbase\work\git\ovs\datapath-windows\ovsext\driver.c @ 140]
ffffd000`201e6850 fffff802`b0936292 : 00000000`00000000 00000000`00000000 fffff802`b071b1c0 ffffe000`023e9040 : nt!IopLoadDriver+0x5e2
ffffd000`201e6b10 fffff802`b04b53cd : fffff802`00000000 ffffffff`80000530 fffff802`b0936244 ffffe000`00ee7320 : nt!IopLoadUnloadDriver+0x4e
ffffd000`201e6b50 fffff802`b0560664 : 00000000`00000000 ffffe000`023e9040 ffffe000`023e9040 ffffe000`00124940 : nt!ExpWorkerThread+0x2b5
ffffd000`201e6c00 fffff802`b05cf6c6 : fffff802`b076a180 ffffe000`023e9040 ffffe000`01dee080 fffff802`b0568407 : nt!PspSystemThreadStartup+0x58
ffffd000`201e6c60 00000000`00000000 : ffffd000`201e7000 ffffd000`201e1000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16


STACK_COMMAND:  kb

FOLLOWUP_IP: 
OVSExt!OvsExtAttach+12c [c:\1.data\cloudbase\work\git\ovs\datapath-windows\ovsext\switch.c @ 86]
fffff800`0207041c 488b0ddd560100  mov     rcx,qword ptr [OVSExt!gOvsCtrlLock (fffff800`02085b00)]

FAULTING_SOURCE_LINE:  c:\1.data\cloudbase\work\git\ovs\datapath-windows\ovsext\switch.c

FAULTING_SOURCE_FILE:  c:\1.data\cloudbase\work\git\ovs\datapath-windows\ovsext\switch.c

FAULTING_SOURCE_LINE_NUMBER:  86

FAULTING_SOURCE_CODE:  
    82:         ASSERT(FALSE);
    83:         goto cleanup;
    84:     }
    85: 
>   86:     NdisAcquireSpinLock(gOvsCtrlLock);
    87:     if (gOvsSwitchContext) {
    88:         NdisReleaseSpinLock(gOvsCtrlLock);
    89:         OVS_LOG_TRACE("Exit: Failed to create OVS Switch, only one datapath is"
    90:                       "supported, %p.", gOvsSwitchContext);
    91:         goto cleanup;


SYMBOL_STACK_INDEX:  7

SYMBOL_NAME:  OVSExt!OvsExtAttach+12c

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: OVSExt

IMAGE_NAME:  OVSExt.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  54bec213

IMAGE_VERSION:  6.3.9600.17038

BUCKET_ID_FUNC_OFFSET:  12c

FAILURE_BUCKET_ID:  AV_OVSExt!OvsExtAttach

BUCKET_ID:  AV_OVSExt!OvsExtAttach

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:av_ovsext!ovsextattach

FAILURE_ID_HASH:  {b50384d2-3718-fe08-787d-439092a86a55}

Followup: MachineOwner
---------

1: kd> ?? gOvsExtDriverHandle
void * 0xffffe000`011c2d90
1: kd> ?? gOvsCtrlLock
struct _NDIS_SPIN_LOCK * 0x00000000`00000000
@svinturis svinturis self-assigned this Jan 22, 2015
@svinturis svinturis added the bug label Jan 22, 2015
@svinturis svinturis changed the title BSOD when loading an actived extension BSOD when loading an activated extension Jan 22, 2015
blp pushed a commit to openvswitch/ovs that referenced this issue Jan 30, 2015
@svinturis @blp
datapath-windows: Solved BSOD when loading an activated extension
448d667 
If the OVS extension was previously enabled and the driver unloaded,
when the driver is loaded again a BSOD is triggered.

This happens because the OVS extension registers its FilterXxx routines
to NDIS, by calling NdisFRegisterFilterDriver, before performing all
the necessary initialization. Because drivers that call
NdisFRegisterFilterDriver must be prepared for an immediate call to any
of their FilterXxx functions.

The BSOD is triggered because the FilterAttach routine, OvsExtAttach,
tries to acquire the control lock, when the lock is not yet initialized.
This happens because the FilterAttach is called before the driver
finishes initialization, in OvsInit().

The solution is to perform all necessary initialization before
registering OVS FilterXxx routines.

If device object creation fails, all allocated resources during init
phase are released by calling OvsCleanup and NdisFDeregisterFilterDriver
functions.

Signed-off-by: Sorin Vinturis <svinturis@cloudbasesolutions.com>
Reported-by: Sorin Vinturis <svinturis@cloudbasesolutions.com>
Reported-at: openvswitch/ovs-issues#67
Acked-by: Nithin Raju <nithin@vmware.com>
Signed-off-by: Ben Pfaff <blp@nicira.com>
@svinturis
Copy link
Author

This BSOD happened because the OVS extension registered its FilterXxx routines to NDIS, before performing all the necessary initialization. Solved this issue by performing OVS initialization before registering to NDIS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@svinturis svinturis

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@svinturis