/qubes-issues

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

QWT BSOD: QUOTA_UNDERFLOW #2220

Open
omeg opened this Issue Aug 1, 2016 · 1 comment

Comments

Assignees

@omeg omeg

Labels
Projects
None yet
Milestone
  Far in the future
2 participants
@omeg @andrewdavidwong
@omeg
Member

omeg commented Aug 1, 2016

QWT 3.2.0+

Hard to debug as the object causing it is already freed. Seems to be a process terminating with not all of its memory returned (xeniface likely culprit).

XENIFACE|IoctlGnttabMapForeignPages: > Ref 38
XENIFACE|IoctlGnttabMapForeignPages: > Ref 39
XENIFACE|IoctlGnttabMapForeignPages: > Ref 40
XENIFACE|IoctlGnttabMapForeignPages: > Ref 41
XENIFACE|IoctlGnttabMapForeignPages: > Ref 42

*** Fatal System Error: 0x00000021
                       (0x0000000000000000,0x0000000000000000,0x0000000003000228,0x0000000002D7DFC8)

Break instruction exception - code 80000003 (first chance)

A fatal system error has occurred.
Debugger entered on first try; Bugcheck callbacks have not been invoked.

A fatal system error has occurred.

Connected to Windows 7 7601 x64 target at (Mon Aug  1 16:30:22.579 2016 (UTC + 2:00)), ptr64 TRUE
Loading Kernel Symbols
...............................................................
.......................................

Press ctrl-c (cdb, kd, ntsd) or ctrl-break (windbg) to abort symbol loads that take too long.
Run !sym noisy before .reload to track down problems loading symbols.

.........................
....
Loading User Symbols
..........................................................
Loading unloaded module list
.....
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 21, {0, 0, 3000228, 2d7dfc8}

Probably caused by : ntkrnlmp.exe ( nt! ?? ::FNODOBFM::`string'+32a56 )

Followup: MachineOwner
---------

nt!DbgBreakPointWithStatus:
fffff800`02679b50 cc              int     3
kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

QUOTA_UNDERFLOW (21)
This bugcheck occurs if a kernel component mishandles quota charges and
returns more quota than was previously charged to a particular quota block.
Arguments:
Arg1: 0000000000000000, The process (if any) that was initially charged.
Arg2: 0000000000000000, The quota type in question (paged pool, nonpaged pool, etc.)
Arg3: 0000000003000228, The initial charge amount to return.
Arg4: 0000000002d7dfc8, The remaining (unreturned) charge.

Debugging Details:
------------------


DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT

BUGCHECK_STR:  0x21

PROCESS_NAME:  lsass.exe

CURRENT_IRQL:  2

ANALYSIS_VERSION: 6.3.9600.17237 (debuggers(dbg).140716-0327) amd64fre

LAST_CONTROL_TRANSFER:  from fffff80002769b22 to fffff80002679b50

STACK_TEXT:  
fffff880`01e64198 fffff800`02769b22 : 00000000`00000000 fffffa80`02b94990 00000000`00000065 fffff800`026bd768 : nt!DbgBreakPointWithStatus
fffff880`01e641a0 fffff800`0276a90e : 00000000`00000003 00000000`00000000 fffff800`026bdfc0 00000000`00000021 : nt!KiBugCheckDebugBreak+0x12
fffff880`01e64200 fffff800`02628e16 : fffff8a0`00fa8f10 fffff800`02925f5e fffffa80`03094070 00000000`00010000 : nt!KeBugCheck2+0x71e
fffff880`01e648d0 fffff800`0297833d : fffff800`02807900 00000000`00000001 fffffa80`03094070 fffff800`02925005 : nt! ?? ::FNODOBFM::`string'+0x32a56
fffff880`01e64950 fffff800`0268bb1c : fffff800`02807900 00000000`00000000 fffffa80`0288b730 fffffa80`01925950 : nt!ObpFreeObject+0x13d
fffff880`01e649a0 fffff800`02978614 : fffffa80`0288b730 00000000`00000000 fffffa80`02b94990 00000000`00000000 : nt!ObfDereferenceObject+0xdc
fffff880`01e64a00 fffff800`02978bc4 : 00000000`00000740 fffffa80`0288b730 fffff8a0`05dcc580 00000000`00000740 : nt!ObpCloseHandleTableEntry+0xc4
fffff880`01e64a90 fffff800`02680f93 : fffffa80`02b94990 fffff880`01e64b60 00000000`0030a270 00000000`0031f1a0 : nt!ObpCloseHandle+0x94
fffff880`01e64ae0 00000000`77bd140a : 000007fe`fe3d3f44 00000000`0036afa0 00000000`0030a270 00000000`00000001 : nt!KiSystemServiceCopyEnd+0x13
00000000`014ff7f8 000007fe`fe3d3f44 : 00000000`0036afa0 00000000`0030a270 00000000`00000001 00000000`00000001 : ntdll!ZwClose+0xa
00000000`014ff800 000007fe`fe3d3ec4 : 00000000`0036b098 00000000`00000001 00000000`00000000 00000000`0030a270 : RPCRT4!LRPC_SASSOCIATION::~LRPC_SASSOCIATION+0x54
00000000`014ff840 000007fe`fe3ff074 : 00000000`0036afa0 00000000`00000001 00000000`0030a270 00000000`00000000 : RPCRT4!LRPC_SASSOCIATION::`scalar deleting destructor'+0x14
00000000`014ff870 000007fe`fe3d44f9 : 00000000`0036afa0 000007fe`fe3d4477 00000000`0036afa0 00000000`00000000 : RPCRT4!REFERENCED_OBJECT::FreeObject+0x14
00000000`014ff8a0 000007fe`fe3d43fb : 00000000`00000005 00000000`0030a270 00000000`00000000 00000000`00000000 : RPCRT4!LRPC_SASSOCIATION::MessageReceivedWithClosePending+0x65
00000000`014ff8e0 000007fe`fe3f2a35 : 00000000`003053c8 00000000`00000000 00000000`0030a370 00000000`00000000 : RPCRT4!LRPC_ADDRESS::ProcessIO+0x7b5
00000000`014ffa20 00000000`77b9b68b : 00000000`00305b00 00000000`00323c00 00000000`00323cb0 00000000`014ffc48 : RPCRT4!LrpcIoComplete+0xa5
00000000`014ffab0 00000000`77b9feff : 00000000`00000000 00000000`00000000 00000000`0000ffff 00000000`00000000 : ntdll!TppAlpcpExecuteCallback+0x26b
00000000`014ffb40 00000000`77a7652d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x3f8
00000000`014ffe40 00000000`77bac521 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`014ffe70 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

ObpFreeObject(fffff800`02807900) causes the bugcheck in a call to PspReturnQuota -- probably on process teardown when lsass released last handle to the process.

Listing all processes shows this:

PROCESS fffffa80031b6b30
    SessionId: 1  Cid: 08f8    Peb: 7fffffd4000  ParentCid: 06d8
    DirBase: 06da3000  ObjectTable: 00000000  HandleCount:   0.
    Image: qrexec-wrapper.exe

kd> dt _eprocess fffffa80031b6b30
ntdll!_EPROCESS
   +0x000 Pcb              : _KPROCESS
   +0x160 ProcessLock      : _EX_PUSH_LOCK
   +0x168 CreateTime       : _LARGE_INTEGER 0x01d1ec01`33aa3c20
   +0x170 ExitTime         : _LARGE_INTEGER 0x01d1ec01`33ea8140
   +0x178 RundownProtect   : _EX_RUNDOWN_REF
   +0x180 UniqueProcessId  : 0x00000000`000008f8 Void
   +0x188 ActiveProcessLinks : _LIST_ENTRY [ 0xfffffa80`03195528 - 0xfffffa80`030f2b68 ]
   +0x198 ProcessQuotaUsage : [2] 0xc0
   +0x1a8 ProcessQuotaPeak : [2] 0x2500
   +0x1b8 CommitCharge     : 0
   +0x1c0 QuotaBlock       : 0xfffff800`02807900 _EPROCESS_QUOTA_BLOCK
   +0x1c8 CpuQuotaBlock    : (null) 
   +0x1d0 PeakVirtualSize  : 0x341a000
   +0x1d8 VirtualSize      : 0x1ffa000

Handle table is empty, ExitTime and RundownProtect are set -- the process is being torn down.

kd> !process 0xfffffa80031b6b30 7
PROCESS fffffa80031b6b30
    SessionId: 1  Cid: 08f8    Peb: 7fffffd4000  ParentCid: 06d8
    DirBase: 06da3000  ObjectTable: 00000000  HandleCount:   0.
    Image: qrexec-wrapper.exe
    VadRoot 0000000000000000 Vads 0 Clone 0 Private 1. Modified 3. Locked 0.
    DeviceMap fffff8a000008bb0
    Token                             fffff8a0012ad6d0
    ElapsedTime                       00:00:14.523
    UserTime                          00:00:00.000
    KernelTime                        00:00:00.171
    QuotaPoolUsage[PagedPool]         1584
    QuotaPoolUsage[NonPagedPool]      192
    Working Set Sizes (now,min,max)  (5, 50, 345) (20KB, 200KB, 1380KB)
    PeakWorkingSetSize                1067
    VirtualSize                       31 Mb
    PeakVirtualSize                   52 Mb
    PageFaultCount                    1109
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      0

No active threads

All memory is freed (empty vad) but quota is not balanced.

@omeg omeg added bug C: windows tools P: major labels Aug 1, 2016

@omeg omeg self-assigned this Aug 1, 2016

@omeg

This comment has been minimized.

Show comment
Hide comment
@omeg

omeg Aug 2, 2016

Member

This time drvinst was terminating, but memory wasn't yet freed. Seems like this occurs in random process context, check around KeStackAttachProcess in xeniface/gnttab.

XENIFACE|IoctlGnttabMapForeignPages: > Ref 75
XENIFACE|IoctlGnttabMapForeignPages: > Ref 76
XENIFACE|IoctlGnttabMapForeignPages: > Ref 77
XENIFACE|IoctlGnttabMapForeignPages: > Ref 78
XENIFACE|IoctlGnttabMapForeignPages: > Ref 79

BugCheck 21, {0, 0, 3000228, 2dc6e0c}

Probably caused by : ntkrnlmp.exe ( nt! ?? ::FNODOBFM::`string'+32a56 )

Followup: MachineOwner
---------

QUOTA_UNDERFLOW (21)
This bugcheck occurs if a kernel component mishandles quota charges and
returns more quota than was previously charged to a particular quota block.
Arguments:
Arg1: 0000000000000000, The process (if any) that was initially charged.
Arg2: 0000000000000000, The quota type in question (paged pool, nonpaged pool, etc.)
Arg3: 0000000003000228, The initial charge amount to return.
Arg4: 0000000002dc6e0c, The remaining (unreturned) charge.

Debugging Details:
------------------


DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT

BUGCHECK_STR:  0x21

PROCESS_NAME:  drvinst.exe

CURRENT_IRQL:  2

ANALYSIS_VERSION: 6.3.9600.17237 (debuggers(dbg).140716-0327) amd64fre

LAST_CONTROL_TRANSFER:  from fffff80002786b22 to fffff80002696b50

STACK_TEXT:  
fffff880`04282fa8 fffff800`02786b22 : 00000000`00000000 fffffa80`03139060 00000000`00000065 fffff800`026da768 : nt!DbgBreakPointWithStatus
fffff880`04282fb0 fffff800`0278790e : 00000000`00000003 00000000`00000000 fffff800`026dafc0 00000000`00000021 : nt!KiBugCheckDebugBreak+0x12
fffff880`04283010 fffff800`02645e16 : fffff8a0`012b2720 fffff800`02942f5e 00000000`00000000 fffff800`029421bf : nt!KeBugCheck2+0x71e
fffff880`042836e0 fffff800`0299533d : fffff800`02824900 fffffa80`01925950 fffffa80`030e8070 fffff800`02942005 : nt! ?? ::FNODOBFM::`string'+0x32a56
fffff880`04283760 fffff800`026a8b1c : fffff800`02824900 00000000`00000000 fffff8a0`012ac680 fffffa80`01925950 : nt!ObpFreeObject+0x13d
fffff880`042837b0 fffff800`02995614 : fffff8a0`012ac680 00000000`00000000 fffffa80`03139060 00000000`00000000 : nt!ObfDereferenceObject+0xdc
fffff880`04283810 fffff800`02957734 : 00000000`00000018 fffff8a0`012ac680 fffff8a0`012b0060 00000000`00000018 : nt!ObpCloseHandleTableEntry+0xc4
fffff880`042838a0 fffff800`02957634 : 00000000`00000004 00000000`00000000 fffffa80`0312bb30 fffff800`02944811 : nt!ObpCloseHandleProcedure+0x30
fffff880`042838e0 fffff800`02957cb2 : fffff8a0`012afa01 00000000`00000001 fffffa80`0312bb30 00000000`00000001 : nt!ExSweepHandleTable+0x74
fffff880`04283920 fffff800`029741c2 : fffff8a0`012afa20 00000000`00000000 00000000`00000000 00000000`00000000 : nt!ObKillProcess+0x62
fffff880`04283960 fffff800`0295787c : 00000000`00000000 00000000`00000001 000007ff`fffde000 00000000`00000000 : nt!PspExitThread+0x522
fffff880`04283a60 fffff800`0269df93 : fffffa80`0312bb30 000007fe`00000000 fffffa80`03139060 000007fe`ff1226c8 : nt!NtTerminateProcess+0x138
fffff880`04283ae0 00000000`779c15da : 00000000`7799418b 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`0019fa18 00000000`7799418b : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000080 : ntdll!NtTerminateProcess+0xa
00000000`0019fa20 00000000`ffe462f8 : 00000000`00000000 00000000`001ef5f0 00000000`0019faa8 00000000`00000000 : ntdll!RtlExitUserProcess+0x9b
00000000`0019fa50 00000000`ffe5468d : 00000000`00000000 00000000`ffe41808 01d1ec3f`061a0000 00000000`00000200 : DrvInst!wmain+0xd64
00000000`0019fbf0 00000000`7786652d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : DrvInst!CreateUserSecuredEvent+0x4b5
00000000`0019fc30 00000000`7799c521 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0019fc60 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d


kd> !object fffff800`02824900
fffff80002824900: Not a valid object (ObjectType invalid)
kd> !process
PROCESS fffffa800312bb30
    SessionId: 0  Cid: 08c0    Peb: 7fffffdc000  ParentCid: 0338
    DirBase: 0ad54000  ObjectTable: fffff8a0012ac680  HandleCount: 206.
    Image: drvinst.exe
    VadRoot fffffa8003118d60 Vads 80 Clone 0 Private 550. Modified 1305. Locked 0.
    DeviceMap fffff8a000008bb0
    Token                             fffff8a0012afa20
    ElapsedTime                       00:00:50.247
    UserTime                          00:00:00.171
    KernelTime                        00:00:00.218
    QuotaPoolUsage[PagedPool]         82552
    QuotaPoolUsage[NonPagedPool]      9608
    Working Set Sizes (now,min,max)  (1901, 50, 345) (7604KB, 200KB, 1380KB)
    PeakWorkingSetSize                2165
    VirtualSize                       36 Mb
    PeakVirtualSize                   58 Mb
    PageFaultCount                    6400
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      899

        THREAD fffffa8003139060  Cid 08c0.08c4  Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 0
Member

omeg commented Aug 2, 2016

This time drvinst was terminating, but memory wasn't yet freed. Seems like this occurs in random process context, check around KeStackAttachProcess in xeniface/gnttab.

XENIFACE|IoctlGnttabMapForeignPages: > Ref 75
XENIFACE|IoctlGnttabMapForeignPages: > Ref 76
XENIFACE|IoctlGnttabMapForeignPages: > Ref 77
XENIFACE|IoctlGnttabMapForeignPages: > Ref 78
XENIFACE|IoctlGnttabMapForeignPages: > Ref 79

BugCheck 21, {0, 0, 3000228, 2dc6e0c}

Probably caused by : ntkrnlmp.exe ( nt! ?? ::FNODOBFM::`string'+32a56 )

Followup: MachineOwner
---------

QUOTA_UNDERFLOW (21)
This bugcheck occurs if a kernel component mishandles quota charges and
returns more quota than was previously charged to a particular quota block.
Arguments:
Arg1: 0000000000000000, The process (if any) that was initially charged.
Arg2: 0000000000000000, The quota type in question (paged pool, nonpaged pool, etc.)
Arg3: 0000000003000228, The initial charge amount to return.
Arg4: 0000000002dc6e0c, The remaining (unreturned) charge.

Debugging Details:
------------------


DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT

BUGCHECK_STR:  0x21

PROCESS_NAME:  drvinst.exe

CURRENT_IRQL:  2

ANALYSIS_VERSION: 6.3.9600.17237 (debuggers(dbg).140716-0327) amd64fre

LAST_CONTROL_TRANSFER:  from fffff80002786b22 to fffff80002696b50

STACK_TEXT:  
fffff880`04282fa8 fffff800`02786b22 : 00000000`00000000 fffffa80`03139060 00000000`00000065 fffff800`026da768 : nt!DbgBreakPointWithStatus
fffff880`04282fb0 fffff800`0278790e : 00000000`00000003 00000000`00000000 fffff800`026dafc0 00000000`00000021 : nt!KiBugCheckDebugBreak+0x12
fffff880`04283010 fffff800`02645e16 : fffff8a0`012b2720 fffff800`02942f5e 00000000`00000000 fffff800`029421bf : nt!KeBugCheck2+0x71e
fffff880`042836e0 fffff800`0299533d : fffff800`02824900 fffffa80`01925950 fffffa80`030e8070 fffff800`02942005 : nt! ?? ::FNODOBFM::`string'+0x32a56
fffff880`04283760 fffff800`026a8b1c : fffff800`02824900 00000000`00000000 fffff8a0`012ac680 fffffa80`01925950 : nt!ObpFreeObject+0x13d
fffff880`042837b0 fffff800`02995614 : fffff8a0`012ac680 00000000`00000000 fffffa80`03139060 00000000`00000000 : nt!ObfDereferenceObject+0xdc
fffff880`04283810 fffff800`02957734 : 00000000`00000018 fffff8a0`012ac680 fffff8a0`012b0060 00000000`00000018 : nt!ObpCloseHandleTableEntry+0xc4
fffff880`042838a0 fffff800`02957634 : 00000000`00000004 00000000`00000000 fffffa80`0312bb30 fffff800`02944811 : nt!ObpCloseHandleProcedure+0x30
fffff880`042838e0 fffff800`02957cb2 : fffff8a0`012afa01 00000000`00000001 fffffa80`0312bb30 00000000`00000001 : nt!ExSweepHandleTable+0x74
fffff880`04283920 fffff800`029741c2 : fffff8a0`012afa20 00000000`00000000 00000000`00000000 00000000`00000000 : nt!ObKillProcess+0x62
fffff880`04283960 fffff800`0295787c : 00000000`00000000 00000000`00000001 000007ff`fffde000 00000000`00000000 : nt!PspExitThread+0x522
fffff880`04283a60 fffff800`0269df93 : fffffa80`0312bb30 000007fe`00000000 fffffa80`03139060 000007fe`ff1226c8 : nt!NtTerminateProcess+0x138
fffff880`04283ae0 00000000`779c15da : 00000000`7799418b 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`0019fa18 00000000`7799418b : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000080 : ntdll!NtTerminateProcess+0xa
00000000`0019fa20 00000000`ffe462f8 : 00000000`00000000 00000000`001ef5f0 00000000`0019faa8 00000000`00000000 : ntdll!RtlExitUserProcess+0x9b
00000000`0019fa50 00000000`ffe5468d : 00000000`00000000 00000000`ffe41808 01d1ec3f`061a0000 00000000`00000200 : DrvInst!wmain+0xd64
00000000`0019fbf0 00000000`7786652d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : DrvInst!CreateUserSecuredEvent+0x4b5
00000000`0019fc30 00000000`7799c521 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0019fc60 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d


kd> !object fffff800`02824900
fffff80002824900: Not a valid object (ObjectType invalid)
kd> !process
PROCESS fffffa800312bb30
    SessionId: 0  Cid: 08c0    Peb: 7fffffdc000  ParentCid: 0338
    DirBase: 0ad54000  ObjectTable: fffff8a0012ac680  HandleCount: 206.
    Image: drvinst.exe
    VadRoot fffffa8003118d60 Vads 80 Clone 0 Private 550. Modified 1305. Locked 0.
    DeviceMap fffff8a000008bb0
    Token                             fffff8a0012afa20
    ElapsedTime                       00:00:50.247
    UserTime                          00:00:00.171
    KernelTime                        00:00:00.218
    QuotaPoolUsage[PagedPool]         82552
    QuotaPoolUsage[NonPagedPool]      9608
    Working Set Sizes (now,min,max)  (1901, 50, 345) (7604KB, 200KB, 1380KB)
    PeakWorkingSetSize                2165
    VirtualSize                       36 Mb
    PeakVirtualSize                   58 Mb
    PageFaultCount                    6400
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      899

        THREAD fffffa8003139060  Cid 08c0.08c4  Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 0

@andrewdavidwong andrewdavidwong added help wanted and removed help wanted labels Jan 14, 2017

@andrewdavidwong andrewdavidwong added this to the Far in the future milestone Dec 16, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment