python hung or stuck somtimes randomly on windows server 2008R2

[CaoHongfu]

BPO	35418
Nosy	@pfmoore, @vstinner, @tjguk, @zware, @eryksun, @zooba, @izbyshev

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-12-06.01:46:49.687>
created_at = <Date 2018-12-05.12:51:47.964>
labels = ['3.7', 'invalid', 'OS-windows', 'type-crash']
title = 'python hung or stuck somtimes randomly on windows server 2008R2'
updated_at = <Date 2018-12-06.02:53:24.058>
user = 'https://bugs.python.org/CaoHongfu'

bugs.python.org fields:

activity = <Date 2018-12-06.02:53:24.058>
actor = 'eryksun'
assignee = 'none'
closed = True
closed_date = <Date 2018-12-06.01:46:49.687>
closer = 'Cao Hongfu'
components = ['Windows']
creation = <Date 2018-12-05.12:51:47.964>
creator = 'Cao Hongfu'
dependencies = []
files = []
hgrepos = []
issue_num = 35418
keywords = []
message_count = 7.0
messages = ['331105', '331132', '331168', '331175', '331195', '331202', '331203']
nosy_count = 8.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'izbyshev', 'Cao Hongfu']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue35418'
versions = ['Python 3.7']

[CaoHongfu]

Recently, python frequently(but randomly) hung or stuck at initialization(when I click the python.exe or use python cmd prompt) on my server(running windows server 2008R2), But everything is fine on my windows 7 PC). I tried reinstall python, but not working(also tried 3.6).

I tried process-explorer and found that the normal python process allocated about 12MB memory but the stuck one only allocated 8MB memory.

here is the stack information for stuck python process(having 3 threads):
------------------thread1------------------
ntdll.dll!ZwWaitForSingleObject+0xa
ntdll.dll!RtlImageDirectoryEntryToData+0x118
ntdll.dll!RtlEnterCriticalSection+0xd1
ntdll.dll!EtwDeliverDataBlock+0x777
ntdll.dll!LdrLoadDll+0xed
!TlsGetValue+0x4756
!UuidCreate+0x1b00
!I_RpcBindingIsServerLocal+0x12899
!RegEnumKeyExW+0x13a
!RegEnumKeyExW+0xbe
!RpcBindingFree+0x320
!RpcAsyncRegisterInfo+0x10ff
!Ndr64AsyncClientCall+0x9da
!Ndr64AsyncClientCall+0xc9b
!NdrClientCall3+0xf5
!LsaOpenPolicy+0xb9
!LsaOpenPolicy+0x56
!LookupPrivilegeValueW+0x6f
!LookupPrivilegeValueA+0x84
!PyNamespace_New+0xd4
!PyCodec_LookupTextEncoding+0xb5
!PyObject_SetAttrId+0x21e
!PyMethodDef_RawFastCallDict+0x115
!PyObject_SetAttr+0x352
!PyEval_EvalFrameDefault+0x1182
!PyEval_EvalCodeWithName+0x1a0
!PyMethodDef_RawFastCallKeywords+0xc32
!PyEval_EvalFrameDefault+0x4b1
!PyMethodDef_RawFastCallKeywords+0xa77
!PyEval_EvalFrameDefault+0x913
!PyMethodDef_RawFastCallKeywords+0xa77
!PyEval_EvalFrameDefault+0x4b1
!PyMethodDef_RawFastCallKeywords+0xa77
!PyEval_EvalFrameDefault+0x4b1
!PyMethodDef_RawFastCallKeywords+0xa77
!PyEval_EvalFrameDefault+0x913
!PyMethodDef_RawFastCallKeywords+0xa77
!PyEval_EvalFrameDefault+0x4b1
!PyMethodDef_RawFastCallKeywords+0xa77
!PyEval_EvalFrameDefault+0x913
!PyFunction_FastCallDict+0xdd
!PyObject_CallMethod+0xef
!PyObject_CallMethod+0xa2
!PyObject_CallMethod+0x3c
!PyTime_MulDiv+0x47
!Py_InitializeMainInterpreter+0x95
!PyMainInterpreterConfig_Read+0x309
!PyMapping_SetItemString+0x306
!PyBytes_AsString+0x142
!Py_Main+0x52
!BaseThreadInitThunk+0xd
ntdll.dll!RtlUserThreadStart+0x1d
------------------thread2------------------
ntdll.dll!ZwWaitForSingleObject+0xa
ntdll.dll!RtlImageDirectoryEntryToData+0x118
ntdll.dll!RtlEnterCriticalSection+0xd1
!UuidCreate+0x1ae2
!NdrFullPointerQueryPointer+0x35d
!LsaLookupGetDomainInfo+0xb8
!RpcBindingFree+0x320
!RpcAsyncRegisterInfo+0x10ff
!Ndr64AsyncClientCall+0x9da
!Ndr64AsyncClientCall+0xc9b
!NdrClientCall3+0xf5
!LsaLookupOpenLocalPolicy+0x41
!LookupAccountNameLocalW+0xaf
!LookupAccountSidLocalW+0x25
!LookupAccountSidW+0x57
!MBCGlobal::get_proc_user_name+0x1f7
!MBCGlobal::init+0x240a
!HDirSnap::operator=+0xda
!LVPVTBase::to_file+0x46ef
ntdll.dll!RtlDeactivateActivationContextUnsafeFast+0x34e
ntdll.dll!EtwDeliverDataBlock+0xa44
ntdll.dll!LdrLoadDll+0xed
!TlsGetValue+0x4756
!PublicService+0x13ec
!BaseThreadInitThunk+0xd
ntdll.dll!RtlUserThreadStart+0x1d
------------------thread3------------------
ntdll.dll!ZwWaitForSingleObject+0xa
ntdll.dll!RtlImageDirectoryEntryToData+0x118
ntdll.dll!RtlEnterCriticalSection+0xd1
ntdll.dll!LdrQueryModuleServiceTags+0x13f
ntdll.dll!CsrIdentifyAlertableThread+0x9d
ntdll.dll!EtwSendNotification+0x16d
ntdll.dll!RtlQueryProcessDebugInformation+0x371
ntdll.dll!EtwDeliverDataBlock+0xf00
!BaseThreadInitThunk+0xd
ntdll.dll!RtlUserThreadStart+0x1d

here is the stack info for normal python process(have 2 threads)
------------------thread1------------------
ntdll.dll!ZwRequestWaitReplyPort+0xa
kernel32.dll!GetConsoleMode+0xf8
kernel32.dll!VerifyConsoleIoHandle+0x281
kernel32.dll!ReadConsoleW+0xbc
python37.dll!PyOS_Readline+0x4f4
python37.dll!PyOS_Readline+0x333
python37.dll!PyOS_Readline+0xfa
python37.dll!PyErr_NoMemory+0xc228
python37.dll!PyUnicode_AsUnicode+0x553
python37.dll!PyUnicode_AsUnicode+0x9c
python37.dll!PyParser_ParseFileObject+0x86
python37.dll!PyParser_ASTFromFileObject+0x82
python37.dll!PyRun_InteractiveOneObject+0x24a
python37.dll!PyRun_InteractiveLoopFlags+0xf6
python37.dll!PyRun_AnyFileExFlags+0x45
python37.dll!Py_UnixMain+0x50b
python37.dll!Py_UnixMain+0x5b3
python37.dll!PyErr_NoMemory+0x195a4
python37.dll!PyBytes_AsString+0x14f
python37.dll!Py_Main+0x52
python.exe+0x1258
kernel32.dll!BaseThreadInitThunk+0xd
ntdll.dll!RtlUserThreadStart+0x1d
------------------thread2------------------
ntdll.dll!NtWaitForMultipleObjects+0xa
ntdll.dll!RtlIsCriticalSectionLockedByThread+0xd4d
kernel32.dll!BaseThreadInitThunk+0xd
ntdll.dll!RtlUserThreadStart+0x1d

One of my friend say that this may be issues with
https://support.microsoft.com/en-us/help/2545627/a-multithreaded-application-might-crash-in-windows-7-or-in-windows-ser

Thx.

[izbyshev]

You might try to check the list of DLLs loaded into the stuck python process and find third-party ones (e.g., antivirus). If there are any, disable the third-party software and try again.

[vstinner]

You can try to use faulthandler.dump_traceback_later() with a file to get the traceback of the stuck threads.

------------------thread1------------------
...
!UuidCreate+0x1b00
...
!RegEnumKeyExW+0xbe
...
!LookupPrivilegeValueA+0x84
!PyNamespace_New+0xd4
!PyCodec_LookupTextEncoding+0xb5
!PyObject_SetAttrId+0x21e
...
!PyObject_CallMethod+0x3c
!PyTime_MulDiv+0x47
!Py_InitializeMainInterpreter+0x95
!PyMainInterpreterConfig_Read+0x309
!PyMapping_SetItemString+0x306
!PyBytes_AsString+0x142
!Py_Main+0x52
!BaseThreadInitThunk+0xd

This traceback doesn't make sense:

• Py_Main() doesn't call directly PyBytes_AsString().
• PyTime_MulDiv shouldn't call PyObject_CallMethod().
• I don't see how PyNamespace_New() can call LookupPrivilegeValueA()

[izbyshev]

How is it possible to use faulthandler if the interpreter hasn't even started yet?

[eryksun]

This traceback doesn't make sense

Enable the installer options for the debug binaries and symbol files. Ensure that this installs the *_d.[exe|dll] debug binaries and also the *.pdb symbol files for the release and debug builds.

Run the debug build via python_d.exe to check whether it aborts from a failed assertion. If it still hangs, in Process Explorer open the dialog for Options -> Configure Symbols and add the installation directory of Python 3.7 to the beginning of the "Symbols path". At least now your stack traces should have proper symbols instead of only the names of exported functions, for what that's worth. It's a poor substitute for a [mini]dump file and a debugger.

[CaoHongfu]

I tried windows resource manager and found that stuck python process does not have load these three DLL files(or stuck on loading these DLL files):
MozartBreathBolo.dll
MozartBreathNet.dll
MozartBreathProcess.dll.
These DLL files were created by our company's supervisory software，I contact the IT guy and disabled it. Now everything is fine.
Without the supervisory software, the normal python should allocate only 8MB memeory.
THX you guys, and sorry for wasting your time~

[eryksun]

I don't see how PyNamespace_New() can call LookupPrivilegeValueA()

For the record, in the 3.7.1 release build, PyNamespace_New + d4 is in enable_symlink (Modules/posixmodule.c), which gets called when the nt (aka posix) module gets initialized. It's the return address for the LookupPrivilegeValueA call:

0:000> u (python37!PyNamespace_New + d4 - 6) l2
python37!enable_symlink+0x42:
00007ffd`e4b9a356 
    ff153ccd1500    call    qword ptr 
    [python37!_imp_LookupPrivilegeValueA (00007ffd`e4cf7098)]
00007ffd`e4b9a35c 
    85c0            test    eax,eax

It's just that Cao didn't have the python37.pdb symbol file available in Process Explorer.