Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: make.bat hangs #36492

Open
alexbrainman opened this issue Jan 10, 2020 · 3 comments
Open

runtime: make.bat hangs #36492

alexbrainman opened this issue Jan 10, 2020 · 3 comments

Comments

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Jan 10, 2020

I have Windows 7 computer - windows/amd64.

I am building Go from source using go1.4 as bootstrap.

I am using commit 56d6b87

I am running make.bat command.

What did you expect to see?

I expected make.bat command finish successfully.

What did you see instead?

make.bat never finishes. It hangs, for example, like this:

c:\Users\alexb\dev\go\src>make
Building Go cmd/dist using c:\users\alexb\dev\\go1.4
Building Go toolchain1 using c:\users\alexb\dev\\go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.

I used process explorer https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer to examine process tree, and that is what I see:

image

I also used WinDbg to attach to go_bootstrap.exe (pid 9788) and print stacks of all its threads. And that is what I see:

0:019> !uniqstack
Processing 20 threads, please wait

.  0  Id: 263c.23d8 Suspend: 1 Teb: 000007ff`fffde000 Unfrozen
      Start: go_bootstrap+0x648e0 (00000000`004648e0) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`0022fae8 000007fe`fcc710ac ntdll!ZwWaitForSingleObject+0xa
00000000`0022faf0 00000000`0046494e KERNELBASE!WaitForSingleObjectEx+0x9c
00000000`0022fb90 00000000`00b37be0 go_bootstrap+0x6494e
00000000`0022fb98 00000000`004270d3 go_bootstrap+0x737be0
00000000`0022fba0 00000000`00000000 go_bootstrap+0x270d3

.  1  Id: 263c.314c Suspend: 1 Teb: 000007ff`fffdc000 Unfrozen
      Start: tmmon64+0x8adac (00000000`7478adac) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`0302fcb8 000007fe`fcc71430 ntdll!ZwWaitForMultipleObjects+0xa
00000000`0302fcc0 00000000`76df06c0 KERNELBASE!GetCurrentProcess+0x40
00000000`0302fdc0 00000000`747aeed4 kernel32!WaitForMultipleObjects+0xb0
00000000`0302fe50 00000000`7478ad07 tmmon64+0xaeed4
00000000`0302ff00 00000000`7478aeae tmmon64+0x8ad07
00000000`0302ff30 00000000`76df59cd tmmon64+0x8aeae
00000000`0302ff60 00000000`76f2a561 kernel32!BaseThreadInitThunk+0xd
00000000`0302ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x21

.  2  Id: 263c.2d5c Suspend: 1 Teb: 000007ff`fffda000 Unfrozen
      Start: ntdll!RtlDestroyHandleTable+0x270 (00000000`76f1f6f0) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`0322fcc8 00000000`76f1ed15 ntdll!ZwWaitForWorkViaWorkerFactory+0xa
00000000`0322fcd0 00000000`76df59cd ntdll!RtlValidateHeap+0x155
00000000`0322ff60 00000000`76f2a561 kernel32!BaseThreadInitThunk+0xd
00000000`0322ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x21

.  3  Id: 263c.e74 Suspend: 1 Teb: 000007ff`fffd6000 Unfrozen
      Start: ntdll!TpIsTimerSet+0x8b0 (00000000`76f1a280) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`0342fcb8 00000000`76f1a3c7 ntdll!ZwWaitForMultipleObjects+0xa
00000000`0342fcc0 00000000`76df59cd ntdll!TpIsTimerSet+0x9f7
00000000`0342ff60 00000000`76f2a561 kernel32!BaseThreadInitThunk+0xd
00000000`0342ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x21

.  4  Id: 263c.196c Suspend: 1 Teb: 000007ff`fffd4000 Unfrozen
      Start: go_bootstrap+0x64d00 (00000000`00464d00) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`28e5eb68 00000000`76f48f58 ntdll!ZwWaitForSingleObject+0xa
00000000`28e5eb70 00000000`76f48e54 ntdll!RtlDeNormalizeProcessParams+0x5a8
00000000`28e5ec20 000007fe`fa3b7f0b ntdll!RtlDeNormalizeProcessParams+0x4a4
00000000`28e5ec50 000007fe`fa3b8504 TmUmEvt64+0x17f0b
00000000`28e5eeb0 000007fe`fa3b8c96 TmUmEvt64+0x18504
00000000`28e5ef10 000007fe`fa4565ca TmUmEvt64+0x18c96
00000000`28e5efa0 000007fe`fa455f8e TmUmEvt64+0xb65ca
00000000`28e5f000 000007fe`fa410686 TmUmEvt64+0xb5f8e
00000000`28e5f150 000007fe`fa439730 TmUmEvt64+0x70686
00000000`28e5f260 00000000`7472f146 TmUmEvt64+0x99730
00000000`28e5f4a0 00000000`747e2d7d tmmon64+0x2f146
00000000`28e5f580 00000000`747e29f4 tmmon64+0xe2d7d
00000000`28e5f640 00000000`74733748 tmmon64+0xe29f4
00000000`28e5f6b0 000007fe`fcc77c3f tmmon64+0x33748
00000000`28e5f780 00000000`0046494e KERNELBASE!ResumeThread+0xf
00000000`28e5f7b0 ffffffff`ffffffff go_bootstrap+0x6494e
00000000`28e5f7b8 00000000`00000001 0xffffffff`ffffffff
00000000`28e5f7c0 ffffffff`ffffffff 0x1
00000000`28e5f7c8 00000000`28e5f928 0xffffffff`ffffffff
00000000`28e5f7d0 00000000`00000000 0x28e5f928

.  5  Id: 263c.3204 Suspend: 1 Teb: 000007ff`fffae000 Unfrozen
      Start: go_bootstrap+0x64d00 (00000000`00464d00) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`2905fb08 000007fe`fcc710ac ntdll!ZwWaitForSingleObject+0xa
00000000`2905fb10 00000000`0046494e KERNELBASE!WaitForSingleObjectEx+0x9c
00000000`2905fbb0 000000c0`0002c980 go_bootstrap+0x6494e
00000000`2905fbb8 00000000`00000000 0xc0`0002c980

.  6  Id: 263c.32f0 Suspend: 1 Teb: 000007ff`fffac000 Unfrozen
      Start: go_bootstrap+0x64d00 (00000000`00464d00) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`2925fb08 000007fe`fcc710ac ntdll!ZwWaitForSingleObject+0xa
00000000`2925fb10 00000000`0046494e KERNELBASE!WaitForSingleObjectEx+0x9c
00000000`2925fbb0 000000c0`0002cd00 go_bootstrap+0x6494e
00000000`2925fbb8 00000000`00000164 0xc0`0002cd00
00000000`2925fbc0 7fffffff`00000000 0x164
00000000`2925fbc8 00000000`00000160 0x7fffffff`00000000
00000000`2925fbd0 000000c0`004c6300 0x160
00000000`2925fbd8 00000000`2925fcc0 0xc0`004c6300
00000000`2925fbe0 00000000`0043daa2 0x2925fcc0
00000000`2925fbe8 00000000`00b36bf8 go_bootstrap+0x3daa2
00000000`2925fbf0 00000000`00000000 go_bootstrap+0x736bf8

.  7  Id: 263c.29d8 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen
      Start: go_bootstrap+0x64d00 (00000000`00464d00) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`2945fb08 000007fe`fcc710ac ntdll!ZwWaitForSingleObject+0xa
00000000`2945fb10 00000000`0046494e KERNELBASE!WaitForSingleObjectEx+0x9c
00000000`2945fbb0 000000c0`00080280 go_bootstrap+0x6494e
00000000`2945fbb8 00000000`00000000 0xc0`00080280

.  8  Id: 263c.104c Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen
      Start: go_bootstrap+0x64d00 (00000000`00464d00) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`29a8fb08 000007fe`fcc710ac ntdll!ZwWaitForSingleObject+0xa
00000000`29a8fb10 00000000`0046494e KERNELBASE!WaitForSingleObjectEx+0x9c
00000000`29a8fbb0 000000c0`00206280 go_bootstrap+0x6494e
00000000`29a8fbb8 00000000`00000000 0xc0`00206280

.  9  Id: 263c.31b4 Suspend: 1 Teb: 000007ff`fffa6000 Unfrozen
      Start: go_bootstrap+0x64d00 (00000000`00464d00) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`29ccfb08 000007fe`fcc710ac ntdll!ZwWaitForSingleObject+0xa
00000000`29ccfb10 00000000`0046494e KERNELBASE!WaitForSingleObjectEx+0x9c
00000000`29ccfbb0 000000c0`00207080 go_bootstrap+0x6494e
00000000`29ccfbb8 00000000`00000000 0xc0`00207080

. 10  Id: 263c.2b14 Suspend: 1 Teb: 000007ff`fffa4000 Unfrozen
      Start: go_bootstrap+0x64d00 (00000000`00464d00) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`29eceb68 00000000`76f48f58 ntdll!ZwWaitForSingleObject+0xa
00000000`29eceb70 00000000`76f48e54 ntdll!RtlDeNormalizeProcessParams+0x5a8
00000000`29ecec20 000007fe`fa3b7f0b ntdll!RtlDeNormalizeProcessParams+0x4a4
00000000`29ecec50 000007fe`fa3b8504 TmUmEvt64+0x17f0b
00000000`29eceeb0 000007fe`fa3b8c96 TmUmEvt64+0x18504
00000000`29ecef10 000007fe`fa4565ca TmUmEvt64+0x18c96
00000000`29ecefa0 000007fe`fa455f8e TmUmEvt64+0xb65ca
00000000`29ecf000 000007fe`fa410686 TmUmEvt64+0xb5f8e
00000000`29ecf150 000007fe`fa439730 TmUmEvt64+0x70686
00000000`29ecf260 00000000`7472f146 TmUmEvt64+0x99730
00000000`29ecf4a0 00000000`747e2d7d tmmon64+0x2f146
00000000`29ecf580 00000000`747e29f4 tmmon64+0xe2d7d
00000000`29ecf640 00000000`74733748 tmmon64+0xe29f4
00000000`29ecf6b0 000007fe`fcc77c3f tmmon64+0x33748
00000000`29ecf780 00000000`0046494e KERNELBASE!ResumeThread+0xf
00000000`29ecf7b0 ffffffff`ffffffff go_bootstrap+0x6494e
00000000`29ecf7b8 00000000`00000000 0xffffffff`ffffffff

. 11  Id: 263c.30b4 Suspend: 1 Teb: 000007ff`fffa2000 Unfrozen
      Start: go_bootstrap+0x64d00 (00000000`00464d00) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`2a10fb08 000007fe`fcc710ac ntdll!ZwWaitForSingleObject+0xa
00000000`2a10fb10 00000000`0046494e KERNELBASE!WaitForSingleObjectEx+0x9c
00000000`2a10fbb0 000000c0`00195400 go_bootstrap+0x6494e
00000000`2a10fbb8 00000000`00000009 0xc0`00195400
00000000`2a10fbc0 00000000`00000000 0x9

. 12  Id: 263c.30fc Suspend: 1 Teb: 000007ff`fffa0000 Unfrozen
      Start: go_bootstrap+0x64d00 (00000000`00464d00) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`2a30f248 00000000`76f48f58 ntdll!ZwWaitForSingleObject+0xa
00000000`2a30f250 00000000`76f48e54 ntdll!RtlDeNormalizeProcessParams+0x5a8
00000000`2a30f300 000007fe`fa3b7f0b ntdll!RtlDeNormalizeProcessParams+0x4a4
00000000`2a30f330 000007fe`fa3b8504 TmUmEvt64+0x17f0b
00000000`2a30f590 000007fe`fa3b8c96 TmUmEvt64+0x18504
00000000`2a30f5f0 000007fe`fa4565ca TmUmEvt64+0x18c96
00000000`2a30f680 000007fe`fa455f8e TmUmEvt64+0xb65ca
00000000`2a30f6e0 000007fe`fa410686 TmUmEvt64+0xb5f8e
00000000`2a30f830 000007fe`fa4363b0 TmUmEvt64+0x70686
00000000`2a30f940 00000000`7472f146 TmUmEvt64+0x963b0
00000000`2a30fb80 00000000`747e2d7d tmmon64+0x2f146
00000000`2a30fc60 00000000`747e29f4 tmmon64+0xe2d7d
00000000`2a30fd20 00000000`74732869 tmmon64+0xe29f4
00000000`2a30fd90 00000000`0046494e tmmon64+0x32869
00000000`2a30fe60 000000c0`00306100 go_bootstrap+0x6494e
00000000`2a30fe68 000000c0`0016b840 0xc0`00306100
00000000`2a30fe70 00000000`2a30fe60 0xc0`0016b840
00000000`2a30fe78 00000000`00000000 0x2a30fe60

. 13  Id: 263c.13a0 Suspend: 2 Teb: 000007ff`fff9e000 Unfrozen
      Start: go_bootstrap+0x64d00 (00000000`00464d00) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`2a50ec50 000007fe`fa3b8504 TmUmEvt64+0x17fa0
00000000`2a50eeb0 000007fe`fa3b8c96 TmUmEvt64+0x18504
00000000`2a50ef10 000007fe`fa4565ca TmUmEvt64+0x18c96
00000000`2a50efa0 000007fe`fa455f8e TmUmEvt64+0xb65ca
00000000`2a50f000 000007fe`fa410686 TmUmEvt64+0xb5f8e
00000000`2a50f150 000007fe`fa439730 TmUmEvt64+0x70686
00000000`2a50f260 00000000`7472f146 TmUmEvt64+0x99730
00000000`2a50f4a0 00000000`747e2d7d tmmon64+0x2f146
00000000`2a50f580 00000000`747e29f4 tmmon64+0xe2d7d
00000000`2a50f640 00000000`74733748 tmmon64+0xe29f4
00000000`2a50f6b0 000007fe`fcc77c3f tmmon64+0x33748
00000000`2a50f780 00000000`0046494e KERNELBASE!ResumeThread+0xf
00000000`2a50f7b0 ffffffff`ffffffff go_bootstrap+0x6494e
00000000`2a50f7b8 00000000`00000000 0xffffffff`ffffffff

. 15  Id: 263c.1c34 Suspend: 1 Teb: 000007ff`fff9a000 Unfrozen
      Start: go_bootstrap+0x64d00 (00000000`00464d00) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`2a90faf8 000007fe`fcc710ac ntdll!ZwWaitForSingleObject+0xa
00000000`2a90fb00 00000000`0046494e KERNELBASE!WaitForSingleObjectEx+0x9c
00000000`2a90fba0 000000c0`003b9400 go_bootstrap+0x6494e
00000000`2a90fba8 ffffffff`fff85ee0 0xc0`003b9400
00000000`2a90fbb0 00000000`00000000 0xffffffff`fff85ee0

. 16  Id: 263c.2530 Suspend: 1 Teb: 000007ff`fff98000 Unfrozen
      Start: go_bootstrap+0x64d00 (00000000`00464d00) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`2ab0fb08 000007fe`fcc710ac ntdll!ZwWaitForSingleObject+0xa
00000000`2ab0fb10 00000000`0046494e KERNELBASE!WaitForSingleObjectEx+0x9c
00000000`2ab0fbb0 000000c0`003b9780 go_bootstrap+0x6494e
00000000`2ab0fbb8 00000000`00000009 0xc0`003b9780
00000000`2ab0fbc0 00000000`00000000 0x9

. 17  Id: 263c.3114 Suspend: 1 Teb: 000007ff`fff96000 Unfrozen
      Start: go_bootstrap+0x64d00 (00000000`00464d00) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`2ad0fb08 000007fe`fcc710ac ntdll!ZwWaitForSingleObject+0xa
00000000`2ad0fb10 00000000`0046494e KERNELBASE!WaitForSingleObjectEx+0x9c
00000000`2ad0fbb0 000000c0`00508d00 go_bootstrap+0x6494e
00000000`2ad0fbb8 00000000`00000000 0xc0`00508d00

. 18  Id: 263c.3344 Suspend: 3 Teb: 000007ff`fff94000 Unfrozen
      Start: go_bootstrap+0x64d00 (00000000`00464d00) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`2af0fb08 000007fe`fcc71430 ntdll!ZwWaitForMultipleObjects+0xa
00000000`2af0fb10 00000000`76df06c0 KERNELBASE!GetCurrentProcess+0x40
00000000`2af0fc10 00000000`0046494e kernel32!WaitForMultipleObjects+0xb0
00000000`2af0fca0 000007ff`fff94000 go_bootstrap+0x6494e
00000000`2af0fca8 00000000`00000000 0x7ff`fff94000

. 19  Id: 263c.3128 Suspend: 1 Teb: 000007ff`fff92000 Unfrozen
      Start: ntdll!DbgUiRemoteBreakin (00000000`76ff2dd0) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`2b10ff28 00000000`76ff2e08 ntdll!DbgBreakPoint
00000000`2b10ff30 00000000`76df59cd ntdll!DbgUiRemoteBreakin+0x38
00000000`2b10ff60 00000000`76f2a561 kernel32!BaseThreadInitThunk+0xd
00000000`2b10ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x21

Total threads: 20
Duplicate callstacks: 1 (windbg thread #s follow):
14

I was able to use Delve to examine this bug once (see #35775 (comment)), but not anymore. Delve just fails to attach now.

I can reproduce this pretty reliably on this particular computer - make.bat never completes. Sometimes it hangs in go_bootstrap.exe and sometimes in compile.exe. Sometimes there are more than single hung compile.exe.

I can make problem go away, if I change source code to have runtime.preemptMSupported set to false.

/cc @aclements

Alex

@aclements

This comment has been minimized.

Copy link
Member

@aclements aclements commented Jan 10, 2020

Thanks for the detailed report.

Most of the threads look uninteresting, except, I think, these two:

.  4  Id: 263c.196c Suspend: 1 Teb: 000007ff`fffd4000 Unfrozen
      Start: go_bootstrap+0x64d00 (00000000`00464d00) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`28e5eb68 00000000`76f48f58 ntdll!ZwWaitForSingleObject+0xa
00000000`28e5eb70 00000000`76f48e54 ntdll!RtlDeNormalizeProcessParams+0x5a8
00000000`28e5ec20 000007fe`fa3b7f0b ntdll!RtlDeNormalizeProcessParams+0x4a4
00000000`28e5ec50 000007fe`fa3b8504 TmUmEvt64+0x17f0b
00000000`28e5eeb0 000007fe`fa3b8c96 TmUmEvt64+0x18504
00000000`28e5ef10 000007fe`fa4565ca TmUmEvt64+0x18c96
00000000`28e5efa0 000007fe`fa455f8e TmUmEvt64+0xb65ca
00000000`28e5f000 000007fe`fa410686 TmUmEvt64+0xb5f8e
00000000`28e5f150 000007fe`fa439730 TmUmEvt64+0x70686
00000000`28e5f260 00000000`7472f146 TmUmEvt64+0x99730
00000000`28e5f4a0 00000000`747e2d7d tmmon64+0x2f146
00000000`28e5f580 00000000`747e29f4 tmmon64+0xe2d7d
00000000`28e5f640 00000000`74733748 tmmon64+0xe29f4
00000000`28e5f6b0 000007fe`fcc77c3f tmmon64+0x33748
00000000`28e5f780 00000000`0046494e KERNELBASE!ResumeThread+0xf
00000000`28e5f7b0 ffffffff`ffffffff go_bootstrap+0x6494e
00000000`28e5f7b8 00000000`00000001 0xffffffff`ffffffff
00000000`28e5f7c0 ffffffff`ffffffff 0x1
00000000`28e5f7c8 00000000`28e5f928 0xffffffff`ffffffff
00000000`28e5f7d0 00000000`00000000 0x28e5f928

. 10  Id: 263c.2b14 Suspend: 1 Teb: 000007ff`fffa4000 Unfrozen
      Start: go_bootstrap+0x64d00 (00000000`00464d00) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`29eceb68 00000000`76f48f58 ntdll!ZwWaitForSingleObject+0xa
00000000`29eceb70 00000000`76f48e54 ntdll!RtlDeNormalizeProcessParams+0x5a8
00000000`29ecec20 000007fe`fa3b7f0b ntdll!RtlDeNormalizeProcessParams+0x4a4
00000000`29ecec50 000007fe`fa3b8504 TmUmEvt64+0x17f0b
00000000`29eceeb0 000007fe`fa3b8c96 TmUmEvt64+0x18504
00000000`29ecef10 000007fe`fa4565ca TmUmEvt64+0x18c96
00000000`29ecefa0 000007fe`fa455f8e TmUmEvt64+0xb65ca
00000000`29ecf000 000007fe`fa410686 TmUmEvt64+0xb5f8e
00000000`29ecf150 000007fe`fa439730 TmUmEvt64+0x70686
00000000`29ecf260 00000000`7472f146 TmUmEvt64+0x99730
00000000`29ecf4a0 00000000`747e2d7d tmmon64+0x2f146
00000000`29ecf580 00000000`747e29f4 tmmon64+0xe2d7d
00000000`29ecf640 00000000`74733748 tmmon64+0xe29f4
00000000`29ecf6b0 000007fe`fcc77c3f tmmon64+0x33748
00000000`29ecf780 00000000`0046494e KERNELBASE!ResumeThread+0xf
00000000`29ecf7b0 ffffffff`ffffffff go_bootstrap+0x6494e
00000000`29ecf7b8 00000000`00000000 0xffffffff`ffffffff

. 13  Id: 263c.13a0 Suspend: 2 Teb: 000007ff`fff9e000 Unfrozen
      Start: go_bootstrap+0x64d00 (00000000`00464d00) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`2a50ec50 000007fe`fa3b8504 TmUmEvt64+0x17fa0
00000000`2a50eeb0 000007fe`fa3b8c96 TmUmEvt64+0x18504
00000000`2a50ef10 000007fe`fa4565ca TmUmEvt64+0x18c96
00000000`2a50efa0 000007fe`fa455f8e TmUmEvt64+0xb65ca
00000000`2a50f000 000007fe`fa410686 TmUmEvt64+0xb5f8e
00000000`2a50f150 000007fe`fa439730 TmUmEvt64+0x70686
00000000`2a50f260 00000000`7472f146 TmUmEvt64+0x99730
00000000`2a50f4a0 00000000`747e2d7d tmmon64+0x2f146
00000000`2a50f580 00000000`747e29f4 tmmon64+0xe2d7d
00000000`2a50f640 00000000`74733748 tmmon64+0xe29f4
00000000`2a50f6b0 000007fe`fcc77c3f tmmon64+0x33748
00000000`2a50f780 00000000`0046494e KERNELBASE!ResumeThread+0xf
00000000`2a50f7b0 ffffffff`ffffffff go_bootstrap+0x6494e
00000000`2a50f7b8 00000000`00000000 0xffffffff`ffffffff

These are all stuck in ResumeThread, which I didn't know threads could get stuck in.

Thread 13 is also interesting because the "suspend count" is 2, suggesting that some other thread has suspended it and is failing to resume it. This may also be why threads 4 and 10 are stopped in obviously blocking operations, while thread 13 is stopped at a seemingly random point.

Do you know what "tmmon64" and "TmUmEvt64" are?

Maybe we just need to hold the suspendLock for longer (though I don't have a theory for why this would be). What happens if you move the unlock(&suspendLock) in preemptM to the very bottom of the function?

@alexbrainman

This comment has been minimized.

Copy link
Member Author

@alexbrainman alexbrainman commented Jan 10, 2020

Do you know what "tmmon64" and "TmUmEvt64" are?

This is not my computer, so I cannot properly poke at it. But I suspect this computer has some standard anrivirus software installed. And these kinds of software often would install their code to intercept real Win32 API calls.

Maybe we just need to hold the suspendLock for longer (though I don't have a theory for why this would be). What happens if you move the unlock(&suspendLock) in preemptM to the very bottom of the function?

I won't see her on the weekend. But I will try it Monday or Tuesday. Also please show diff of your suggested change, because I don't trust myself with reading your mind.

Thank you.

Alex

@alexbrainman

This comment has been minimized.

Copy link
Member Author

@alexbrainman alexbrainman commented Jan 13, 2020

Maybe we just need to hold the suspendLock for longer (though I don't have a theory for why this would be). What happens if you move the unlock(&suspendLock) in preemptM to the very bottom of the function?

I changed the code, like this

diff --git a/src/runtime/os_windows.go b/src/runtime/os_windows.go
index 91e147fca9..9166aeb323 100644
--- a/src/runtime/os_windows.go
+++ b/src/runtime/os_windows.go
@@ -1201,8 +1201,6 @@ func preemptM(mp *m) {
        // GetThreadContext actually blocks until it's suspended.
        stdcall2(_GetThreadContext, thread, uintptr(unsafe.Pointer(c)))

-       unlock(&suspendLock)
-
        // Does it want a preemption and is it safe to preempt?
        gp := gFromTLS(mp)
        if wantAsyncPreempt(gp) && isAsyncSafePoint(gp, c.ip(), c.sp(), c.lr())
{
@@ -1231,6 +1229,8 @@ func preemptM(mp *m) {

        stdcall1(_ResumeThread, thread)
        stdcall1(_CloseHandle, thread)
+
+       unlock(&suspendLock)
 }

 // osPreemptExtEnter is called before entering external code that may

And I managed to run make.bat once to successful completion. But then it hung as before, when I run make.bat second time. This time, it is compile.exe that hung. Here is the stack trace from windbg

0:009> !uniqstack
Processing 10 threads, please wait

.  0  Id: 2c08.3140 Suspend: 1 Teb: 000007ff`fffde000 Unfrozen
      Start: compile+0x67ef0 (00000000`00467ef0) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`0022fd98 000007fe`fcd610ac ntdll!ZwWaitForSingleObject+0xa
00000000`0022fda0 00000000`00467f5e KERNELBASE!WaitForSingleObjectEx+0x9c
00000000`0022fe40 00000000`0160d320 compile+0x67f5e
00000000`0022fe48 00000000`00000001 compile+0x120d320
00000000`0022fe50 00000000`00000000 0x1

.  1  Id: 2c08.243c Suspend: 1 Teb: 000007ff`fffdc000 Unfrozen
      Start: tmmon64+0x8adac (00000000`7480adac) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`03cdfcb8 000007fe`fcd61430 ntdll!ZwWaitForMultipleObjects+0xa
00000000`03cdfcc0 00000000`76d806c0 KERNELBASE!GetCurrentProcess+0x40
00000000`03cdfdc0 00000000`7482eed4 kernel32!WaitForMultipleObjects+0xb0
00000000`03cdfe50 00000000`7480ad07 tmmon64+0xaeed4
00000000`03cdff00 00000000`7480aeae tmmon64+0x8ad07
00000000`03cdff30 00000000`76d859cd tmmon64+0x8aeae
00000000`03cdff60 00000000`76fba561 kernel32!BaseThreadInitThunk+0xd
00000000`03cdff90 00000000`00000000 ntdll!RtlUserThreadStart+0x21

.  2  Id: 2c08.20ec Suspend: 1 Teb: 000007ff`fffda000 Unfrozen
      Start: ntdll!RtlDestroyHandleTable+0x270 (00000000`76faf6f0) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`03fefcc8 00000000`76faed15 ntdll!ZwWaitForWorkViaWorkerFactory+0xa
00000000`03fefcd0 00000000`76d859cd ntdll!RtlValidateHeap+0x155
00000000`03feff60 00000000`76fba561 kernel32!BaseThreadInitThunk+0xd
00000000`03feff90 00000000`00000000 ntdll!RtlUserThreadStart+0x21

.  3  Id: 2c08.2ecc Suspend: 1 Teb: 000007ff`fffd8000 Unfrozen
      Start: ntdll!TpIsTimerSet+0x8b0 (00000000`76faa280) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`041efcb8 00000000`76faa3c7 ntdll!ZwWaitForMultipleObjects+0xa
00000000`041efcc0 00000000`76d859cd ntdll!TpIsTimerSet+0x9f7
00000000`041eff60 00000000`76fba561 kernel32!BaseThreadInitThunk+0xd
00000000`041eff90 00000000`00000000 ntdll!RtlUserThreadStart+0x21

.  4  Id: 2c08.2cf4 Suspend: 1 Teb: 000007ff`fffd4000 Unfrozen
      Start: compile+0x68330 (00000000`00468330) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`299def98 00000000`76fd8f58 ntdll!ZwWaitForSingleObject+0xa
00000000`299defa0 00000000`76fd8e54 ntdll!RtlDeNormalizeProcessParams+0x5a8
00000000`299df050 00000000`76fde0b5 ntdll!RtlDeNormalizeProcessParams+0x4a4
00000000`299df080 00000000`76fdddd8 ntdll!RtlAllocateHeap+0x455
00000000`299df260 000007fe`fcd61635 ntdll!RtlAllocateHeap+0x178
00000000`299df370 000007fe`fa36bf7d KERNELBASE!LocalAlloc+0x75
00000000`299df3e0 000007fe`fa36c187 TmUmEvt64!TmmonDestoryAddonObject+0x63d
00000000`299df430 000007fe`fa36bbf1 TmUmEvt64!TmmonDestoryAddonObject+0x847
00000000`299df4b0 00000000`74862cd3 TmUmEvt64!TmmonDestoryAddonObject+0x2b1
00000000`299df580 00000000`748629f4 tmmon64+0xe2cd3
00000000`299df640 00000000`747b3de4 tmmon64+0xe29f4
00000000`299df6b0 00000000`76d72a4a tmmon64+0x33de4
00000000`299df780 00000000`00467f5e kernel32!GetThreadContext+0xa
00000000`299df7b0 ffffffff`ffffffff compile+0x67f5e
00000000`299df7b8 00000000`00000000 0xffffffff`ffffffff

.  5  Id: 2c08.ea4 Suspend: 1 Teb: 000007ff`fffae000 Unfrozen
      Start: compile+0x68330 (00000000`00468330) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`29bdfb08 000007fe`fcd610ac ntdll!ZwWaitForSingleObject+0xa
00000000`29bdfb10 00000000`00467f5e KERNELBASE!WaitForSingleObjectEx+0x9c
00000000`29bdfbb0 000000c0`0002c980 compile+0x67f5e
00000000`29bdfbb8 00000000`00000000 0xc0`0002c980

.  6  Id: 2c08.558 Suspend: 2 Teb: 000007ff`fffac000 Unfrozen
      Start: compile+0x68330 (00000000`00468330) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`29ddf530 00000000`76ffcf71 ntdll!RtlUnlockHeap+0x6a8
00000000`29ddf5c0 00000000`76fdddd8 ntdll!TpAlpcRegisterCompletionList+0xaf91
00000000`29ddf7a0 000007fe`fcd61635 ntdll!RtlAllocateHeap+0x178
00000000`29ddf8b0 000007fe`fa36bf7d KERNELBASE!LocalAlloc+0x75
00000000`29ddf920 000007fe`fa36c187 TmUmEvt64!TmmonDestoryAddonObject+0x63d
00000000`29ddf970 000007fe`fa36bbf1 TmUmEvt64!TmmonDestoryAddonObject+0x847
00000000`29ddf9f0 00000000`74862cd3 TmUmEvt64!TmmonDestoryAddonObject+0x2b1
00000000`29ddfac0 00000000`748629f4 tmmon64+0xe2cd3
00000000`29ddfb80 00000000`747b0aab tmmon64+0xe29f4
00000000`29ddfbf0 00000000`00467f5e tmmon64+0x30aab
00000000`29ddfcc0 000000c0`005c4000 compile+0x67f5e
00000000`29ddfcc8 00000000`00002000 0xc0`005c4000
00000000`29ddfcd0 00000000`00001000 0x2000
00000000`29ddfcd8 00000000`00000004 0x1000
00000000`29ddfce0 00000000`00432b66 0x4
00000000`29ddfce8 00000000`76d914cf compile+0x32b66
00000000`29ddfcf0 00000000`0000016c kernel32!GetTickCount+0x1f
00000000`29ddfcf8 00000000`ffffffff 0x16c
00000000`29ddfd00 00000000`00000000 0xffffffff

.  7  Id: 2c08.2f14 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen
      Start: compile+0x68330 (00000000`00468330) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`2a01fb08 000007fe`fcd610ac ntdll!ZwWaitForSingleObject+0xa
00000000`2a01fb10 00000000`00467f5e KERNELBASE!WaitForSingleObjectEx+0x9c
00000000`2a01fbb0 000000c0`00080280 compile+0x67f5e
00000000`2a01fbb8 00000000`01e7b5c0 0xc0`00080280
00000000`2a01fbc0 00000000`00000000 0x1e7b5c0

.  8  Id: 2c08.2f64 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen
      Start: ntdll!RtlDestroyHandleTable+0x270 (00000000`76faf6f0) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`2a68f3a8 00000000`76fd8f58 ntdll!ZwWaitForSingleObject+0xa
00000000`2a68f3b0 00000000`76fd8e54 ntdll!RtlDeNormalizeProcessParams+0x5a8
00000000`2a68f460 00000000`76fde0b5 ntdll!RtlDeNormalizeProcessParams+0x4a4
00000000`2a68f490 00000000`76fdddd8 ntdll!RtlAllocateHeap+0x455
00000000`2a68f670 00000000`76fba762 ntdll!RtlAllocateHeap+0x178
00000000`2a68f780 000007fe`fcd62898 ntdll!RtlUserThreadStart+0x222
00000000`2a68f7c0 000007fe`fa38ae5e KERNELBASE!FlsSetValue+0x168
00000000`2a68f7f0 000007fe`fa38a152 TmUmEvt64!GetInterface+0x1e50e
00000000`2a68f820 000007fe`fa389c2d TmUmEvt64!GetInterface+0x1d802
00000000`2a68f850 000007fe`fa385c19 TmUmEvt64!GetInterface+0x1d2dd
00000000`2a68f880 000007fe`fa385fc9 TmUmEvt64!GetInterface+0x192c9
00000000`2a68f8b0 000007fe`fa3861f9 TmUmEvt64!GetInterface+0x19679
00000000`2a68f8e0 00000000`76fba719 TmUmEvt64!GetInterface+0x198a9
00000000`2a68f940 00000000`76fba46f ntdll!RtlUserThreadStart+0x1d9
00000000`2a68fa40 00000000`76fba36e ntdll!LdrInitializeThunk+0x10f
00000000`2a68fab0 00000000`00000000 ntdll!LdrInitializeThunk+0xe

.  9  Id: 2c08.2b84 Suspend: 1 Teb: 000007ff`fffa6000 Unfrozen
      Start: ntdll!DbgUiRemoteBreakin (00000000`77082dd0) 
      Priority: 0  Priority class: 32  Affinity: f
Child-SP          RetAddr           Call Site
00000000`2a88ff28 00000000`77082e08 ntdll!DbgBreakPoint
00000000`2a88ff30 00000000`76d859cd ntdll!DbgUiRemoteBreakin+0x38
00000000`2a88ff60 00000000`76fba561 kernel32!BaseThreadInitThunk+0xd
00000000`2a88ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x21

Total threads: 10

Alex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.