Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: TestLibraryCtrlHandler fails occasionally with register ABI enabled #45638

Open
mknyszek opened this issue Apr 19, 2021 · 52 comments
Open

runtime: TestLibraryCtrlHandler fails occasionally with register ABI enabled #45638

mknyszek opened this issue Apr 19, 2021 · 52 comments

Comments

@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented Apr 19, 2021

Found a failure on the windows-amd64-regabi bot:

--- FAIL: TestLibraryCtrlHandler (6.32s)
    signal_windows_test.go:205: Program exited with error: exit status 1
        FAILURE: No signal received
FAIL
FAIL	runtime	52.447s

Seems like a flake, but we should look into it.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 19, 2021

I was able to reproduce with -count=1000.

@aclements
Copy link
Member

@aclements aclements commented Apr 19, 2021

I did rewrite the ctrlHandler runtime code recently in preparation for regabi, though not specifically in a regabi way. If you can repro, it would be worth doing a check around that commit to see if that was the cause.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 20, 2021

Looking at the build dashboard logs, it doesn't seem like this is regabi related. There's one other failure:

2021-04-16T03:16:11-cf2396c/windows-amd64-2012:
--- FAIL: TestLibraryCtrlHandler (6.17s)
    signal_windows_test.go:205: Program exited with error: exit status 1
        FAILURE: No signal received
FAIL
FAIL    runtime

The fact that there are no other failures suggests that the CL @aclements mentioned is probably the culprit (https://golang.org/cl/309632).

By the way, it's harder to reproduce than I thought.

Background on the test:

  1. It's a C program that links up to a Go DLL, but it doesn't actually call anything in the DLL.
  2. It waits to get a control signal and fails if it doesn't get one in time.
  3. It expects the control signal to be produced externally, so the actual Go test (in signal_windows_test.go) sends the signal.

Here's my working theory: LoadLibrary (in the C code) returns before the runtime is fully initialized, signaling the Go test to send its signal. Note that rt0_amd64_lib (which is used to initialize a c-shared library) spawns a thread to finish initialization asynchronously. Basically, the signalling in the test was always racy with initialization and setup of the control handler.As of https://golang.org/cl/309632, the control handler is now set up slightly later in the initialization process and it takes longer to get there too, because we now call compileCallback which is non-trivial.

As a result, the test now flakes a little bit. I will try to confirm this by adding further delay in initialization to see if the test failure reproduces more easily.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 20, 2021

I don't think my working theory is far off from the problem, but I don't fully understand anymore why the race pertains to this test. This test just makes sure that the C process receives the signal on its handler. It doesn't care about the Go handler at all.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 21, 2021

Print debugging isn't working because the initialization code doesn't seem to have access to stdout (or the equivalent) on Windows.

I just tried switching the timeout in the test's C code from 5 seconds to INFINITE to see if it hangs and maybe I can get it into a debuggable state.

Unfortunately when I did that I came back to my terminal filled with binary garbage followed by "fail" reported by the parent process. I suspect this problem goes a little deeper than I realize...

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 21, 2021

For a taste, my terminal looks something like this:

        $)euz���
        JZ<���������Xn��r�����������Xn��w|�����
        _r���������Xnz������������XnAM�����������Xn�����������Xn��������������Xn^ez�1o7'�"NW|�8C���@C�]���������-Xn"(H���������/Xnjno����������/Xnuyz����������/Xn
*3I^hy�����������@1Xn�*3IQhy�������� 5Xn *16T\]�������� 5Xn *16���������5Xn7Fadgw���������������5Xndgw������������6Xn8��������@7Xn\e�����������9Xn$)FMk�����������`:Xn(*��������`:Xn=Hknq����������������`:Xn������������`:Xn�����������`:Xn!��Unq�����/9J��������`:Xn����
        9J��������`:Xnnq����������`:X]Tf��������`:Xn	59��������`:Xnn�JT��������`:Xnos����������`:Xn������������� >Xn8PS����������� >Xn{���������������?Xn*0;=��������?Xn@ELQe�������������?XnLQe�������������?Xnhq{���������������?Xn04Qp���#&O��������?Xn04�����������#&+/2EO��������?Xn��#&��������?Xn������������`CXnr�����IL����r���������`CXn"R]r��������`CXn9MOU��������`CXn�������������`CXn,CFILSm�������������������@LXnfk���!L��������@LXn,�����������@LXn{�����������@LXn����!L��������@LXn%,���m���������PXn��>A��m���������PXn�����������������PXn7HJS���������PXn^oqz���������PXnkm�>As���������PXn���������������PXn���������������PXn9<@WZ]`h��������������������WXn^boty����������WXnboty���������WXnkoty��������@ZXn-=�����������^Xn�������������_Xn����/F���������_Xn%���������gXn4=Klny�����������gXnlny����������� iXnLOWa���u���������mXndlpx}��NVw���KT����'S�������������mXnpx��%*IV^cw
--- FAIL: TestLibraryCtrlHandler (15.03s)
FAIL
FAIL	runtime	306.632s
FAIL
Error running run: exit status 1

except the garbage extends far beyond what my scrollback buffer can carry. I can reproduce this with the same frequency as the original failure.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 21, 2021

Something is very broken here. This kind of reads like something is trying to print (maybe the printf("ready\n")?) and it's memory corruption leading to a buffer overrun. There aren't very many prints at all in all these code paths. In fact, that may be the only one.

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Apr 24, 2021

I tried to reproduce this issue on my computer, but I could not.

Building programs (Go and especially C) is slow on Windows, so I modified this test to build everything once and then run executable in a loop.

diff --git a/src/runtime/signal_windows_test.go b/src/runtime/signal_windows_test.go
index d0f61fb6c2..ecacafd9d2 100644
--- a/src/runtime/signal_windows_test.go
+++ b/src/runtime/signal_windows_test.go
@@ -170,8 +170,14 @@ func TestLibraryCtrlHandler(t *testing.T) {
                t.Fatalf("failed to build c exe: %s\n%s", err, out)
        }

+       for i := 0; i < 10000; i++ {
+               testLibraryCtrlHandler(t, exe)
+       }
+}
+
+func testLibraryCtrlHandler(t *testing.T, exe string) {
        // run test program
-       cmd = exec.Command(exe)
+       cmd := exec.Command(exe)
        var stderr bytes.Buffer
        cmd.Stderr = &stderr
        outPipe, err := cmd.StdoutPipe()

And my test succeeds

c:\Users\Alex\dev\go\src>go test -count=1 -run=CtrlH runtime
ok      runtime 222.704s

c:\Users\Alex\dev\go\src>

Alex

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 26, 2021

Change https://golang.org/cl/313350 mentions this issue: runtime: add missing import "C" in TestLibraryCtrlHandler

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 26, 2021

Change https://golang.org/cl/313351 mentions this issue: runtime: replace --buildmode with -buildmode in tests

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

@alexbrainman FWIW my -count to the test binary was at about 1000 (and even then it only failed roughly every other time, so maybe once in 2000, 2500 runs?), so it's not easy to reproduce but it is possible. (EDIT: I realize now you run the test 10000 times...). Out of curiosity, what happens if you change the WaitForSingleObject from 5 seconds to INFINITE on your machine?

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

Ah, nevermind, I did not look closely enough at your code. My bad!

Huh, what version of Windows are you running? Maybe it's a bug specific to the windows-amd64-2016 builders?

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

3 consecutive runs with @alexbrainman's patch from #45638 (comment) came up clean. That's... interesting.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

Wow, I spoke too soon. It literally JUST failed. Once out 30000 runs, roughly.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

Interestingly, this is how it failed:

=== RUN   TestLibraryCtrlHandler
    signal_windows_test.go:210: Program exited with error: exit status 1
        runtime: signal received on thread not created by Go.
        �o�<f�N9꙾�&����vl��3U0�{\K���X�س���64oMˡ1��!
                                                    R��n��)�Џ�jG'��T��-�0�i�
                                                                            �!Ρ6�)��,p??�G�j�V���k6˩���G
                                                                                                        �
                                                                                                         �6�runtime: signal received on thread not created by Go.
        �o�<f�N9꙾�&����vl��3U0�{\K���X�س���64oMˡ1��!
                                                    R��n��)�Џ�jG'��T��-�0�i�
                                                                            �!Ρ6�)��,p??�G�j�V���k6˩���G
                                                                                                        �
                                                                                                         �6�h�h�runtime: signal received on thread not created by Go.
        �o�<f�N9꙾�&����vl��3U0�{\K���X�س���64oMˡ1��!
                                                    R��n��)�Џ�jG'��T��-�0�i�
                                                                            �!Ρ6�)��,p??�G�j�V���k6˩���G
                                                                                                        �
                                                                                                         �6�h�h�d@�?@DDD �"""C:\Windows\system32\runtime: signal received on thread not created by Go.
        �o�<f�N9꙾�&����vl��3U0�{\K���X�س���64oMˡ1��!
                                                    R��n��)�Џ�jG'��T��-�0�i�
                                                                            �!Ρ6�)��,p??�G�j�V���k6˩���G
                                                                                                        �
                                                                                                         �6�h�h�d@�?@DDD �"""C:\Windows\system32\FAILURE: No signal received
--- FAIL: TestLibraryCtrlHandler (68.32s)
FAIL
FAIL	runtime	68.350s
FAIL
Error running run: exit status 1

Again with the terminal garbage, but it's very interesting to see C:\Windows\system32\ appear amongst that garbage.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

I have a suspicion that callbackasm is not a valid mechanism for implementing ctrlHandler. Control signals are delivered as actual signals on Windows. Something tells me it's not safe to go through the whole cgo callback path in these contexts...

The runtime: signal received on thread not created by Go. message has me a little worried in this direction, anyway.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

Er, actually I don't see why this would be a problem. The Windows docs claim (https://docs.microsoft.com/en-us/windows/console/handlerroutine) that a new thread is created to handle these signals, so actually callbackasm seems like exactly the right mechanism. Now I'm getting the impression that something else is going wrong, raises an exception, and the exception lands on a non-Go thread?

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

Here's a theory about how the above messages are getting produced: something causes us to get into an exception that lands on a thread that has GS set (where we put TLS on Windows). That exception tries to print the badsignalmsg but as it turns out badsignallen got corrupted or something. It's otherwise wrong. So then we get this buffer overrun, garbage data is printed out after the message until we reach a point where we hit invalid/unmapped/uncommitted memory, triggering another exception on the same thread. And so on a few more times.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

If that theory is true, then it raises three questions:

  1. What causes the signal to trigger in the first place?
  2. Is that cause related to why badsignallen is written over?
  3. How does the signal land on a "bad" thread? (Maybe this one is easy -- TLS is initialized but there's no G? Would be nice to see what that pointer actually looks like (if it's a heap address, it IS a Go thread). The specific check here is that the slot where we put a TLS pointer is set, but then where we would usually put the G is zero.)
@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

Good news: in the current failure mode I can actually print out things when the DLL is getting initialized.

gopherbot pushed a commit that referenced this issue Apr 27, 2021
CL 211139 added TestLibraryCtrlHandler. But the CL left out import "C"
line in the test file that is supposed to be build with Cgo.

While debugging issue #45638, I discovered that the DLL built during
TestLibraryCtrlHandler does not have Dummy function. Adding import "C"
makes Dummy function appear in DLL function list.

TestLibraryCtrlHandler does not actually calls Dummy function. So I
don't see how this change affects issue #45638, but still let's make
this code correct.

Updates #45638

Change-Id: Ibab8fed29ef2ae446d0815842cf0bd040a5fb943
Reviewed-on: https://go-review.googlesource.com/c/go/+/313350
Trust: Alex Brainman <alex.brainman@gmail.com>
Run-TryBot: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
gopherbot pushed a commit that referenced this issue Apr 27, 2021
While debugging issue #45638, I discovered that some tests were using
--buildmode command line parameter instead of -buildmode.

The --buildmode parameter is handled properly by the flag package - it
is read as -buildmode. But we should correct code anyway.

Updates #45638

Change-Id: I75cf95c7d11dcdf4aeccf568b2dea77bd8942352
Reviewed-on: https://go-review.googlesource.com/c/go/+/313351
Trust: Alex Brainman <alex.brainman@gmail.com>
Run-TryBot: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Apr 28, 2021

I just run my test from #45638 (comment) on current tip. It failed once:

c:\Users\Alex\dev\go\src>go test -count=1 -run=CtrlH runtime
--- FAIL: TestLibraryCtrlHandler (179.74s)
    signal_windows_test.go:210: Program exited with error: exit status 1
        FAILURE: No signal received
FAIL
FAIL    runtime 185.257s
FAIL

and succeeded 3 times.

Out of curiosity, what happens if you change the WaitForSingleObject from 5 seconds to INFINITE on your machine?

The test still succeeds. But that gives me an idea. If you modify the test to use INFINITE in WaitForSingleObject, your C program will block on that WaitForSingleObject Windows API call indefinetly. And you should be able to attach to the process with some debugger and try to analyze program state.

Unfortunately I am not familiar with debuggers, so I cannot help you, but surly we can find help with debugging. You can use Delve debugger on Windows - I am not sure how good it is with Cgo program. Or you can use WindDbg - it is used by real Windows developers (not like me) - you should be able to do anything with it, except it does not understand Go program symbols. Or you can use gdb.

Huh, what version of Windows are you running?

C:\Users\Alex>ver

Microsoft Windows [Version 10.0.17763.1697]

C:\Users\Alex>

Maybe it's a bug specific to the windows-amd64-2016 builders?

It certainly can be. Do not forget that we are testing C program with Go DLL (built with Cgo) here. And you need gcc to build these.

Windows does not come with gcc installed. so whatever gcc you have installed on the builder is what you use. So you would have to account for the fact that this bug could be in gcc tools or runtime. And this builder, I suspect, have very old version of gcc. For example, see #35006 and #43195.

I use

c:\Users\Alex\dev\go\src>gcc --version
gcc (x86_64-win32-seh-rev0, Built by MinGW-W64 project) 8.1.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


c:\Users\Alex\dev\go\src>

here. I suspect this gcc is different from what is installed on builder. And the fact that test failed once here (see above), probably means it is not gcc bug.

Again with the terminal garbage, but it's very interesting to see C:\Windows\system32\ appear amongst that garbage.

This was, probably, generated by gcc linker, not Go linker. But otherwise, I have not comment.

Here's a theory about how the above messages are getting produced: something causes us to get into an exception that lands on a thread that has GS set (where we put TLS on Windows). That exception tries to print the badsignalmsg but as it turns out badsignallen got corrupted or something. It's otherwise wrong. So then we get this buffer overrun, garbage data is printed out after the message until we reach a point where we hit invalid/unmapped/uncommitted memory, triggering another exception on the same thread. And so on a few more times.

Unfortunately am not familiar enough with Go runtime and gcc runtime to speculate. But that sounds possible. There a 2 control handlers installed by SetConsoleCtrlHandler - one by gcc runtime and another by Go runtime. And there are 2 exception handlers installed - one by gcc runtime and another by Go runtime. So it can get messy.

Alex

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Apr 28, 2021

The test failed again just as I submitted previous message. Unfortunately this time the output is huge, so I am providing the end of it.

                                                                                                                                                                                                                                                                                    �#� �   �runtime �    ��]n        . Go cmd/compile devel go1.17-92d1afe989 Wed Apr 28 02:39:09 2021 +0000 runtime �runtime.callbackasm ��]n    ��]n    ���   � �<unspecified> "unsafe.Pointer �uintptr�1_n    %runtime._type 0��V�     �size  �    �ptrdata�    �hash ���   �tflag ���   �align ���   �fieldAlign ���   �kind ���   �equal ���   �gcdata  p�   �str (��   �ptrToThis ,��    &runtime._type �   �uint32 �
         �      �runtime.tflag�~      �uint8 ��      �func(unsafe.Pointer, unsafe.Pointer) bool���      ��   ��   �\�   &func(unsafe.Pointer, unsafe.Pointer) bool ��  �bool ��� v      !*bool K�   �J      !*uint8 ��  �@e      �runtime.nameOff ����}      �runtime.typeOff ��� �      %runtime.arraytype H�        �typ  ��   �elem 0!�   �slice 8!�   �len @�     &runtime.arraytype ��  !*runtime._type ��  ����     %runtime.chantype @�        �typ  ��   �elem 0!�   �dir 8�     &runtime.chantype >�  %runtime.functype 8�        �typ  ��   �inCount 0��   �outCount 2��    &runtime.functype ��  �uint16 �       ��      %runtime.maptype X�        �typ  ��   �key 0!�   �elem 8!�   �bucket @!�   �hasher H��   �keysize P��   �elemsize Q��   �bucketsize R��   �flags T��    &runtime.maptype ��  �func(unsafe.Pointer, uintptr) uintptr�        ��   ��   ���   &func(unsafe.Pointer, uintptr) uintptr ��  !*uintptr �    �e      %runtime.ptrtype 8� �      �typ  ��   �elem 0!�    &runtime.ptrtype 3�  %runtime.slicetype 8�        �typ  ��   �elem 0!�    &runtime.slicetype {�  %runtime.structtype P�        �typ  ��   �pkgPath 0L�   �fields 8^�    &runtime.structtype ��  %runtime.name� >�     �bytes  p�    &runtime.name &�  #[]runtime.structfield ��        ��  �array  �6   �l   �cap �    %runtime.structfield ��        �name  L�   �typ!�   �offsetAnon ��     &runtime.structfield ��  %runtime.interfacetype P� ��     �typ  ��   �pkgpath 0L�   �mhdr 8l    &runtime.interfacetype     #[]runtime.imethod ���m      �  �array  �E   �le   �cap     %runtime.imethod���      �name  ��   �ityp ���    &runtime.imethod �  %runtime.itab  �@.�     �inter     �_type!�   �hash ���   �_ �   �fun     &runtime.itab �  !*runtime.interfacetype Q  ��V      �[4]uint8 ��  ��@�      ��   � �[1]uintptr �  �@�      ��   � �int��y      $string �� �      �str  p�   �le    !*internal/cpu.option �     ��M      %internal/cpu.option  ��)�     �Name    �Feature �\�   �Specified �K�   �Enable �K�   �Required �K�    &internal/cpu.option            !*[15]internal/cpu.option �     �        �[15]internal/cpu.option �         ��� �      ��   � �runtime/internal/sys.ArchFamilyType�        �uint64�`�      !*internal/abi.RegArgs �
          �@M      %internal/abi.RegArgs ������     �Ints  �
           �Floats H�
           �Ptrs ����   �ReturnIsPtr ��t�    &internal/abi.RegArgs `
          �[9]uintptr �   H� �      ��           �[15]uint64 )
          x���      ��   � �[9]unsafe.Pointer �   H�`�      ��           �internal/abi.IntArgRegBitmap ��  �� �      ��   � &internal/abi.IntArgRegBitmap A�  �runtime.boundsErrorCode  |      �runtime.lockRank���      �int8 ����z      �runtime.statDep        �runtime.metricKind�        �runtime.gcMarkWorkerMode�`}      �runtime.gcMode�        �runtime.gcTriggerKind�        �runtime.gcDrainFlags�        �runtime.sweepClass �
                �runtime.mSpanState�}      �runtime.spanClass `�      �runtime.spanAllocType         �runtime.pallocSum� �      �runtime.bucketType �`|      �runtime.profIndex�        �runtime.profBufReadMode �        �runtime.waitReason  �      �runtime.selectDir �        �runtime.semaProfileFlags �        �uint��      �runtime.funcID �|      �runtime.funcFlag �|      �runtime.abiPartKind��{      !*runtime.stringStruct ��  ��_      %runtime.stringStruct ����      �str  �    �l    &runtime.stringStruct ��  !*runtime.itab   ���      !*unsafe.Pointer �   ��e      !*runtime.wbBuf w�  ���      %runtime.wbBuf � �     �next  �    �end�    �buf ���    &runtime.wbBuf :�  �[512]uintptr �   � �`�      ��   �� !*[2]uintptr ��  �        �[2]uintptr �   ��        ��   � �int32 ���`z      !*runtime.m V�  ��W      %runtime.m� y�     �g0  e�   �morebu)�   �divmod @��   �procid H)
           �gsignal Pe�   �goSigStack X�    �sigmask X�    �tls !   �mstartfn ��<!   �curg ��e�   �caughtsig ��<�   �p ��k�   �nextp ��k�   �oldp ��k�   �id ��Y�   �mallocing ����   �throwing ����   �preemptoff ��   �locks ����   �dying ����   �profilehz ����   �spinning ��K�   �blocked ��K�   �newSigstack ��K�   �printlock ����   �incgo ��K�   �freeWait ����   �fastrand ��H!   �needextram ��K�   �traceback ����   �ncgocall ��)
           �ncgo ����   �cgoCallersUse ����   �cgoCallers ��h!   �doesPark ��K�   �park ���!   �alllink ����   �schedlink ����   �lockedg ��<�   �createstack ���"   �lockedExt ����   �lockedInt ����   �nextwaitm ����   �waitunlockf ��f"   �waitlock ���    �waittraceev ����   �waittraceskip ��   �startingtrace ��K�   �syscalltick ����   �freelink ����   �mFixup ���"   �libcall ���#   �libcallpc ���    �libcallsp ���    �libcallg ��<�   �syscall ���#   �vdsoSP ���    �vdsoPC ���    �preemptGen ����   �signalPending ����   �dlogPerM ���#  ��mOS ��w$  ��locksHeldLen ��   �locksHeld ���$    &runtime.m ��  !*runtime.g ��  ��T      %runtime.g ����s�     �stack  S�   �stackguard0 ��    �stackguard1 ��    �_panic  f�   �_defer (��   �m 0��   �sched 8)�   �syscallsp p�    �syscallpc x�    �stktopsp ���    �param ���    �atomicstatus ����   �stackLock ����   �goid ��Y�   �schedlink ��<�   �waitsince ��Y�   �wa   �preempt ��K�   �preemptStop ��K�   �preemptShrink ��K�   �asyncSafePoint ��K�   �paniconfault ��K�   �gcscandone ��K�   �throwsplit ��K�   �activeStackChans ��K�   �parkingOnChan ����   �raceignore ����   �sysblocktraced ��K�   �tracking ��K�   �trackingSeq ����   �runnableStamp ��Y�   �runnableTime ��Y�   �sysexitticks ��Y�   �traceseq ��)
           �tracelastp ��k�   �lockedm ����   �sig ����   �writebuf ����   �sigcode0 ���    �sigcode1 ���    �sigpc ���    �gopc ���    �ancestors ����   �startpc ���    �racectx ���    �waiting ����   �cgoCtxt ����   �labels ���    �timer ����   �selectDone ����   �gcAssistBytes ��Y�    &runtime.g ~�  %runtime.stack ��@�      �lo  �    �h�     &runtime.stack %�  !*runtime._panic ��  ��O      %runtime._panic 8��D�     �argp  �    �arg4�   �link �f�   �pc  �    �sp (�    �recovered 0K�   �aborted 1K�   �goexit 2K�    &runtime._panic ��  �interface {} y�  � �       &interface {} ��  %runtime.eface ��        �_type  !�   �data�     &runtime.eface F�  !*runtime._defer Y�  �@O      %runtime._defer H�@[�     �siz  ��   �started �K�   �heap �K�   �openDefer �K�   �sp�    �pc ��    �fn �m�   �_panic  f�   �link (��   �fd 0�    �varp 8�    �framepc @�     &runtime._defer ��  !*runtime.funcval ��  �@T      %runtime.funcval���      �fn  �     &runtime.funcval ��  %runtime.gobuf 8� @�     �sp  �    �pc�    �g �<�   �ctxt ��    �ret  �    �lr (�    �bp 0�     &runtime.gobuf ��  �runtime.guintpt@�      �int64 ��z      �runtime.puintptr`�      �runtime.muintptr �      #[]uint8 ��@r      ��  �array  p�   �len   �cap     !*[]runtime.ancestorInfo ��  �@J      #[]runtime.ancestorInfo �� l      ��  �array  ��   �len   �cap �    %runtime.ancestorInfo (� �      �pcs  ��   �goid �Y�   �gopc  �     &runtime.ancestorInfo P�  #[]uintptr ���r      �   �array  ��   �len   �cap     !*runtime.sudog ��  �@`      %runtime.sudog X��_�     �g  e�   �next��   �prev ���   �elem ��    �acquiretime  Y�   �releasetime (Y�   �ticket 0��   �isSelect 4K�   �success 5K�   �parent 8��   �waitlink @��   �waittail H��   �c P��    &runtime.sudo�  !*runtime.hchan ��  ���      %runtime.hchan `�@X�     �qcount  ��   �dataqsi��   �buf ��    �elemsize ���   �closed ���   �elemtype  !�   �sendx (��   �recvx 0��   �recvq 8 �   �sendq H �   �lock XN�    &runtime.hchan      �  %runtime.waitq ��@�      �first  ��   �las��    &runtime.waitq ��  %runtime.mutex���      �lockRankStruct  ��  ��key  �     &runtime.mutex ��  %runtime.lockRankStruct  ���       &runtime.lockRankStruct a�  !*runtime.timer 8   ��a      %runtime.timer H��F�     �pp  k�   �whenY�   �period �Y�   �f �}    �arg  4�   �seq 0�    �nextwhen 8Y�   �status @��    &runtime.timer ��  �func(interface {}, uintptr)� �      �4�  ��    &func(interface {}, uintptr) K   %runtime.gsignalStack  ���       &runtime.gsignalStack �   %runtime.sigset  � �       &runtime.sigset �   �[6]uintptr �   0���      ��   � �func()� s       &func() )!  �[2]uint32 �� �`�      ��   � !*runtime.cgoCallers �!  ��Q      �runtime.cgoCallers �   �����      ��     &runtime.cgoCallers �!  %runtime.note���      �key  �     &runtime.note �!  �[32]uintptr �   ��� �      ��     �func(*runtime.g, unsafe.Pointer) bool�`�      �e�  ��   �\�   &func(*runtime.g, unsafe.Pointer) bool %"  %struct { runtime.lock runtime.mutex; runtime.used uint32; runtime.fn func(bool) bool } ����      �lock  N�   �use��   �fn �<#    �func(bool) bool���      �K�  �\�   &func(bool) bool �#  %runtime.libcall 0� :�     �fn  �    �n�    �args ��    �r1 ��    �r2  �    �err (�     &runtime.libcall Q#  %runtime.dlogPerM  ���       &runtime.dlogPerM �#  %runtime.mOS 0� ;�     �threadLock  N�   �threa�    �waitsema ��    �resumesema ��    �highResTimer  �    �preemptExtLock (��    &runtime.mOS �#  �[10]runtime.heldLockInfo �$  �����      ��
         %runtime.heldLockInfo �� �      �lockAddr  �    �ran��    &runtime.heldLockInfo �$  !*runtime.guintptr <�  � �      !*runtime.muintptr ��  ���      %noalg.struct { F uintptr; runtime..autotmp_20 *runtime.g; runtime..autotmp_21 func(*runtime.g) } ����      �.F  �    �.autotmp_2e�   �.autotmp_21 �t&    &noalg.struct { F uintptr; runtime..autotmp_20 *runtime.g; runtime..autotmp_21 func(*runtime.g) } O%  �func(*runtime.g)��y      �e�   &func(*runtime.g) R&  %noalg.struct { F uintptr; runtime..autotmp_22 *bool } �� �      �.F  �    �.autotmp_2\�    &noalg.struct { F uintptr; runtime..autotmp_22 *bool } �&  !*runtime.moduledata �)  ��X      %runtime.moduledata ��� k�     �pcHeader  �)   �funcnametab��   �cutab  �*   �filetab 8��   �pctab P��   �pclntable h��   �ftab ��0+   �findfunctab ���    �minpc ���    �maxpc ���    �text ���    �etext ���    �noptrdata ���    �enoptrdata ���    �data ���    �edata ���    �bss ���    �ebss ���    �noptrbss ���    �enoptrbss ���    �end ���    �gcdata ���    �gcbss ���    �types ���    �etypes ���    �textsectmap ���+   �typelinks ��f,   �itablinks ���,   �ptab ���,   �pluginpath �   �pkghashes ��w-   �modulename ��   �modulehashes ��w-   �hasmain ����   �gcdatamask ��|.   �gcbssmask ��|.   �typemap ���.   �bad ��K�   �next ��$'    &runtime.moduledata F'  !*runtime.pcHeader �*  ��Z      %runtime.pcHeader @��\�     �magic  ��   �pad1 ���   �pad2 ���   �minLC ���   �ptrSize ��   �nfunc   �nfiles ���   �funcnameOffset ��    �cuOffset  �    �filetabOffset (�    �pctabOffset 0�    �pclnOffset 8�     &runtime.pcHeader �*  #[]uint32 ���q      ��  �array  ,<   �le   �cap     #[]runtime.functab ���l      �+  �array  :��  �l   �cap �    %runtime.functab ��`�      �entry  �    �funcoff�     &runtime.functab u+  #[]runtime.textsect ��@p      P,  �array  ���  �len   �cap     %runtime.textsect ��     �vaddr  �    �length�    �baseaddr ��     &runtime.textsec,  #[]int32 ���j      ��  �array  1�   �len   �cap     #[]*runtime.itab ���g      ��  �array  �D   �len   �cap     #[]runtime.ptabEntry �� o      `-  �array  ���  �len   �cap     %runtime.ptabEntr�`�      �name  ��   �typ ���    &runtime.ptabEntry +-  #[]runtime.modulehash ���m      �.  �array  ��   �len   �cap �    %runtime.modulehash (� ��     �modulename    �linktimehash �   �runtimehash  /.    &runtime.modulehash �-  !*string   ��c      %runtime.bitvector �� �      �n  ��   �bytedatp�    &runtime.bitvector E.   map[runtime.typeOff]*runtime._type ��� �@�      ��  !�  !*[]*runtime.moduledata �.  �        #[]*runtime.moduledata �� h      $'  �array  eG   �len   �cap �    %noalg.struct { F uintptr; runtime.src uintptr; runtime.dst *uintptr } ����      �.F  �    �src�    �dst ���    &noalg.struct { F uintptr; runtime.src uintptr; runtime.dst *uintptr } :/  %noalg.struct { F uintptr; runtime.typ *runtime._type; runtime.src unsafe.Pointer; runtime.off uintptr; runtime.size uintptr } (��%�     �.F  �    �typ!�   �src ��    �off ��    �size  �     &noalg.struct { F uintptr; runtime.typ *runtime._type; runtime.src unsafe.Pointer; runtime.off uintptr; runtime.size uintptr } �/  !*runtime.mspan �2  ��J�     %runtime.mspan ���@e�     �next  ;1   �pre;1   �list ��2   �startAddr ��    �npages  �    �manualFreeList (A3   �freeindex 0�    �nelems 8�    �allocCache @)
           �allocBits H_3   �gcmarkBits P_3   �sweepgen X��   �divMul \��   �allocCount `��   �spanclass b�   �state c�3   �needzero d��   �elemsize h�    �limit p�    �speciallock xN�   �specials ���3    &runtime.mspan X1  !*runtime.mSpanList *3  ���      %runtime.mSpanList ����      �first  ;1   �las;1    &runtime.mSpanList �2  �runtime.gclinkptr��      !*runtime.gcBits }3  � �      �runtime.gcBits  }      %runtime.mSpanStateBox ����      �s  �    &runtime.mSpanStateBox �3  !*runtime.special ?4  ��\      %runtime.special ��`�     �next  �3   �offset��   �kind
        ��    &runtime.special �3  %runtime.heapBits ��        �bitp  p�   �shift��   �arena ��   �last �p�    &runtime.heapBits T4  �runtime.arenaId �      !*runtime.heapArena �5  ��U      %runtime.heapArena �@A�     �bitmap  �5   �spans ��5   �pageInUse ��6   �pageMarks �6   �pageSpecials ��6   �checkmarks #6   �zeroedBase�     &runtime.heapArena �4  �[131072]uint8 ��  �� �      ��   � �[512]*runtime.mspan ;1  � � �      ��   �� �[64]uint8 ��  @� �      ��   @ !*runtime.checkmarksMap x6  ��R      �runtime.checkmarksMap ��  ���� �      ��   ��� &runtime.checkmarksMap H6  !*runtime.structfield ��  �        !*runtime.arraytype
        �  �        !*runtime.structtype ��  �        !*runtime.chantype }�  �        !**runtime.hchan ��  �        !**runtime.sudog ��  �        %noalg.struct { F uintptr; runtime.c *runtime.hchan } �� �      �.F  �    ���    &noalg.struct { F uintptr; runtime.c *runtime.hchan } U7  !*runtime.mutex N�  �@Y      !*runtime.waitq  �  ���      %runtime.gList� �      �head  <�    &runtime.gList �8  !*runtime.gList C8  �@�      %noalg.struct { F uintptr; runtime.pc *uintptr; runtime.ret *string } �� �      �.F  �    �pc��   �ret �/.    &noalg.struct { F uintptr; runtime.pc *uintptr; runtime.ret *string } s8  !*runtime._func �:  ��O      %runtime._func 0�@^�     �entry  �    �nameoff��   �args ��   �deferreturn ���   �pcsp ���   �pcfile ���   �pcln ���   �npcdata  ��   �cuOffset $��   �funcID (��   �flag )1�   �_ *+:   �nfuncdata +��    &runtime._func J9  �[1]uint8 ��  ����      ��   � %runtime.funcInfo �� ��     �_func  -9  ��datap$'    &runtime.funcInfo J:  %noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.callerpc uintptr; runtime.dispatch uintptr; runtime.lockedm *bool; runtime.lockedExt *uint32 } 0��3�     �.F  �    �gpe�   �callerpc ��    �dispatch ��    �lockedm  \�   �lockedExt (,<    &noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.callerpc uintptr; runtime.dispatch uintptr; runtime.lockedm *bool; runtime.lockedExt *uint32 } �:  !*uint32 ��  ��d      !*runtime.debugCallWrapArgs �<  ��R      %runtime.debugCallWrapArgs ����      �dispatch  �    �callingGe�    &runtime.debugCallWrapArgs k<  %noalg.struct { F uintptr; runtime.ok *bool } �� �      �.F  �    �ok\�    &noalg.struct { F uintptr; runtime.ok *bool } �<  �runtime.he�        !*runtime.context 9@  �        %runtime.context �        �        �p1home  )
           �p2home)
           �p3home �)
           �p4home �)
           �p5home  )
           �p6home ()
           �contextflags 0��   �mxcsr 4��   �segcs 8��   �segds :��   �seges <��   �segfs >��   �seggs @��   �segss B��   �eflags D��   �dr0 H)
           �dr1 P)
           �dr2 X)
           �dr3 `)
           �dr6 h)
           �dr7 p)
           �rax x)
           �rcx ��)
           �rdx ��)
           �rbx ��)
           �rsp ��)
           �rbp ��)
           �rsi ��)
           �rdi ��)
           �r8 ��)
           �r9 ��)
           �r10 ��)
           �r11 ��)
           �r12 ��)
           �r13 ��)
           �r14 ��)
           �r15 ��)
           �rip ��)
           �anon0 ��N@   �vectorregister ��q@   �vectorcontrol �        )
           �debugcontrol �      )
           �lastbranchtorip �   )
           �lastbranchfromrip � )
           �lastexceptiontorip �        )
           �lastexceptionfromrip �      )
            &runtime.context �=  �[512]uint8 ��  ���        ��   �� �[26]runtime.m128a �@  ���        ��   � %runtime.m128a ��        �low  )
           �highY�    &runtime.m128a �@  #[]string �� q        �array  /.   �len   �cap �    �[6]string  `�`�      ��   � !*runtime.TypeAssertionError �A  � �      %runtime.TypeAssertionError (����     �_interface  !�   �concrete!�   �asserted �!�   �missingMethod �    &runtime.TypeAssertionError dA  $runtime.errorString �� �      �str  p�   �len    %runtime.errorAddressString �� ��     �msg    �addr ��     &runtime.errorAddressString (B  $runtime.plainError ����      �str  p�   �len    %runtime.boundsError ��@5�     �x  Y�   �y   �signed �K�   �code ���    &runtime.boundsError �B  �[100]uint8 ��  d� `w      �float64 ��w      �complex64� w      �complex128 ����v      �[32]uint8 ��   �        ��     �[9]string   �����      ��       !*[8]uint8 �D  �        �[8]uint8 ���@�      ��   !*[4]uint8  �        !*runtime.itabTableType �D  � �      %runtime.itabTableType � ����     �size  �    �count�    �entries ��D    &runtime.itabTableType kD  �[512]*runtime.itab ��  � ���      ��   �� !**runtime.itab ��  �        %noalg.struct { F uintptr; R *runtime.itabTableType } ����      �F  �    �RFD    &noalg.struct { F uintptr; R *runtime.itabTableType } �E  !*runtime.imethod �  �@V      !*runtime.uncommontype JF  �        %runtime.uncommontype ��        �pkgpath  ��   �mcount ���   �xcount ���   �moff��   �_ ��    &runtime.uncommontype �E  #[]runtime.method ��        �F  �array  ���  �l   �cap     %runtime.method ��        �name  ��   �mtyp ���   �ifn�G   �tfn �G    &runtime.method �F  �runtime.textOff ���        #[]unsafe.Pointer ���r      �   �array  ��   �len   �cap �    !**runtime.moduledata $'  �        �func(*runtime.itab)�        ���   &func(*runtime.itab) �G  !*runtime.lfstack �G  �        �runtime.lfstack�        !*runtime.lfnode UH  ��V      %runtime.lfnode ��@�      �next  )
           �pushcnt�     &runtime.lfnode �H  !*runtime.note �!  � Z      !*runtime.arenaHint �H  ��P      %runtime.arenaHint ����      �addr  �    �downK�   �next ��H    &runtime.arenaHint �H  !*runtime.fixalloc �I  ���      %runtime.fixalloc H��K�     �size  �    �firs�I   �arg ��    �list �'J   �chunk  �    �nchunk (��   �inuse 0�    �stat 8}J   �zero @K�    &runtime.fixalloc �I  �func(unsafe.Pointer, unsafe.Pointer�@�      ��   ��    &func(unsafe.Pointer, unsafe.Pointer) �I  !*runtime.mlink jJ  ��X      %runtime.mlink� �      �next  'J    &runtime.mlink DJ  !*runtime.sysMemStat �J  � �      �runtime.sysMemStat��~      !**runtime.heapArena �4  �        !**[1048576]*runtime.heapArena K  �        !*[1048576]*runtime.heapArena 7K  ��I      �[1048576]*runtime.heapArena �4  �������      ��   ��@ !*runtime.mheap FN  ��S�     %runtime.mheap �����g�     �lock  N�   �page�O   �sweepgen �����   �sweepDrained �����   �sweepers �����   �allspans ����S   �_ Ђ���   �pagesInUse ؂�)
           �pagesSwept ���)
           �pagesSweptBasis ���)
           �sweepHeapLiveBasis ���)
           �sweepPagesPerByte ���uC   �scavengeGoal ���)
           �reclaimIndex ���)
           �reclaimCredit ����    �arenas ����S   �heapArenaAlloc ���XT   �arenaHints ����H   �arena ���XT   �allArenas ���qT   �sweepArenas ���qT   �markArenas ���qT   �curArena ����T   �_ �����   �central ����U   �spanalloc ����I   �cachealloc ����I   �specialfinalizeralloc ����I   �specialprofilealloc ����I   �specialReachableAlloc ����I   �speciallock ���N�   �arenaHintAlloc ����I   �unused ���%W    &runtime.mheap �K  %runtime.pageAlloc ����@N�     �summary  ,O   �chunks x�O   �searchAddr ���1Q   �start ���FQ   �end ���FQ   �inUse ����Q   �scav ���^R   �mheapLock ����7   �sysStat ���}J   �test ���K�    &runtime.pageAlloc YN  �[5][]runtime.pallocSum YO  x���      ��   � #[  �array  ψ   �len   �cap     �[8192]*[8192]runtime.pallocData �O  ���� �      ��   �@ !*[8192]runtime.pallocData �P  ��I      �[8192]runtime.pallocData vP  ��@���      ��   �@ %runtime.pallocData ��� �      �pallocBits  �P  ��scavenged @�P    &runtime.pallocData 3P  �runtime.pallocBits )
          @���      ��   &runtime.pallocBits �P  �runtime.pageBits )
          @���      ��   &runtime.pageBits �P  %runtime.offAdd� 8�     �a  �     &runtime.offAddr Q  �runtime.chunkIdx`�      %runtime.addrRanges (�`�      �ranges  �Q   �totalBytes ��    �sysStat  }J    &runtime.addrRanges cQ  #[]runtime.addrRange ���k      GR  �array  ��   �len   �cap     %runtime.addrRange ��`&�     �base  1Q   �limi1Q    &runtime.addrRange �R  %struct { runtime.inUse runtime.addrRanges; runtime.gen uint32; runtime.reservationBytes uintptr; runtime.released uintptr; runtime.scavLWM runtime.offAddr; runtime.freeHWM runtime.offAddr } P� 6�     �inUse  �Q   �gen (��   �reservationBytes 0�    �released 8�    �scavLWM @1Q   �freeHWM H1Q    #[]*runtime.mspan ��@h      ;1  �array  ��   �l   �cap     �[64]*[1048576]*runtime.heapArena K  ���@�      ��   @ %runtime.linearAlloc  ����     �next  �    �mappe�    �end ��    �mapMemory �K�    &runtime.linearAlloc �T  #[]runtime.arenaIdx ��@l      �4  �array  �k   �len   �cap     %struct { runtime.base uintptr; runtime.end uintptr } �� �      �base  �    �en�     �[136]struct { runtime.mcentral runtime.mcentral; runtime.pad [24]uint8 } qU  ������      ��   �� %struct { runtime.mcentral runtime.mcentral; runtime.pad [24]uint8 } �����      �mcentral  (V   �pad ���W    %runtime.mcentral ���`��     �spanclass  �   �partial>V   �full X>V    &runtime.mcentral �U  �[2]runtime.spanSet �V  P�@�      ��   � %runtime.spanSet (� 0�     �spineLock  N�   �spine�    �spineLen ��    �spineCap ��    �index  �V    &runtime.spanSet gV  �runtime.headTailInde� �      �[24]uint8 ��  ����      ��   � !*runtime.specialfinalizer �W  � ]      %runtime.specialfinalizer 0��0�     �special  ?4   �fn �m�   �nret ��    �fint  !�   �ot (�W    &runtime.specialfinalizer MW  !*runtime.ptrtype f�  � \      !*runtime.notInHeap )X  ���      %runtime.notInHeap  �@�       &runtime.notInHeap �X  !*runtime.mcache �X  �@�      %runtime.mcache �     ��E�     �nextSample  �    �scanAlloc�    �tiny ��    �tinyoffset ��    �tinyAllocs  �    �alloc (�Y   �stackcache >Y   �flushGen �     ��    &runtime.mcache ^X  �[136]*runtime.mspan ;1  �`�      ��   �� �[2]runtime.stackfreelist �Y   ���      ��   � %runtime.stackfreelist �� �      �list  A3   �size�     &runtime.stackfreelist mY  %runtime.gcTrigger ��        �kind  c   �nowY�   �n ���    &runtime.gcTrigger �Y  !*runtime.p q]  � �      %runtime.p �N�@o�     �id  ��   �status ���   �link�   �schedtick ���   �syscalltick ���   �sysmontick ��]   �m 8��   �mcache @@X   �pcache HB^   �raceprocctx `�    �deferpool hY^   �deferpoolbuf ���^   �goidcache ��)
           �goidcacheend ��)
           �runqhead ����   �runqtail ����   �runq ��"_   �runnext ��<�   �gFree ��P_   �sudogcache ���_   �sudogbuf ���_   �mspancache �$`   �tracebuf �,�`   �traceSweep �,K�   �traceSwept �,�    �traceReclaimed �,�    �palloc �,�`   �_ �,��   �timer0When �,)
           �timerModifiedEarliest �,)
           �gcAssistTime �,Y�   �gcFractionalMarkTime �-Y�   �gcMarkWorkerMode �-#   �gcMarkWorkerStartTime �-Y�   �gcw �-za   �wbBuf �-w�   �runSafePointFn �M��   �statsSeq �M��   �timersLock �MN�   �timers �Mpb   �numTimers �M��   �adjustTimers �M��   �deletedTimers �N��   �timerRaceCtx �N�    �preempt �NK�    &runtime.p 0Z  %runtime.sysmontick  ����     �schedtick  ��   �schedwheY�   �syscalltick ���   �syscallwhen �Y�    &runtime.sysmontick �]  %runtime.pageCache �����     �base  �    �cache)
           �scav �)
            &runtime.pageCache �]  �[5][]*runtime._defer �^  x� �      ��   � #[]*runtime._defer ���f      ��  �array  5�   �len   �cap �    �[5][32]*runtime._defer �^  �
        ���      ��   � �[32]*runtime._defer ��  �����      ��     �[256]runtime.guintptr <�  �����      ��   �� %struct { runtime.gList; runtime.n int32 } ����      �gList  C8  ����    #[]*runtime.sudog ���h      ��  �array  77   �l   �cap     �[128]*runtime.sudog ��  ���      ��   �� %struct { runtime.len int; runtime.buf [128]*runtime.mspan } � �      �len    �bufk`    �[128]*runtime.mspan ;1  �@�      ��   �� �runtime.traceBufPtr��      %runtime.persistentAlloc ����      �base  �W   �of�     &runtime.persistentAlloc �`  %runtime.gcWork (�`-�     �wbuf1  �a   �wbuf2�a   �bytesMarked �)
           �scanWork �Y�   �flushedWork  K�    &runtime.gcWork �a  !*runtime.workbuf �a  � �      %runtime.workbuf �����      �workbufhdr  3b  ��obj �Kb    &runtime.workbuf �a  %runtime.workbufhdr �� �      �node  UH   �nobj �    &runtime.workbufhdr �a  �[253]uintptr �   ���`�      ��   �� #[]*runtime.timer ���h      ��  �array  ��   �l   �cap �    %noalg.struct { F uintptr; runtime.size uintptr; runtime.align uintptr; runtime.sysStat *runtime.sysMemStat; runtime.p **runtime.notInHeap } (��#�     �.F  �    �size�    �align ��    �sysStat �}J   �p  �d    &noalg.struct { F uintptr; runtime.size uintptr; runtime.align uintptr; runtime.sysStat *runtime.sysMemStat; runtime.p **runtime.notInHeap } �b  !**runtime.notInHeap �W  ��H      !*runtime.persistentAlloc �`  ��[      !*runtime.linearAlloc XT  ���      !*runtime.hmap 9e  ����     %runtime.hmap 0� M�     �count     �flag��   �B    ��   �noverflow
        ��   �hash0 ��   �buckets ��    �oldbuckets ��    �nevacuate  �    �extra (Ke    &runtime.hmap �d  !*runtime.mapextra �e  �@X      %runtime.mapextra �����     �overflow  �e   �oldoverflow�e   �nextOverflow �7f    &runtime.mapextra ke  !*[]*runtime.bmap �e  � J      #[]*runtime.bmap ��@g      7f  �array  ���  �l   �cap �    !*runtime.bmap {f  � �      %runtime.bmap� �      �tophash  �D    &runtime.bmap Sf  !*runtime.maptype ��  �        �[2]runtime.evacDst �g  @���      ��   � %runtime.evacDst  ����     �b  7f   �i   �k ��    �e ��     &runtime.evacDst �f  !*internal/abi.IntArgRegBitmap t�  � �      !*runtime.mSpanStateBox �3  � �      �[40]uint8 ��  (�        ��   ( %noalg.struct { F uintptr; runtime.c **runtime.mcache } ����      �.F  �    �c-h    &noalg.struct { F uintptr; runtime.c **runtime.mcache } �g  !**runtime.mcache @X  �@H      %noalg.struct { F uintptr; runtime.c *runtime.mcache } ��        �.F  �    �c@X    &noalg.struct { F uintptr; runtime.c *runtime.mcache } Lh  �[1]uint64 )
         �        ��   � !*runtime.heapStatsDelta Nj  �        %runtime.heapStatsDelta �    �        �committed  Y�   �releaseY�   �inHeap �Y�   �inStacks �Y�   �inWorkBufs  Y�   �inPtrScalarBits (Y�   �tinyAllocCount 0�    �largeAlloc 8�    �largeAllocCount @�    �smallAllocCount Hjj   �largeFree ���    �largeFreeCount ���    �smallFreeCount ��jj   �_ �    �j    &runtime.heapStatsDelta "i  �[68]uintptr �   ���        ��   D �[0]uint32 ��   �        ��     !*runtime.mcentral (V  �@ �     %runtime.sweepLocker�        �sweepGen  ��   �blocking �K�    &runtime.sweepLocker �j  %runtime.sweepLocked�       k  �        !*runtime.arenaIdx �4  ���      %runtime.markBits �� 7�     �bytep  p�   �mask��   �index ��     &runtime.markBits �k  !*runtime.finblock �l  �        %runtime.finblock ���        �alllinkl   �nexl   �cnt ���   �_ ���   �fin ��l    &runtime.finblock (l  �[101]runtime.finalizer �m  ���        ��   e %runtime.finalizer (�        �fn  m�   �arg�    �nret ��    �fint �!�   �ot  �W    &runtime.finalizer �l  %noalg.struct { F uintptr; runtime.now *int64 } ����      �.F  �    �now�m    &noalg.struct { F uintptr; runtime.now *int64 } 0m  !*int64 Y�  ��L      !*runtime.timeHistogram 1n  �        %runtime.timeHistogram �-�        �counts  Ln   �underflow �-)
            &runtime.timeHistogram �m  �[720]uint64 )
          �-�        ��   �� %noalg.struct { F uintptr; runtime.restart *bool } ����      �.F  �    �restart\�    &noalg.struct { F uintptr; runtime.restart *bool } pn  !**runtime.p �Z  �        !*runtime.gcWork za  �@'�     �[3]int64 Y�  ��        ��   � �[5]int64 Y�  (�        ��   � %noalg.struct { F uintptr; runtime.startTime int64 } ��        �.F  �    �startTimeY�    &noalg.struct { F uintptr; runtime.startTime int64 } to  %noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.pp *runtime.p } �� �      �.F  �    �gpe�   �pp ��Z    &noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.pp *runtime.p }p  !*runtime.gcBgMarkWorkerNode 9q  ��T      %runtime.gcBgMarkWorkerNode  �`��     �node  UH   �gp �<�   �m ���    &runtime.gcBgMarkWorkerNode �p  !*runtime.gQueue �q  �        %runtime.gQueue ��        �head  <�   �tai<�    &runtime.gQueue wq  !*runtime.sweepClass �  �        %noalg.struct { F uintptr; runtime.i *int } ����      �.F  �    �iZr    &noalg.struct { F uintptr; runtime.i *int } �q  !*int   � L      %noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.gcw *runtime.gcWork } ��`�      �.F  �    �gpe�   �gcw ��o    &noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.gcw *runtime.gcWork } mr  %noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.scanWork int64 } ����      �.F  �    �gpe�   �scanWork �Y�    &noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.scanWork int64 } ;s  %noalg.struct { F uintptr; runtime.state *runtime.stackScanState; runtime.gcw *runtime.gcWork } �� �      �.F  �    �state�t   �gcw ��o    &noalg.struct { F uintptr; runtime.state *runtime.stackScanState; runtime.gcw *runtime.gcWork } �t  !*runtime.stackScanState �u  ���      %runtime.stackScanState ����O�     �cache  �v   �stack ��S�   �conservative ��K�   �buf ���w   �freeBuf ���w   �cbuf ���w   �head x   �tail x   �nobjs �   �root ���y    &runtime.stackScanState �u  %runtime.pcvalueCache ��� �      �entries  4v    &runtime.pcvalueCache �u  �[2][8]runtime.pcvalueCacheEnt iv  �����      ��   � �[8]runtime.pcvalueCacheEnt �v  �����      ��  %runtime.pcvalueCacheEnt �����     �targetpc  �    �of��   �val ��    &runtime.pcvalueCacheEnt �v  !*runtime.stackWorkBuf jw  �@^      %runtime.stackWorkBuf �����      �stackWorkBufHdr  �w  ��obj  �w    &runtime.stackWorkBuf &w  %runtime.stackWorkBufHdr  �`�      �workbufhdr  3b  ��next ��w    &runtime.stackWorkBufHdr �w  �[252]uintptr �   ��� �      ��   �� !*runtime.stackObjectBuf vx  ��]      %runtime.stackObjectBuf �����      �stackObjectBufHdr  �x  ��obj  �x    &runtime.stackObjectBuf .x  %runtime.stackObjectBufHdr  ���      �workbufhdr  3b  ��next x    &runtime.stackObjectBufHdr �x  �[63]runtime.stackObject �y  �����      ��   ? %runtime.stackObject  ��1�     �off  ��   �size ���   �typ!�   �left ��y   �right ��y    &runtime.stackObject $y  !*runtime.stackObject �y  �@�      �func(*runtime.stkframe, unsafe.Pointer) bool�        �5z  ��   �\�   &func(*runtime.stkframe, unsafe.Pointer) bool �y  !*runtime.stkframe �z  �@_      %runtime.stkframe X� Q�     �fn  �:   �pc ��    �continpc ��    �lr  �    �sp (�    �fp 0�    �varp 8�    �argp @�    �arglen H�    �argmap P�z    &runtime.stkframe Uz  !*runtime.bitvector |.  � �      !*runtime.stackObjectRecord �{  � ^      %runtime.stackObjectRecord �� �      �off     �ty!�    &runtime.stackObjectRecord H{  #[]runtime.stackObjectRecord ���o      �{  �array  �{   �len   �cap �    �func() bool�`x      �\�   &func() bool �{  !*runtime.gcControllerState ,~  �        %runtime.gcControllerState ���        �gcPercent  ��   �_ ���   �heapMinimum)
           �triggerRatio �uC   �trigger �)
           �heapGoal  )
           �lastHeapGoal ()
           �heapLive 0)
           �heapScan 8)
           �heapMarked @)
           �scanWork HY�   �bgScanCredit PY�   �assistTime XY�   �dedicatedMarkTime `Y�   �fractionalMarkTime hY�   �idleMarkTime pY�   �markStartTime xY�   �dedicatedMarkWorkersNeeded ��Y�   �assistWorkPerByte ��)
           �assistBytesPerWork ��)
           �fractionalUtilizationGoal ��uC   �_ ��z~    &runtime.gcControllerState I|  %internal/cpu.CacheLinePad @�        �_  �6    &internal/cpu.CacheLinePad K~  %noalg.struct { F uintptr; runtime.released *uintptr; runtime.crit *float64 } �� �      �.F  �    �released��   �crit �j�    &noalg.struct { F uintptr; runtime.released *uintptr; runtime.crit *float64 } �~  !*float64 uC  ��K      !*runtime.pageAlloc �O  ��Y�     !*runtime.addrRanges �Q  �`��     %noalg.struct { F uintptr; runtime.p *runtime.pageAlloc; runtime.minPages uintptr } ����      �.F  �    �p��   �minPages ��     &noalg.struct { F uintptr; runtime.p *runtime.pageAlloc; runtime.minPages uintptr } ��  �func(runtime.addrRange) (runtime.chunkIdx, bool)�        �GR  � �  �\�   &func(runtime.addrRange) (runtime.chunkIdx, bool) ��  !*runtime.chunkIdx FQ   ��      !*runtime.pallocData vP  ��C�     !**runtime.stackWorkBuf �w  �@I      �[2]**runtime.stackWorkBuf b�  �� �      ��   � %runtime.specialsIter ����      �pprev �   �s�3    &runtime.specialsIter ��  !**runtime.special �3  � I      !*runtime.sweepLocked Sk  �        !*runtime.specialsIter ��  � �      !*runtime.markBits �k  ���      %noalg.struct { F uintptr; runtime.s **runtime.mspan } ����      �.F  �    �s��    &noalg.struct { F uintptr; runtime.s **runtime.mspan } ��  !**runtime.mspan ;1  ��H      %noalg.struct { F uintptr; runtime.preemptible bool } ��        �.F  �    �preemptibleK�    &noalg.struct { F uintptr; runtime.preemptible bool } =�  %noalg.struct { F uintptr; runtime.h *runtime.mheap; runtime.npages uintptr; runtime.spanclass runtime.spanClass; runtime.s **runtime.mspan } (� "�     �.F  �    �hnK   �npages ��    �spanclass ��   �s  ��    &noalg.struct { F uintptr; runtime.h *runtime.mheap; runtime.npages uintptr; runtime.spanclass runtime.spanClass; runtime.s **runtime.mspan } Ճ  !*runtime.mSpanState �  � X      !*runtime.pageCache B^  ���      %noalg.struct { F uintptr; runtime.h *runtime.mheap; runtime.s *runtime.mspan } ��        �.F  �    �hnK   �s �;1    &noalg.struct { F uintptr; runtime.h *runtime.mheap; runtime.s *runtime.mspan } ~�  !*runtime.specialprofile ��  �        %runtime.specialprofile ��        �special  ?4   �b �Ɔ    &runtime.specialprofile o�  !*runtime.bucket I�  ���      %runtime.buc   �hash ��    �size  �    �nstk (�     &runtime.bucket ��  !*runtime.specialReachable և  �        %runtime.specialReachable ��        �special  ?4   �done �K�   �reachable �K�    &runtime.specialReachable ��  !*runtime.gcBitsArena ]�  �        %runtime.gcBitsArena ����        �free  �    �next��   �bits �v�    &runtime.gcBitsArena ��  �[65520]runtime.gcBits }3  ����        ��   ���      %noalg.struct { F uintptr; runtime.firstFree *struct { runtime.base runtime.offAddr; runtime.bound runtime.offAddr } } ����      �.F  �    �firstFre�    &noalg.struct { F uintptr; runtime.firstFree *struct { runtime.base runtime.offAddr; runtime.bound runtime.offAddr } } ��  !*struct { runtime.base runtime.offAddr; runtime.bound runtime.offAddr } ^�  ��c      %struct { runtime.base runtime.offAddr; runtime.bound runtime.offAddr } ����      �base  1Q   �bound1Q    �func(runtime.offAddr, uintptr)�        �1Q  ��    &func(runtime.offAddr, uintptr) ʊ  !*runtime.pallocBits �P  � <�     !*runtime.pageBits �P  � )�     �[5]uint ��  (�        ��   � %runtime.notInHeapSlice ��        �array  �W   �l   �cap     &runtime.notInHeapSlice ��  %noalg.struct { F uintptr; runtime.addrRangeToSummaryRange func(int, runtime.addrRange) (int, int); runtime.summaryRangeToSumAddrRange func(int, int, int) runtime.addrRange } �� �      �.F  �    �addrRangeToSummaryRange��   �summaryRangeToSumAddrRange �X�    &noalg.struct { F uintptr; runtime.addrRangeToSummaryRange func(int, runtime.addrRange) (int, int); runtime.summaryRangeToSumAddrRange func(int, int, int) runtime.addrRange } ��  �func(int, runtime.addrRange) (int, int)�`�      �  �GR  �Zr  �Zr   &func(int, runtime.addrRange) (int, int) ��  �func(int, int, int) runtime.addrRang� �      �  �  �  ���   &func(int, int, int) runtime.addrRange ��  !*runtime.addrRange GR   ��      %noalg.struct { F uintptr; runtime.p *runtime.pageAlloc } ����      �.F  �    ���    &noalg.struct { F uintptr; runtime.p *runtime.pageAlloc } ��  �func(int, runtime.addrRange) runtime.addrRange�        �  �GR  ���   &func(int, runtime.addrRange) runtime.addrRange :�  !*[32]uintptr �"  �        !*runtime.memRecord /�  �        %runtime.memRecord ���        �active  ��   �future  Ő    &runtime.memRecord ��  %runtime.memRecordCycle  �        �allocs  �    �free�    �alloc_bytes ��    �free_bytes ��     &runtime.memRecordCycle F�  �[3]runtime.memRecordCycle ��  `�        ��   � !*runtime.memRecordCycle ��  �        %noalg.struct { F uintptr; runtime.p unsafe.Pointer; runtime.b *runtime.bucket } ����      �.F  �    �p�    �b �Ɔ    &noalg.struct { F uintptr; runtime.p unsafe.Pointer; runtime.b *runtime.bucket } ��  %noalg.struct { F uintptr; runtime.pc uintptr; runtime.sp uintptr; runtime.gp *runtime.g }  �`��     �.F  �    �pc�    �sp ��    �gp �e�    &noalg.struct { F uintptr; runtime.pc uintptr; runtime.sp uintptr; runtime.gp *runtime.g } ��  !*runtime.spanSetBlock D�  �        %runtime.spanSetBlock � �        �lfnode  UH  ��popped ���   �spans ��5    &runtime.spanSetBlock ��  !*runtime.spanSet �V  ���      !*runtime.headTailIndex �V  � (�     !*runtime.spanSetBlockAlloc ��  �        %runtime.spanSetBlockAlloc�        �stack  �G    &runtime.spanSetBlockAlloc ˓  !**runtime.spanSetBlock Ւ  �        !*runtime.consistentHeapStats ��  �        %runtime.consistentHeapStats ���        �stats  ��   �gen ����   �noPLock ��N�    &runtime.consistentHeapStats m�  �[3]runtime.heapStatsDelta Nj  ���        ��   � !*runtime.pollDesc ��  ���      %runtime.pollDesc ����a�     �link  ��   �locN�   �fd ��    �closing �K�   �everr �K�   �user ���   �rseq  �    �rg (�    �rt 08    �rd xY�   �wseq ���    �wg ���    �wt ��8    �wd ��Y�   �self ����    &runtime.pollDesc 1�  !*runtime.net_op ��  ��Y      %runtime.net_op 8� /�     �o  ��   �pd  ��   �mode (��   �errno ,��   �qty 0��    &runtime.net_op 2�  %runtime.overlapped  �`��     �internal  �    �internalhigh�    �anon0 ��D   �hevent �p�    &runtime.overlapped ��  �[64]runtime.overlappedEntry ��  �����      ��   @ %runtime.overlappedEntry  � ��     �key  �    �op��   �internal ��    �qty ���    &runtime.overlappedEntry A�  �[10]uint8 ��
        �        ��
 �[14]uint8 ��  ��        ��   � �[15]uint8 ��  ��        ��   � �[16]uint8 ��  ��        ��   � �[17]uint8 ��  ��        ��   � �[18]uint8 ��  ��        ��   � �[22]uint8 ��  ��        ��   � �[23]uint8 ��  ��        ��   � �[27]uint8 ��  ��        ��   � �[39]uint8 ��  '�        ��   ' %runtime._DEVICE_NOTIFY_SUBSCRIBE_PARAMETERS·2 ��        �callback  �    �contex�     &runtime._DEVICE_NOTIFY_SUBSCRIBE_PARAMETERS·2 2�  "runtime.stdFunction %runtime.systeminfo 0�`R�     �anon0    �dwpagesize ���   �lpminimumapplicationaddressp�   �lpmaximumapplicationaddress �p�   �dwactiveprocessormask ��    �dwnumberofprocessors  ��   �dwprocessortype $��   �dwallocationgranularity (��   �wprocessorlevel ,��   �wprocessorrevision .��    &runtime.systeminfo ՙ  �[26]uint8 ��  ��        ��   � !*int32 ��  ��L      !*uint16 ��  ��d      #[]uint16 ��        ��  �array  F�   �len   �cap �    %noalg.struct { F uintptr; runtime.result uintptr } ��        �.F  �    �result�     &noalg.struct { F uintptr; runtime.result uintptr } ��  %runtime.memoryBasicInformation 0�        �baseAddress  �    �allocationBase�    �allocationProtect ���   �regionSize ��    �state  ��   �protect $��   �type_ (��    &runtime.memoryBasicInformation '�  %noalg.struct { F uintptr; runtime.us uint32 } ��        �.F  �    �us��    &noalg.struct { F uintptr; runtime.us uint32 } ��  �[1247]uint8 ��  �  �        ��   �  �[5]int32 ��  ��        ��   � %noalg.struct { F uintptr; runtime.pp *runtime.p; runtime.sc uintptr } ����      �.F  �    �p�Z   �sc ��     &noalg.struct { F uintptr; runtime.pp *runtime.p; runtime.sc uintptr } ��  %noalg.struct { F uintptr; runtime.siz int32; runtime.d **runtime._defer } ����      �.F  �    �si��   �d �5�    &noalg.struct { F uintptr; runtime.siz int32; runtime.d **runtime._defer } r�  !**runtime._defer ��  ��G      !*[]*runtime._defer �^  �        �error Ɵ  ���       &error u�  %runtime.iface ��        �tab  ��   �dat�     &runtime.iface ��  �runtime.stringer Ɵ  � �       &runtime.stringer ٟ  %noalg.struct { F uintptr; runtime.pc uintptr; runtime.sp unsafe.Pointer; runtime.gp *runtime.g; runtime.prevDefer *runtime._defer } (��"�     �.F  �    �pc�    �sp ��    �gp �e�   �prevDefer  ��    &noalg.struct { F uintptr; runtime.pc uintptr; runtime.sp unsafe.Pointer; runtime.gp *runtime.g; runtime.prevDefer *runtime._defer } ��  %noalg.struct { F uintptr; runtime.prevDefer *runtime._defer; runtime.gp *runtime.g } ��`�      �.F  �    �prevDefe��   �gp �e�    &noalg.struct { F uintptr; runtime.prevDefer *runtime._defer; runtime.gp *runtime.g } a�  !**runtime.funcval m�  ��G      !**runtime._panic f�  �        %noalg.struct { F uintptr; runtime.s string } �� �      �.F  �        &noalg.struct { F uintptr; runtime.s string } ��  %noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.pc uintptr; runtime.sp uintptr }  � ��     �.F  �    �ge�   �pc ��    �sp ��     &noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.pc uintptr; runtime.sp uintptr } ��  %noalg.struct { F uintptr; runtime.msgs *runtime._panic; runtime.gp *runtime.g; runtime.pc uintptr; runtime.sp uintptr; runtime.docrash *bool } 0�`4�     �.F  �    �msgf�   �gp �e�   �pc ��    �sp  �    �docrash (\�    &noalg.struct { F uintptr; runtime.msgs *runtime._panic; runtime.gp *runtime.g; runtime.pc uintptr; runtime.sp uintptr; runtime.docrash *bool } ��  %runtime.suspendGState ��        �g  e�   �deaK�   �stopped    K�    &runtime.suspendGState ]�  !*[1048576]runtime.inlinedCall ��  �        �[1048576]runtime.inlinedCall ��  ���
        �        ��   ��@ %runtime.inlinedCall ��        �parent  ��   �funcID ���   �_ ���   �file ���   �lin��   �func_ ��   �parentPc ���    &runtime.inlinedCall "�  �int16 ��� z      !*runtime.slice !�  �        %runtime.slice ��        �array  �    �l   �cap �    &runtime.slice ��  �func(uintptr) uint8�        ��   �p�   &func(uintptr) uint8 4�  %noalg.struct { F uintptr; runtime.needUnlock *bool } �� �      �.F  �    �needUnloc\�    &noalg.struct { F uintptr; runtime.needUnlock *bool } w�  %noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.traceskip int } ��@�      �.F  �    �gpe�   �traceskip     &noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.traceskip int } ��  !*[]*runtime.sudog �_  �        !**runtime.g e�  �        %noalg.struct { F uintptr; runtime.oldval uint32; runtime.newval uint32 } ��        �.F  �    �oldva��   �newval ��    &noalg.struct { F uintptr; runtime.oldval uint32; runtime.newval uint32 } ��  !*runtime.puintptr k�  � �      !**runtime.m ��  � H      �func(*runtime.p�        ��Z   &func(*runtime.p) ��  %noalg.struct { F uintptr; runtime.freem **runtime.m } �� �      �.F  �    �freem��    &noalg.struct { F uintptr; runtime.freem **runtime.m } K�  %runtime.cgothreadstart ��� �     �g  <�   �tls<�   �fn ��     &runtime.cgothreadstart ߪ  !*uint64 )
          � e      #runtime.pMask ��        ��  �array  ,<   �l   �cap     &runtime.pMask R�  #[]*runtime.p ��        �Z  �array  �n   �len   �cap     %runtime.randomEnum ��        �i  ��   �count ���   �po��   �inc ��    &runtime.randomEnum ��  !*runtime.randomOrder ��  �        %runtime.randomOrder  �        �count  ��   �coprimes�*    &runtime.randomOrder l�  !*runtime.randomEnum 1�  �        %noalg.struct { F uintptr; runtime._g_ *runtime.g } ����      �.F  �    �_g_e�    &noalg.struct { F uintptr; runtime._g_ *runtime.g } ��  %noalg.struct { F uintptr; runtime.sp uintptr; runtime._g_ *runtime.g } ��@�      �.F  �    �sp�    �_g_ �e�    &noalg.struct { F uintptr; runtime.sp uintptr; runtime._g_ *runtime.g } p�  %noalg.struct { F uintptr; runtime.sp1 uintptr; runtime.sp2 uintptr; runtime.sp3 uintptr; runtime._g_ *runtime.g } (��$�     �.F  �    �sp1�    �sp2 ��    �sp3 ��    �_g_  e�    &noalg.struct { F uintptr; runtime.sp1 uintptr; runtime.sp2 uintptr; runtime.sp3 uintptr; runtime._g_ *runtime.g } .�  %noalg.struct { F uintptr; runtime.ok *bool; runtime.oldp *runtime.p; runtime._g_ *runtime.g }  ����     �.F  �    �ok\�   �oldp ��Z   �_g_ �e�    &noalg.struct { F uintptr; runtime.ok *bool; runtime.oldp *runtime.p; runtime._g_ *runtime.g } Y�  %noalg.struct { F uintptr; runtime.stacksize int32; runtime.newg *runtime.g } ����      �.F  �    �stacksize��   �newg �e�    &noalg.struct { F uintptr; runtime.stacksize int32; runtime.newg *runtime.g } Q�  %noalg.struct { F uintptr; runtime.fn **runtime.funcval; runtime.argp unsafe.Pointer; runtime.siz int32; runtime.gp *runtime.g; runtime.pc uintptr } 0��2�     �.F  �    �fnA�   �argp ��    �siz ���   �gp  e�   �pc (�     &noalg.struct { F uintptr; runtime.fn **runtime.funcval; runtime.argp unsafe.Pointer; runtime.siz int32; runtime.gp *runtime.g; runtime.pc uintptr } #�  !*runtime.gobuf )�  �@U      !*runtime.ancestorInfo ��  ��P      �[100]uintptr �   ���        ��   d %noalg.struct { F uintptr; runtime.gp *runtime.g } ����      �.F  �    �gpe�    &noalg.struct { F uintptr; runtime.gp *runtime.g }  �  %noalg.struct { F uintptr; runtime.pp *runtime.p } �� �      �.F  �    �pp�Z    &noalg.struct { F uintptr; runtime.pp *runtime.p } ��  %noalg.struct { F uintptr; runtime.grunning *int } �� �      �.F  �    �grunningZr    &noalg.struct { F uintptr; runtime.grunning *int } ��  �[129]*runtime.g e�  �        ��   �� !*[256]runtime.guintptr "_  �        !*runtime.initTask Q�  �        %runtime.initTask ��        �state  �    �ndep�    �nfns ��     &runtime.initTask ��  %runtime.tracestat  �        �active  K�   �iY�   �allocs �)
           �bytes �)
            &runtime.tracestat g�  !**uint8 p�  �        !*runtime.dbgVar 7�  �        %runtime.dbgVar ��        �name     �value �1�    &runtime.dbgVar ��  %noalg.struct { F uintptr; runtime.rw *runtime.rwmutex } �� �      �.F  �    �rw��    &noalg.struct { F uintptr; runtime.rw *runtime.rwmutex } K�  !*runtime.rwmutex ��  � �      %runtime.rwmutex 0�`B�     �rLock  N�   �reader��   �readerPass ���   �wLock �N�   �writer  ��   �readerCount (��   �readerWait ,��    &runtime.rwmutex ��  !*runtime.semaRoot ��  �        %runtime.semaRoot ��        �lock  N�   �trea��   �nwait ���    &runtime.semaRoot ��  !*runtime.exceptionrecord ��  �        %runtime.exceptionrecord ���        �exceptioncode  ��   �exceptionflags ���   �exceptionrecord��   �exceptionaddress �p�   �numberparameters ���   �exceptioninformation  ��    &runtime.exceptionrecord >�  �[15]uintptr �   x�        ��   � !*runtime.adjustinfo ��  �        %runtime.adjustinfo ���        �old  S�   �delta ��    �cache ��v   �sghi ���     &runtime.adjustinfo R�  !*runtime.pcvalueCache �v  � [      !*runtime.stackmap B�  �        %runtime.stackmap �        �n  ��   �nbit ���   �bytedata+:    &runtime.stackmap  �  !*runtime.tmpBuf ��  �        �runtime.tmpBuf ��   �        ��     &runtime.tmpBuf v�  �[2]string    � �      ��   � �[3]string   0���      ��   � �[4]string  @���      ��   � !*[32]int32 (�  �        �[32]int32 ��  ���        ��     !*[70368744177663]uint16 o�  �        �[70368744177663]uint16 ��  ��������        ��   ������� !*runtime.modulehash �.  � Y      !*runtime.findfuncbucket 0�  �        %runtime.findfuncbucket ��        �idx  ��   �subbuckets �R�    &runtime.findfuncbucket ��  !*[8]runtime.pcvalueCacheEnt iv  �        !*runtime.abiPart �  ���      %runtime.abiPart (��,�     �kind  N�   �srcStackOffset�    �dstStackOffset ��    �dstRegister    �len  �     &runtime.abiPart ��  #[]runtime.abiPart ���k      �  �array  v�   �len   �cap �    !*runtime.abiDesc ��  ���      %runtime.abiDesc 8��+�     �parts  ��   �srcStackSize ��    �dstStackSize  �    �dstRegisters (   �retOffset 0�     &runtime.abiDesc ��  %runtime.winCallback H��     �     �fn  m�   �retPop�    �abiMap ���    &runtime.winCallback ��  !*runtime.callbackArgs ��  �        %runtime.callbackArgs  �        �index  �    �args�    �result ��    �retPop ��     &runtime.callbackArgs ��  !**runtime.timer ��  �        #[]uint64 �� r      )
          �array  <�   �len   �cap     !*runtime.traceBufPtr �`  ���      !*runtime.traceBuf ܿ  �        %runtime.traceBuf ����        �traceBufHeader  I�  ��arr ��    &runtime.traceBuf ��  %runtime.traceBufHeader �        �link  �`   �lastTicks)
           �pos �   �stk �e�    &runtime.traceBufHeader ��  �[128]uintptr �   �        ��   �� �[64488]uint8 ��  ����        ��   ��� !*runtime.traceStack (�  �        %runtime.traceStack (�        �link  @�   �hash�    �id ���   �n    �stk  b�    &runtime.traceStack ��  �runtime.traceStackPt        �[0]uintptr �    �        ��     !*runtime.traceStackTable ��  �        %runtime.traceStackTable ����        �lock  N�   �seq��   �mem �P�   �tab  ��    &runtime.traceStackTable ��  %runtime.traceAlloc ��        �head  h�   �of�     &runtime.traceAlloc ��  �runtime.traceAllocBlockPt        �[8192]runtime.traceStackPtr @�  ����        ��   �@ !*runtime.traceAlloc P�  �        !*runtime.traceAllocBlock K�  �        %runtime.traceAllocBlock ����        �ne�  �[65528]uint8 ��  ����        ��   ��� !*runtime.traceAllocBlockPtr h�  �        �[2]uint64 )
          ��        ��   � �[3]uint64 )
          ��        ��   � !*runtime.inlinedCall ��  �        %noalg.struct { F uintptr; runtime.argp unsafe.Pointer } �� �      �.F  �    �arg�     &noalg.struct { F uintptr; runtime.argp unsafe.Pointer } ��  !*[171]uint8 ��  �        �[171]uint8 ��  ���        ��   �� �func(uint8, uint8)�        ���  ���   &func(uint8, uint8) ��  !*runtime.reflectMethodValue ��  �        %runtime.reflectMethodValue ��        �fn  �    �stack�z   �argLen ��     &runtime.reflectMethodValue [�  %runtime.cgoSymbolizerArg 8� ?�     �pc  �    �filp�   �lineno ��    �funcName �p�   �entry  �    �more (�    �data 0�     &runtime.cgoSymbolizerArg ��  %noalg.struct { F uintptr; runtime.pc uintptr; runtime.sp uintptr; runtime.gp *runtime.g; runtime.skip int; runtime.pcbuf []uintptr; runtime.n *int } H� =�     �.F  �    �pc�    �sp ��    �gp �e�   �skip     �pcbuf (��   �n @Zr    &noalg.struct { F uintptr; runtime.pc uintptr; runtime.sp uintptr; runtime.gp *runtime.g; runtime.skip int; runtime.pcbuf []uintptr; runtime.n *int } c�  %noalg.struct { F uintptr; runtime.me *runtime.g; runtime.curgp *runtime.g; runtime.level int32 }  ����     �.F  �    �mee�   �curgp �e�   �level ���    &noalg.struct { F uintptr; runtime.me *runtime.g; runtime.curgp *runtime.g; runtime.level int32 } ��  %noalg.struct { F uintptr; runtime.frame *runtime.stkframe; runtime.bad uintptr } ����      �.F  �    �frame5z   �bad ��     &noalg.struct { F uintptr; runtime.frame *runtime.stkframe; runtime.bad uintptr } ��  !*runtime.cgoSymbolizerArg E�  ��Q      �func(unsafe.Pointer, unsafe.Pointer) int32�        ��   ��   �1�   &func(unsafe.Pointer, unsafe.Pointer) int32 ��  %runtime.cgoTracebackArg  �`��     �context  �    �sigContext�    �buf ���   �max ��     &runtime.cgoTracebackArg \�  !**runtime._type !�  �        %noalg.map.bucket[runtime._typePai�     �topbits  �D   �keys��   �elems ��\�   �overflow ���     &noalg.map.bucket[runtime._typePair]struct {} ��  �noalg.[8]runtime._typePair ��  �����      ��  &noalg.[8]runtime._typePair ��  %runtime._typePair ����      �t1  !�   �t2!�    &runtime._typePair ��  �noalg.[8]struct {} t�   � �      ��   &noalg.[8]struct {} 3�  %struct {}  ���       %noalg.map.bucket[uint32][]*runtime._type ���`��     �topbits  �D   �keysM�   �elems (��   �overflow ���     &noalg.map.bucket[uint32][]*runtime._type ��  �noalg.[8]uint32 ��   �`�      ��  &noalg.[8]uint32 '�  �noalg.[8][]*runtime._type ��  ��� �      ��   &noalg.[8][]*runtime._type b�  #[]*runtime._type �� g      !�  �array  ��   �le   �cap     %noalg.map.hdr[runtime._typePair]struct {} 0� H�     �count    �flag��   �B    ��   �noverflow
        ��   �hash0 ��   �buckets ���   �oldbuckets ���   �nevacuate  �    �extra (�     &noalg.map.hdr[runtime._typePair]struct {} ��  !*map.bucket[runtime._typePair]struct {} f�  ��N      %noalg.map.hdr[uint32][]*runtime._type 0�@I�     �count     �flags��   �B    ��   �noverflow
        ��   �hash0 ��   �buckets ���   �oldbuckets ���   �nevacuate  �    �extra (�     &noalg.map.hdr[uint32][]*runtime._type ��  !*map.bucket[uint32][]*runtime._type ��  � O       map[uint32][]*runtime._type ��� �`�      ��  ��   map[runtime._typePair]struct {} ��� ���      ��  t�  !*runtime.functype ��  �        !*runtime.slicetype ��  �        %runtime.winCallbackKey ����      �fn  m�   �cdeclK�    &runtime.winCallbackKey ��  !*runtime.libcall �#  � W      %struct { runtime.lpFileName *uint16; runtime.hFile uintptr; runtime.flags uint32 } ��`�      �lpFileName  F�   �hFile�    �flags ���    !*runtime.errorString �A  � �      !*runtime.boundsError C  � �      !*runtime.sysmontick �]  ��`      !*struct { runtime.gList; runtime.n int32 } P_  �`�      !*runtime.mOS w$  ��W      !*runtime.lockRank ��  � �      !*runtime.wai  ���      !*runtime.errorAddressString fB  � �      !*struct { runtime.mcentral runtime.mcentral; runtime.pad [24]uint8 } qU  �        !*[136]struct { runtime.mcentral runtime.mcentral; runtime.pad [24]uint8 } �U  �        !*runtime.overlappedEntry ��  ��Z      !*runtime.plainError �B  ���      !*runtime.winCallbackKey ��  � c      !*[2]string ��  �        !*[3]string Ϻ  �        !*[4]string ��  �        !*[64]runtime.overlappedEntry ��  �        !*[9]string �C  �        !*struct { runtime.lpFileName *uint16; runtime.hFile uintptr; runtime.flags uint32 } >�  �        !*sync.Mutex F�  � �      %sync.Mutex���      �state  ��   �sema ���    &sync.Mutex ��  !*sync.Pool ��  �        %sync.Pool (�        �noCopy  ��   �local  �    �localSiz�    �victim ��    �victimSize ��    �New  9�    &sync.Pool o�  %sync.noCopy  �         &sync.noCopy ��  �func() interface {}�        �R�   &func() interface {} ��  !*interface {} 4�           !*struct { runtime/cgo.cstr *uint8 } ��  �        %struct { runtime/cgo.cstr *uint8 �        �cstr  p�    �internal/reflectlite.Kind��      �internal/reflectlite.tflag `{      �internal/reflectlite.chanDir�        �internal/reflectlite.flag        !*internal/reflectlite.rtype ^�  �`c�     %internal/reflectlite.rtype 0�@U�     �size  �    �ptrdat�    �hash ���   �tflag ���   �align ���   �fieldAlign ���   �kind ���   �equal ���   �gcdata  p�   �str (~�   �ptrToThis ,��    &internal/reflectlite.rtype ��  �internal/reflectlite.nameOff ��� {      �internal/reflectlite.typeOff ����{      %internal/reflectlite.name�        �bytes  p�    &internal/reflectlite.name ��  !*internal/unsafeheader.String ��  �        %internal/unsafeheader.String ��        �Data  �    �Len    &internal/unsafeheader.String N�  !*internal/reflectlite.arrayType :�  �        %internal/reflectlite.arrayType H�        �rtype  ^�  ��elem 0w�   �slice 8w�   �len @�     &internal/reflectlite.arrayType ��  !*internal/reflectlite.chanType ��  �        %internal/reflectlite.chanType @�        �rtype  ^�  ��elem 0w�   �dir 8�     &internal/reflectlite.chanType ��  !*internal/reflectlite.mapType ��  �        %internal/reflectlite.mapType X�        �rtype  ^�  ��key 0w�   �elem 8w�   �bucket @w�   �hasher H��   �keysize P��   �valuesize Q��   �bucketsize R��   �flags T��    &internal/reflectlite.mapType (�  !*internal/reflectlite.ptrType `�  �        %internal/reflectlite.ptrType 8�        �rtype  ^�  ��elem 0w�    &internal/reflectlite.ptrType ��  !*internal/reflectlite.sliceType ��  �        %internal/reflectlite.sliceType 8�        �rtype  ^�  ��elem 0w�    &internal/reflectlite.sliceType ��  !*internal/reflectlite.Kind ��  �@�      !*internal/reflectlite.uncommonType ��  ���      %internal/reflectlite.uncommonType ���*�     �pkgPath  ~�   �mcount ���   �xcount ���   �moff��   �_ ��    &internal/reflectlite.uncommonType r�  !*errors.errorString Q�  ���      %errors.errorString ����      �s     &errors.errorString )�  %internal/reflectlite.emptyInterface ��        �typ  w�   �wor�     &internal/reflectlite.emptyInterface i�  �syscall.Handle`�      �syscall.Errno �      �syscall.Signal �        !*[6]string �A  �        !*syscall.DLLError ��  ���      %syscall.DLLError 0�`
        �     �Err  ��   �ObjName    �Msg     &syscall.DLLError b�  !*syscall.DLL �  � �      %syscall.DLL ��`�      �Name     �Handle ���    &syscall.DLL ��  !*syscall.Proc p�  � �      %syscall.Proc  ����     �Dll  ��   �Nam   �addr ��     &syscall.Proc 4�  %noalg.struct { F uintptr; syscall..autotmp_7 *sync.Mutex } ����      �.F  �    �.autotmp_��    &noalg.struct { F uintptr; syscall..autotmp_7 *sync.Mutex } ��  !*syscall.LazyDLL ��  �@�      %syscall.LazyDLL  � ��     �mu  F�   �dl��   �Name     &syscall.LazyDLL D�  !*syscall.LazyProc ��  ���      %syscall.LazyProc (����     �mu  F�   �Na   �l �%�   �proc  ��    &syscall.LazyProc ��  �[300]uint16 ��  ���        ��   �� !*syscall.Errno ��  ���      !*struct { main.r0 int } {�  �        %struct { main.r0 int }�        �r0      �[131]string  ���        ��   �� #[]*sync.Pool ��        V�  �array  ���  �l   �cap �     map[string]bool S�� � �        K�  �[128]uint8 ��  ���        ��   �� �[4]uintptr �    �        ��   � �[8]string  ���        ��  �[33]float64 uC  ���        ��   ! �[256]uint64 )
          ���        ��   �� %struct { runtime.mutex; runtime.persistentAlloc } ��        �mutex  N�  ��persistentAllo�`  � �[1024]uint8 ��  �        ��   �[5]uint8 ��  ��        ��   � �chan int �#� ��v       %struct { runtime.enabled bool; runtime.pad [3]uint8; runtime.needed bool; runtime.cgo bool; runtime.alignme uint64 } ��        �enabled  K�   �pad �a�   �needed �K�   �cgo �K�   �alignm)
            �[3]uint8 ��  ��        ��   � %struct { runtime.full runtime.lfstack; runtime.empty runtime.lfstack; runtime.pad0 internal/cpu.CacheLinePad; runtime.wbufSpans struct { runtime.lock runtime.mutex; runtime.free runtime.mSpanList; runtime.busy runtime.mSpanList }; _ uint32; runtime.bytesMarked uint64; runtime.markrootNext uint32; runtime.markrootJobs uint32; runtime.nproc uint32; runtime.tstart int64; runtime.nwait uint32; runtime.nDataRoots int; runtime.nBSSRoots int; runtime.nSpanRoots int; runtime.nStackRoots int; runtime.baseData uint32; runtime.baseBSS uint32; runtime.baseSpans uint32; runtime.baseStacks uint32; runtime.baseEnd uint32; runtime.startSema uint32; runtime.markDoneSema uint32; runtime.bgMarkReady runtime.note; runtime.bgMarkDone uint32; runtime.mode runtime.gcMode; runtime.userForced bool; runtime.totaltime int64; runtime.initialHeapLive uint64; runtime.assistQueue struct { runtime.lock runtime.mutex; runtime.q runtime.gQueue }; runtime.sweepWaiters struct { runtime.lock runtime.mutex; runtime.list runtime.gList }; runtime.cycles uint32; runtime.stwprocs int32; runtime.maxprocs int32; runtime.tSweepTerm int64; runtime.tMark int64; runtime.tMarkTerm int64; runtime.tEnd int64; runtime.pauseNS int64; runtime.pauseSFAILURE: No signal received
FAIL
FAIL    runtime 99.834s
FAIL

c:\Users\Alex\dev\go\src>

Alex

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Apr 28, 2021

Another avenue to explore here (I don't have time anymore) is to try to run os/signal.TestCtrlBreak to see if it breaks too. Similarly, you should not run it with -counter parameter, because test build Go executable first. So just modify the test to build executable once and then run it many times in a loop.

If Go control handler code is broken, then this test might break too.

Alex

@toothrot
Copy link
Contributor

@toothrot toothrot commented Apr 29, 2021

/cc @bufflig

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 29, 2021

As an update here, I think I was able to show that a signal (or something) lands before runtime initialization is done, in these cases (but after we set up the ConsoleCtrlHandler).

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 29, 2021

Narrowed it down further! Looks like the bad signal is consistently landing between when we finish goenvs in schedinit and before we start package initialization in runtime.main.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 29, 2021

And further (I think), to between the end of goenvs and the end of schedinit...?

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 29, 2021

Apologies for how much progress has slowed down on this. I'm fairly certain there's some very particular timing required for this failure to trigger. As I add print statements in various places, it gets harder and harder to reproduce.

I'll soon take @alexbrainman's advice and try setting an infinite timeout so I can hook in a debugger. For that I need to set up my Windows environment properly, which will take some time.

Another interesting avenue could be to check Go 1.16. It would be nice to somehow rule out that this can't happen in previous releases. If it can, then I think a workaround to whatever the issue is would be more acceptable and prevent blocking the release. But, I will continue trying to get to the bottom of this.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Apr 30, 2021

Does anything change if you add a call to the Dummy function in dummy.dll before printing ready? And what happens if you don't load the DLL at all? I agree with you that the current code has a race, but I wonder which way the race goes.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 30, 2021

@ianlancetaylor I can't seem to reproduce if I add a call to Dummy, but since that involves another syscall to look up the symbol I can't be sure that it's not just skewing the timing by just enough (like the print statements did).

I'll try not loading the DLL next.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 30, 2021

Removing the DLL load also makes the failure go away. So, it's definitely due to a race with runtime initialization. I'm still unclear on the race window though.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 30, 2021

Based on my previous experiments, I think that the race window is between the control handler being set and newproc being called the very first time to create the first G.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 30, 2021

Honestly, I think this would be close to explaining what was wrong, except for two things:

  1. The garbage data emitted isn't accounted for.
  2. Theoretically the control signal path NEVER goes through sigtramp, where the badsignal error message is (presumably) coming from, and is instead going through callbackasm (which goes through the cgocallbackg path).

Here's another theory: we go through the callbackasm path but because there's no G (or M, maybe) something goes wrong before we even get to Go code (nil G, some kind of exception), triggering a sigtramp. Still doesn't explain the garbage data, though, and generally speaking the callbackasm path doesn't assume it has an m.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 30, 2021

The next thing I'm trying is I moved the _SetConsoleCtrlHandler call into runtime.main, once we're basically ready to execute user Go code. If my theory about the race window is at least somewhat on target, the failure should not reproduce.

EDIT: It did not reproduce.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 30, 2021

OK, here's at least one problem: needm needs an extra M to be around. It's going to spin forever if there isn't one there.

needm is populated by newextram which is first called in mstartm0, called by mstart1, called by mstart. The first time mstart is called is for the main goroutine.

Spinning forever and never delivering the signal to the C handler is exactly what this test is checking, BTW.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 30, 2021

Yeah, OK, I think that's it. I can't explain the garbage data, but that CL I referenced long ago basically make this path go through cgocallback and needm where it previously did not.

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 30, 2021

Change https://golang.org/cl/315830 mentions this issue: runtime: delay SetConsoleCtrlHandler until runtime.main

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 30, 2021

Ah!! I can explain the garbage, too.

Turns out that we do actually check for this case! There's a check at the beginning of needm that checks if we initialized far enough yet.

Unfortunately, what that does then is print the earlycgocallback message... but the runtime globals haven't been initialized yet! Hence all sorts of garbage is printed.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Apr 30, 2021

Cute. Nice analysis.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 30, 2021

Actually, I'm less certain now about where the garbage data is coming from. earlycgocallback won't be used (needm won't be called) before osinit, and osinit uses exactly the same kind of global variable on Linux (I've confirmed it works). I still feel like it's that print that's problematic, but I don't understand why earlycgocallback would be invalid at that point.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 30, 2021

Ah, this may be a PE vs. ELF thing? I don't see Windows runtime code relying on a variable being initialized anywhere.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 30, 2021

Yeah... If I print the length of earlycgocallback in osinit it works fine.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented May 3, 2021

Coming back to this... I think my theory is subtly wrong.

According to the MSDN, control handlers are always called on a new thread (https://docs.microsoft.com/en-us/windows/console/console-control-handlers). That thread should be distinct from the runtime initialization thread, and so shouldn't impede progress in the runtime, presumably. That thread will just spin in needm until initialization gets far enough.

The only thing preventing progress though, is that check of cgoHasExtraM at the beginning of needm. If that somehow cascades into a bunch of failures, that would explain why the test fails in such a strange manner (though, admittedly, I still don't understand how it would do so).

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented May 3, 2021

Well, hey I learned one new thing. There should be a runtime.abort call after runtime.badsignal2 on Windows. That probably explains the cascading failures (there's no way to proceed, and yet we do!).

I'm going to try to see where that signal is coming from. I have access to the instruction pointer in that context, it's just a question of how to actually surface it.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented May 4, 2021

The PC that triggers the failure isn't one that's known to Go (and it's consistently the same one). If I were to guess, it's probably a Windows DLL (win32?).

@gopherbot
Copy link

@gopherbot gopherbot commented May 4, 2021

Change https://golang.org/cl/316809 mentions this issue: runtime: abort after emitting message for bad signal on Windows

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented May 4, 2021

The saga continues. I fixed the garbage-getting-printed issue, but there's still that pesky signal firing (FWIW it's an access violation signal, so the equivalent of a segfault, and to catch you up, the PC that triggered it is NOT in the Go binary). The good news is, I've narrowed the source a little further. Namely, if I eliminate the call to SetConsoleCtrlHandler, but keep the compileCallback invocation (printing the PC so the call stays around... also I made sure it still reproduces with everything the same, just the print added in). Doing this does not reproduce.

Now what I did was I kept the print and the compileCallback in, but rather than use the result of compileCallback as the handler, I made my own little handler in assembly that basically just returns FALSE (so the next handler runs). And with that I can still reproduce. So, there's something wrong about where we're calling SetConsoleCtrlHandler specifically. It doesn't seem to involve the callback, or needm, or the cgocallback path at all.

I'm thinking now that there's, very simply (back to the very very beginning of this thread), a race between SetConsoleCtrlHandler and when the control signal lands. I do not understand what the problem could be. MSDN does not discuss any concurrency guarantees (e.g. is it safe to call SetConsoleCtrlHandler concurrently? (Note that in this case it is not being called concurrently, I just mean that as an example)).

At this point I have no idea how to proceed. My CL above is a reasonable workaround. I can also make it so that the test binary calls Dummy in the Go DLL before indicating readiness. That seems to fix it, too.

@ianlancetaylor @alexbrainman WDYT?

gopherbot pushed a commit that referenced this issue May 4, 2021
Currently if a signal lands on a non-Go thread that's handled by the Go
handler, Go will emit a message. However, unlike everywhere else in the
runtime, Go will not abort the process after, and the signal handler
will try to continue executing.

This leads to cascading failures and possibly even memory corruption.

For #45638.

Change-Id: I546f4e82f339d555bed295528d819ac883b92bc6
Reviewed-on: https://go-review.googlesource.com/c/go/+/316809
Trust: Michael Knyszek <mknyszek@google.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented May 6, 2021

The saga continues. I fixed the garbage-getting-printed issue, but there's still that pesky signal firing (FWIW it's an access violation signal, so the equivalent of a segfault, and to catch you up, the PC that triggered it is NOT in the Go binary).

I tried running test on top of 1108cbe and test failed once and I still see garbage printed:

c:\Users\Alex\dev\go\test>go test -count=10 -run=CtrlH runtime
--- FAIL: TestLibraryCtrlHandler (19.45s)
    signal_windows_test.go:211: Program exited with error: exit status 1
        runtime: signal received on thread not created by Go.
                                                                                  ,�ݪY-�A��Z�R��"on�5�M����.R�F�x58D�%��q�ч]�6I#�9�2�g�.i保�C��ߞc�T;:����8���Av���8�*�Z���oZ���۞�X����Ո��~��ӛ��k��O��                                                                                                                                                                                                                                                                runtime: signal received on thread not created by Go.
                                                                                  ,�ݪY-�A��Z�R��"on�5�M����.R�F�x58D�%��q�ч]�6I#�9�2�g�.i保�C��ߞc�T;:����8���Av���8�*�Z���oZ���۞�X����Ո��~��ӛ��k��O��        ���     ���                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             � ��������������                                                                                runtime: signal received on thread not created by Go.
                                                                                  ,�ݪY-�A��Z�R��"on�5�M����.R�F�x58D�%��q�ч]�6I#�9�2�g�.i保�C��ߞc�T;:����8���Av���8�*�Z���oZ���۞�X����Ո��~��ӛ��k��O��        ���     ���     
... more garbage ...                                                           

Is this garbage expected?

I also noticed that this message would only be printed, if runtime.sigtramp called and it could not find g. The fact that runtime.sigtramp is called means we received an exception. It would be nice to see stack trace printed. Unfortunately runtime.sigtramp end-up calling runtime.badsignal2 instead.

Perhaps we can disable runtime.badsignal2 check and hope stack trace is printed.

Another option would be run test program inside a debugger. Debugger will stop when exception. Maybe we can see the program state then. It would be nice to see exactly where exception occurs.

Perhaps your https://go-review.googlesource.com/c/go/+/315830/ fixes this error, and is good enough.

The good news is, I've narrowed the source a little further. Namely, if I eliminate the call to SetConsoleCtrlHandler, but keep the compileCallback invocation (printing the PC so the call stays around... also I made sure it still reproduces with everything the same, just the print added in). Doing this does not reproduce.

That means that your CL 315830 could not be a solution here. Correct?

I'm thinking now that there's, very simply (back to the very very beginning of this thread), a race between SetConsoleCtrlHandler and when the control signal lands. I do not understand what the problem could be. MSDN does not discuss any concurrency guarantees (e.g. is it safe to call SetConsoleCtrlHandler concurrently? (Note that in this case it is not being called concurrently, I just mean that as an example)).

What particular race are you thinking about? You are allowed to call SetConsoleCtrlHandler API many times - that is how the API is designed. In fact I am pretty sure our test calls SetConsoleCtrlHandler twice - Go calls it, and C runtime also calls it.

Perhaps the problem is that Go DLL does not remove SetConsoleCtrlHandler handler before it exits? And Windows calls the handler address when Go runtime is already off.

Similarly SetConsoleCtrlHandler is supposed to handle CTRL_CLOSE_EVENT. Perhaps we get that message (I don't see how that happens) after Go DLL runtime is off, but Windows SetConsoleCtrlHandler handler still points to Go code.

I plan to play with this more on the weekend. Let's delay the decision.

Also, perhaps @zx2c4 can help debug this, if you have time. Thank you.

Alex

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented May 6, 2021

The saga continues. I fixed the garbage-getting-printed issue, but there's still that pesky signal firing (FWIW it's an access violation signal, so the equivalent of a segfault, and to catch you up, the PC that triggered it is NOT in the Go binary).

I tried running test on top of 1108cbe and test failed once and I still see garbage printed:

c:\Users\Alex\dev\go\test>go test -count=10 -run=CtrlH runtime
--- FAIL: TestLibraryCtrlHandler (19.45s)
    signal_windows_test.go:211: Program exited with error: exit status 1
        runtime: signal received on thread not created by Go.
                                                                                  ,�ݪY-�A��Z�R��"on�5�M����.R�F�x58D�%��q�ч]�6I#�9�2�g�.i保�C��ߞc�T;:����8���Av���8�*�Z���oZ���۞�X����Ո��~��ӛ��k��O��                                                                                                                                                                                                                                                                runtime: signal received on thread not created by Go.
                                                                                  ,�ݪY-�A��Z�R��"on�5�M����.R�F�x58D�%��q�ч]�6I#�9�2�g�.i保�C��ߞc�T;:����8���Av���8�*�Z���oZ���۞�X����Ո��~��ӛ��k��O��        ���     ���                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             � ��������������                                                                                runtime: signal received on thread not created by Go.
                                                                                  ,�ݪY-�A��Z�R��"on�5�M����.R�F�x58D�%��q�ч]�6I#�9�2�g�.i保�C��ߞc�T;:����8���Av���8�*�Z���oZ���۞�X����Ո��~��ӛ��k��O��        ���     ���     
... more garbage ...                                                           

Is this garbage expected?

It's not, that's very surprising. It's also surprising that you landed in badsignal2 more than once -- all the failures I got after that CL were clean. I'll try to reproduce this again.

I also noticed that this message would only be printed, if runtime.sigtramp called and it could not find g. The fact that runtime.sigtramp is called means we received an exception. It would be nice to see stack trace printed. Unfortunately runtime.sigtramp end-up calling runtime.badsignal2 instead.

Perhaps we can disable runtime.badsignal2 check and hope stack trace is printed.

Unfortunately I don't think that'll work. There's a lot of stuff that depends on having some kind of G (including a g0); I'm not even positive we can throw in that circumstance.

Another option would be run test program inside a debugger. Debugger will stop when exception. Maybe we can see the program state then. It would be nice to see exactly where exception occurs.

I've been putting this aside because I don't have an environment where I can easily attach a debugger and follow subprocesses. The Windows world feels alien to me. I don't have high confidence in Delve handling this very specific situation very well, but I can try.

Perhaps your https://go-review.googlesource.com/c/go/+/315830/ fixes this error, and is good enough.

The good news is, I've narrowed the source a little further. Namely, if I eliminate the call to SetConsoleCtrlHandler, but keep the compileCallback invocation (printing the PC so the call stays around... also I made sure it still reproduces with everything the same, just the print added in). Doing this does not reproduce.

That means that your CL 315830 could not be a solution here. Correct?

What the commit message of that CL describes as the problem is incorrect, yes. However, it's clear that calling SetConsoleControlHandler later in initialization resolves the problem. Hence my theory about some kind of race condition.

I'm thinking now that there's, very simply (back to the very very beginning of this thread), a race between SetConsoleCtrlHandler and when the control signal lands. I do not understand what the problem could be. MSDN does not discuss any concurrency guarantees (e.g. is it safe to call SetConsoleCtrlHandler concurrently? (Note that in this case it is not being called concurrently, I just mean that as an example)).

What particular race are you thinking about? You are allowed to call SetConsoleCtrlHandler API many times - that is how the API is designed. In fact I am pretty sure our test calls SetConsoleCtrlHandler twice - Go calls it, and C runtime also calls it.

I don't have a specific one in mind, because I don't have any documentation on SetConsoleCtrlHandler's requirements. But I think it's clear to me that there's some kind of race going on. Runtime initialization is happening on a separate thread in parallel with the rest of the C code in the test. In particular, it's happening in parallel with the CTRL_BREAK landing in the process. I think there's some kind of invariant in the Windows APIs that we're violating. The sigtramp is coming from some PC that I can't identify, so my default assumption is that it's some shared library in the address space (could be wrong though).

Perhaps the problem is that Go DLL does not remove SetConsoleCtrlHandler handler before it exits? And Windows calls the handler address when Go runtime is already off.

Similarly SetConsoleCtrlHandler is supposed to handle CTRL_CLOSE_EVENT. Perhaps we get that message (I don't see how that happens) after Go DLL runtime is off, but Windows SetConsoleCtrlHandler handler still points to Go code.

Interesting, I hadn't thought about a situation where runtime initialization is complete. What do you mean by when the Go runtime is already "off"? Do you mean the rest of the process is already exiting?

I plan to play with this more on the weekend. Let's delay the decision.

Also, perhaps @zx2c4 can help debug this, if you have time. Thank you.

Alex

Thank you so much for taking the time to look at this!

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented May 9, 2021

It's not, that's very surprising. It's also surprising that you landed in badsignal2 more than once -- all the failures I got after that CL were clean. I'll try to reproduce this again.

badsignal2 calls abort at the end now. And abort starts with INT $3. And INT $3 will raise exception again (we just came from badsignal2 and sigtramp), and will call sigtramp again (I did not check that, I am just guessing).

I managed to reproduce this error just twice. But I did not have debugger ready at that time. I will try again next week.

Alex

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented May 10, 2021

Huh. I figured abort meant something closer to SIGABRT but yeah I guess I would expect INT $3 to call back into the runtime.

Should this just ExitProcess then? It's not hard to arrange, I was doing that in some of my testing (I surfaced the crashing PC and SP via the exit value).

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented May 11, 2021

Huh. I figured abort meant something closer to SIGABRT

I don't know enough about SIGABRT to comment.

but yeah I guess I would expect INT $3 to call back into the runtime.

INT $3 raises an exception on Windows. So our exception handler is called when that happens. I don't remember what happens when exception is raised when exception handler is run, but, I think, it will just call exception handler recursively.

Should this just ExitProcess then?

I think we should do that.

We should also print more exception details, if possible. See winthrow function. Exception code and exception address might help users to debug. If it is not hard to do.

Alex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
@toothrot @mknyszek @aclements @ianlancetaylor @gopherbot @alexbrainman and others