Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: TestLibraryCtrlHandler fails occasionally with register ABI enabled #45638

Closed
mknyszek opened this issue Apr 19, 2021 · 63 comments
Closed

runtime: TestLibraryCtrlHandler fails occasionally with register ABI enabled #45638

mknyszek opened this issue Apr 19, 2021 · 63 comments

Comments

@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented Apr 19, 2021

Found a failure on the windows-amd64-regabi bot:

--- FAIL: TestLibraryCtrlHandler (6.32s)
    signal_windows_test.go:205: Program exited with error: exit status 1
        FAILURE: No signal received
FAIL
FAIL	runtime	52.447s

Seems like a flake, but we should look into it.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 19, 2021

I was able to reproduce with -count=1000.

Loading

@aclements
Copy link
Member

@aclements aclements commented Apr 19, 2021

I did rewrite the ctrlHandler runtime code recently in preparation for regabi, though not specifically in a regabi way. If you can repro, it would be worth doing a check around that commit to see if that was the cause.

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 20, 2021

Looking at the build dashboard logs, it doesn't seem like this is regabi related. There's one other failure:

2021-04-16T03:16:11-cf2396c/windows-amd64-2012:
--- FAIL: TestLibraryCtrlHandler (6.17s)
    signal_windows_test.go:205: Program exited with error: exit status 1
        FAILURE: No signal received
FAIL
FAIL    runtime

The fact that there are no other failures suggests that the CL @aclements mentioned is probably the culprit (https://golang.org/cl/309632).

By the way, it's harder to reproduce than I thought.

Background on the test:

  1. It's a C program that links up to a Go DLL, but it doesn't actually call anything in the DLL.
  2. It waits to get a control signal and fails if it doesn't get one in time.
  3. It expects the control signal to be produced externally, so the actual Go test (in signal_windows_test.go) sends the signal.

Here's my working theory: LoadLibrary (in the C code) returns before the runtime is fully initialized, signaling the Go test to send its signal. Note that rt0_amd64_lib (which is used to initialize a c-shared library) spawns a thread to finish initialization asynchronously. Basically, the signalling in the test was always racy with initialization and setup of the control handler.As of https://golang.org/cl/309632, the control handler is now set up slightly later in the initialization process and it takes longer to get there too, because we now call compileCallback which is non-trivial.

As a result, the test now flakes a little bit. I will try to confirm this by adding further delay in initialization to see if the test failure reproduces more easily.

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 20, 2021

I don't think my working theory is far off from the problem, but I don't fully understand anymore why the race pertains to this test. This test just makes sure that the C process receives the signal on its handler. It doesn't care about the Go handler at all.

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 21, 2021

Print debugging isn't working because the initialization code doesn't seem to have access to stdout (or the equivalent) on Windows.

I just tried switching the timeout in the test's C code from 5 seconds to INFINITE to see if it hangs and maybe I can get it into a debuggable state.

Unfortunately when I did that I came back to my terminal filled with binary garbage followed by "fail" reported by the parent process. I suspect this problem goes a little deeper than I realize...

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 21, 2021

For a taste, my terminal looks something like this:

        $)euz���
        JZ<���������Xn��r�����������Xn��w|�����
        _r���������Xnz������������XnAM�����������Xn�����������Xn��������������Xn^ez�1o7'�"NW|�8C���@C�]���������-Xn"(H���������/Xnjno����������/Xnuyz����������/Xn
*3I^hy�����������@1Xn�*3IQhy�������� 5Xn *16T\]�������� 5Xn *16���������5Xn7Fadgw���������������5Xndgw������������6Xn8��������@7Xn\e�����������9Xn$)FMk�����������`:Xn(*��������`:Xn=Hknq����������������`:Xn������������`:Xn�����������`:Xn!��Unq�����/9J��������`:Xn����
        9J��������`:Xnnq����������`:X]Tf��������`:Xn	59��������`:Xnn�JT��������`:Xnos����������`:Xn������������� >Xn8PS����������� >Xn{���������������?Xn*0;=��������?Xn@ELQe�������������?XnLQe�������������?Xnhq{���������������?Xn04Qp���#&O��������?Xn04�����������#&+/2EO��������?Xn��#&��������?Xn������������`CXnr�����IL����r���������`CXn"R]r��������`CXn9MOU��������`CXn�������������`CXn,CFILSm�������������������@LXnfk���!L��������@LXn,�����������@LXn{�����������@LXn����!L��������@LXn%,���m���������PXn��>A��m���������PXn�����������������PXn7HJS���������PXn^oqz���������PXnkm�>As���������PXn���������������PXn���������������PXn9<@WZ]`h��������������������WXn^boty����������WXnboty���������WXnkoty��������@ZXn-=�����������^Xn�������������_Xn����/F���������_Xn%���������gXn4=Klny�����������gXnlny����������� iXnLOWa���u���������mXndlpx}��NVw���KT����'S�������������mXnpx��%*IV^cw
--- FAIL: TestLibraryCtrlHandler (15.03s)
FAIL
FAIL	runtime	306.632s
FAIL
Error running run: exit status 1

except the garbage extends far beyond what my scrollback buffer can carry. I can reproduce this with the same frequency as the original failure.

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 21, 2021

Something is very broken here. This kind of reads like something is trying to print (maybe the printf("ready\n")?) and it's memory corruption leading to a buffer overrun. There aren't very many prints at all in all these code paths. In fact, that may be the only one.

Loading

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Apr 24, 2021

I tried to reproduce this issue on my computer, but I could not.

Building programs (Go and especially C) is slow on Windows, so I modified this test to build everything once and then run executable in a loop.

diff --git a/src/runtime/signal_windows_test.go b/src/runtime/signal_windows_test.go
index d0f61fb6c2..ecacafd9d2 100644
--- a/src/runtime/signal_windows_test.go
+++ b/src/runtime/signal_windows_test.go
@@ -170,8 +170,14 @@ func TestLibraryCtrlHandler(t *testing.T) {
                t.Fatalf("failed to build c exe: %s\n%s", err, out)
        }

+       for i := 0; i < 10000; i++ {
+               testLibraryCtrlHandler(t, exe)
+       }
+}
+
+func testLibraryCtrlHandler(t *testing.T, exe string) {
        // run test program
-       cmd = exec.Command(exe)
+       cmd := exec.Command(exe)
        var stderr bytes.Buffer
        cmd.Stderr = &stderr
        outPipe, err := cmd.StdoutPipe()

And my test succeeds

c:\Users\Alex\dev\go\src>go test -count=1 -run=CtrlH runtime
ok      runtime 222.704s

c:\Users\Alex\dev\go\src>

Alex

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 26, 2021

Change https://golang.org/cl/313350 mentions this issue: runtime: add missing import "C" in TestLibraryCtrlHandler

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 26, 2021

Change https://golang.org/cl/313351 mentions this issue: runtime: replace --buildmode with -buildmode in tests

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

@alexbrainman FWIW my -count to the test binary was at about 1000 (and even then it only failed roughly every other time, so maybe once in 2000, 2500 runs?), so it's not easy to reproduce but it is possible. (EDIT: I realize now you run the test 10000 times...). Out of curiosity, what happens if you change the WaitForSingleObject from 5 seconds to INFINITE on your machine?

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

Ah, nevermind, I did not look closely enough at your code. My bad!

Huh, what version of Windows are you running? Maybe it's a bug specific to the windows-amd64-2016 builders?

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

3 consecutive runs with @alexbrainman's patch from #45638 (comment) came up clean. That's... interesting.

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

Wow, I spoke too soon. It literally JUST failed. Once out 30000 runs, roughly.

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

Interestingly, this is how it failed:

=== RUN   TestLibraryCtrlHandler
    signal_windows_test.go:210: Program exited with error: exit status 1
        runtime: signal received on thread not created by Go.
        �o�<f�N9꙾�&����vl��3U0�{\K���X�س���64oMˡ1��!
                                                    R��n��)�Џ�jG'��T��-�0�i�
                                                                            �!Ρ6�)��,p??�G�j�V���k6˩���G
                                                                                                        �
                                                                                                         �6�runtime: signal received on thread not created by Go.
        �o�<f�N9꙾�&����vl��3U0�{\K���X�س���64oMˡ1��!
                                                    R��n��)�Џ�jG'��T��-�0�i�
                                                                            �!Ρ6�)��,p??�G�j�V���k6˩���G
                                                                                                        �
                                                                                                         �6�h�h�runtime: signal received on thread not created by Go.
        �o�<f�N9꙾�&����vl��3U0�{\K���X�س���64oMˡ1��!
                                                    R��n��)�Џ�jG'��T��-�0�i�
                                                                            �!Ρ6�)��,p??�G�j�V���k6˩���G
                                                                                                        �
                                                                                                         �6�h�h�d@�?@DDD �"""C:\Windows\system32\runtime: signal received on thread not created by Go.
        �o�<f�N9꙾�&����vl��3U0�{\K���X�س���64oMˡ1��!
                                                    R��n��)�Џ�jG'��T��-�0�i�
                                                                            �!Ρ6�)��,p??�G�j�V���k6˩���G
                                                                                                        �
                                                                                                         �6�h�h�d@�?@DDD �"""C:\Windows\system32\FAILURE: No signal received
--- FAIL: TestLibraryCtrlHandler (68.32s)
FAIL
FAIL	runtime	68.350s
FAIL
Error running run: exit status 1

Again with the terminal garbage, but it's very interesting to see C:\Windows\system32\ appear amongst that garbage.

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

I have a suspicion that callbackasm is not a valid mechanism for implementing ctrlHandler. Control signals are delivered as actual signals on Windows. Something tells me it's not safe to go through the whole cgo callback path in these contexts...

The runtime: signal received on thread not created by Go. message has me a little worried in this direction, anyway.

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

Er, actually I don't see why this would be a problem. The Windows docs claim (https://docs.microsoft.com/en-us/windows/console/handlerroutine) that a new thread is created to handle these signals, so actually callbackasm seems like exactly the right mechanism. Now I'm getting the impression that something else is going wrong, raises an exception, and the exception lands on a non-Go thread?

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

Here's a theory about how the above messages are getting produced: something causes us to get into an exception that lands on a thread that has GS set (where we put TLS on Windows). That exception tries to print the badsignalmsg but as it turns out badsignallen got corrupted or something. It's otherwise wrong. So then we get this buffer overrun, garbage data is printed out after the message until we reach a point where we hit invalid/unmapped/uncommitted memory, triggering another exception on the same thread. And so on a few more times.

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

If that theory is true, then it raises three questions:

  1. What causes the signal to trigger in the first place?
  2. Is that cause related to why badsignallen is written over?
  3. How does the signal land on a "bad" thread? (Maybe this one is easy -- TLS is initialized but there's no G? Would be nice to see what that pointer actually looks like (if it's a heap address, it IS a Go thread). The specific check here is that the slot where we put a TLS pointer is set, but then where we would usually put the G is zero.)

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 26, 2021

Good news: in the current failure mode I can actually print out things when the DLL is getting initialized.

Loading

gopherbot pushed a commit that referenced this issue Apr 27, 2021
CL 211139 added TestLibraryCtrlHandler. But the CL left out import "C"
line in the test file that is supposed to be build with Cgo.

While debugging issue #45638, I discovered that the DLL built during
TestLibraryCtrlHandler does not have Dummy function. Adding import "C"
makes Dummy function appear in DLL function list.

TestLibraryCtrlHandler does not actually calls Dummy function. So I
don't see how this change affects issue #45638, but still let's make
this code correct.

Updates #45638

Change-Id: Ibab8fed29ef2ae446d0815842cf0bd040a5fb943
Reviewed-on: https://go-review.googlesource.com/c/go/+/313350
Trust: Alex Brainman <alex.brainman@gmail.com>
Run-TryBot: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
gopherbot pushed a commit that referenced this issue Apr 27, 2021
While debugging issue #45638, I discovered that some tests were using
--buildmode command line parameter instead of -buildmode.

The --buildmode parameter is handled properly by the flag package - it
is read as -buildmode. But we should correct code anyway.

Updates #45638

Change-Id: I75cf95c7d11dcdf4aeccf568b2dea77bd8942352
Reviewed-on: https://go-review.googlesource.com/c/go/+/313351
Trust: Alex Brainman <alex.brainman@gmail.com>
Run-TryBot: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Apr 28, 2021

I just run my test from #45638 (comment) on current tip. It failed once:

c:\Users\Alex\dev\go\src>go test -count=1 -run=CtrlH runtime
--- FAIL: TestLibraryCtrlHandler (179.74s)
    signal_windows_test.go:210: Program exited with error: exit status 1
        FAILURE: No signal received
FAIL
FAIL    runtime 185.257s
FAIL

and succeeded 3 times.

Out of curiosity, what happens if you change the WaitForSingleObject from 5 seconds to INFINITE on your machine?

The test still succeeds. But that gives me an idea. If you modify the test to use INFINITE in WaitForSingleObject, your C program will block on that WaitForSingleObject Windows API call indefinetly. And you should be able to attach to the process with some debugger and try to analyze program state.

Unfortunately I am not familiar with debuggers, so I cannot help you, but surly we can find help with debugging. You can use Delve debugger on Windows - I am not sure how good it is with Cgo program. Or you can use WindDbg - it is used by real Windows developers (not like me) - you should be able to do anything with it, except it does not understand Go program symbols. Or you can use gdb.

Huh, what version of Windows are you running?

C:\Users\Alex>ver

Microsoft Windows [Version 10.0.17763.1697]

C:\Users\Alex>

Maybe it's a bug specific to the windows-amd64-2016 builders?

It certainly can be. Do not forget that we are testing C program with Go DLL (built with Cgo) here. And you need gcc to build these.

Windows does not come with gcc installed. so whatever gcc you have installed on the builder is what you use. So you would have to account for the fact that this bug could be in gcc tools or runtime. And this builder, I suspect, have very old version of gcc. For example, see #35006 and #43195.

I use

c:\Users\Alex\dev\go\src>gcc --version
gcc (x86_64-win32-seh-rev0, Built by MinGW-W64 project) 8.1.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


c:\Users\Alex\dev\go\src>

here. I suspect this gcc is different from what is installed on builder. And the fact that test failed once here (see above), probably means it is not gcc bug.

Again with the terminal garbage, but it's very interesting to see C:\Windows\system32\ appear amongst that garbage.

This was, probably, generated by gcc linker, not Go linker. But otherwise, I have not comment.

Here's a theory about how the above messages are getting produced: something causes us to get into an exception that lands on a thread that has GS set (where we put TLS on Windows). That exception tries to print the badsignalmsg but as it turns out badsignallen got corrupted or something. It's otherwise wrong. So then we get this buffer overrun, garbage data is printed out after the message until we reach a point where we hit invalid/unmapped/uncommitted memory, triggering another exception on the same thread. And so on a few more times.

Unfortunately am not familiar enough with Go runtime and gcc runtime to speculate. But that sounds possible. There a 2 control handlers installed by SetConsoleCtrlHandler - one by gcc runtime and another by Go runtime. And there are 2 exception handlers installed - one by gcc runtime and another by Go runtime. So it can get messy.

Alex

Loading

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Apr 28, 2021

The test failed again just as I submitted previous message. Unfortunately this time the output is huge, so I am providing the end of it.

                                                                                                                                                                                                                                                                                    �#� �   �runtime �    ��]n        . Go cmd/compile devel go1.17-92d1afe989 Wed Apr 28 02:39:09 2021 +0000 runtime �runtime.callbackasm ��]n    ��]n    ���   � �<unspecified> "unsafe.Pointer �uintptr�1_n    %runtime._type 0��V�     �size  �    �ptrdata�    �hash ���   �tflag ���   �align ���   �fieldAlign ���   �kind ���   �equal ���   �gcdata  p�   �str (��   �ptrToThis ,��    &runtime._type �   �uint32 �
         �      �runtime.tflag�~      �uint8 ��      �func(unsafe.Pointer, unsafe.Pointer) bool���      ��   ��   �\�   &func(unsafe.Pointer, unsafe.Pointer) bool ��  �bool ��� v      !*bool K�   �J      !*uint8 ��  �@e      �runtime.nameOff ����}      �runtime.typeOff ��� �      %runtime.arraytype H�        �typ  ��   �elem 0!�   �slice 8!�   �len @�     &runtime.arraytype ��  !*runtime._type ��  ����     %runtime.chantype @�        �typ  ��   �elem 0!�   �dir 8�     &runtime.chantype >�  %runtime.functype 8�        �typ  ��   �inCount 0��   �outCount 2��    &runtime.functype ��  �uint16 �       ��      %runtime.maptype X�        �typ  ��   �key 0!�   �elem 8!�   �bucket @!�   �hasher H��   �keysize P��   �elemsize Q��   �bucketsize R��   �flags T��    &runtime.maptype ��  �func(unsafe.Pointer, uintptr) uintptr�        ��   ��   ���   &func(unsafe.Pointer, uintptr) uintptr ��  !*uintptr �    �e      %runtime.ptrtype 8� �      �typ  ��   �elem 0!�    &runtime.ptrtype 3�  %runtime.slicetype 8�        �typ  ��   �elem 0!�    &runtime.slicetype {�  %runtime.structtype P�        �typ  ��   �pkgPath 0L�   �fields 8^�    &runtime.structtype ��  %runtime.name� >�     �bytes  p�    &runtime.name &�  #[]runtime.structfield ��        ��  �array  �6   �l   �cap �    %runtime.structfield ��        �name  L�   �typ!�   �offsetAnon ��     &runtime.structfield ��  %runtime.interfacetype P� ��     �typ  ��   �pkgpath 0L�   �mhdr 8l    &runtime.interfacetype     #[]runtime.imethod ���m      �  �array  �E   �le   �cap     %runtime.imethod���      �name  ��   �ityp ���    &runtime.imethod �  %runtime.itab  �@.�     �inter     �_type!�   �hash ���   �_ �   �fun     &runtime.itab �  !*runtime.interfacetype Q  ��V      �[4]uint8 ��  ��@�      ��   � �[1]uintptr �  �@�      ��   � �int��y      $string �� �      �str  p�   �le    !*internal/cpu.option �     ��M      %internal/cpu.option  ��)�     �Name    �Feature �\�   �Specified �K�   �Enable �K�   �Required �K�    &internal/cpu.option            !*[15]internal/cpu.option �     �        �[15]internal/cpu.option �         ��� �      ��   � �runtime/internal/sys.ArchFamilyType�        �uint64�`�      !*internal/abi.RegArgs �
          �@M      %internal/abi.RegArgs ������     �Ints  �
           �Floats H�
           �Ptrs ����   �ReturnIsPtr ��t�    &internal/abi.RegArgs `
          �[9]uintptr �   H� �      ��           �[15]uint64 )
          x���      ��   � �[9]unsafe.Pointer �   H�`�      ��           �internal/abi.IntArgRegBitmap ��  �� �      ��   � &internal/abi.IntArgRegBitmap A�  �runtime.boundsErrorCode  |      �runtime.lockRank���      �int8 ����z      �runtime.statDep        �runtime.metricKind�        �runtime.gcMarkWorkerMode�`}      �runtime.gcMode�        �runtime.gcTriggerKind�        �runtime.gcDrainFlags�        �runtime.sweepClass �
                �runtime.mSpanState�}      �runtime.spanClass `�      �runtime.spanAllocType         �runtime.pallocSum� �      �runtime.bucketType �`|      �runtime.profIndex�        �runtime.profBufReadMode �        �runtime.waitReason  �      �runtime.selectDir �        �runtime.semaProfileFlags �        �uint��      �runtime.funcID �|      �runtime.funcFlag �|      �runtime.abiPartKind��{      !*runtime.stringStruct ��  ��_      %runtime.stringStruct ����      �str  �    �l    &runtime.stringStruct ��  !*runtime.itab   ���      !*unsafe.Pointer �   ��e      !*runtime.wbBuf w�  ���      %runtime.wbBuf � �     �next  �    �end�    �buf ���    &runtime.wbBuf :�  �[512]uintptr �   � �`�      ��   �� !*[2]uintptr ��  �        �[2]uintptr �   ��        ��   � �int32 ���`z      !*runtime.m V�  ��W      %runtime.m� y�     �g0  e�   �morebu)�   �divmod @��   �procid H)
           �gsignal Pe�   �goSigStack X�    �sigmask X�    �tls !   �mstartfn ��<!   �curg ��e�   �caughtsig ��<�   �p ��k�   �nextp ��k�   �oldp ��k�   �id ��Y�   �mallocing ����   �throwing ����   �preemptoff ��   �locks ����   �dying ����   �profilehz ����   �spinning ��K�   �blocked ��K�   �newSigstack ��K�   �printlock ����   �incgo ��K�   �freeWait ����   �fastrand ��H!   �needextram ��K�   �traceback ����   �ncgocall ��)
           �ncgo ����   �cgoCallersUse ����   �cgoCallers ��h!   �doesPark ��K�   �park ���!   �alllink ����   �schedlink ����   �lockedg ��<�   �createstack ���"   �lockedExt ����   �lockedInt ����   �nextwaitm ����   �waitunlockf ��f"   �waitlock ���    �waittraceev ����   �waittraceskip ��   �startingtrace ��K�   �syscalltick ����   �freelink ����   �mFixup ���"   �libcall ���#   �libcallpc ���    �libcallsp ���    �libcallg ��<�   �syscall ���#   �vdsoSP ���    �vdsoPC ���    �preemptGen ����   �signalPending ����   �dlogPerM ���#  ��mOS ��w$  ��locksHeldLen ��   �locksHeld ���$    &runtime.m ��  !*runtime.g ��  ��T      %runtime.g ����s�     �stack  S�   �stackguard0 ��    �stackguard1 ��    �_panic  f�   �_defer (��   �m 0��   �sched 8)�   �syscallsp p�    �syscallpc x�    �stktopsp ���    �param ���    �atomicstatus ����   �stackLock ����   �goid ��Y�   �schedlink ��<�   �waitsince ��Y�   �wa   �preempt ��K�   �preemptStop ��K�   �preemptShrink ��K�   �asyncSafePoint ��K�   �paniconfault ��K�   �gcscandone ��K�   �throwsplit ��K�   �activeStackChans ��K�   �parkingOnChan ����   �raceignore ����   �sysblocktraced ��K�   �tracking ��K�   �trackingSeq ����   �runnableStamp ��Y�   �runnableTime ��Y�   �sysexitticks ��Y�   �traceseq ��)
           �tracelastp ��k�   �lockedm ����   �sig ����   �writebuf ����   �sigcode0 ���    �sigcode1 ���    �sigpc ���    �gopc ���    �ancestors ����   �startpc ���    �racectx ���    �waiting ����   �cgoCtxt ����   �labels ���    �timer ����   �selectDone ����   �gcAssistBytes ��Y�    &runtime.g ~�  %runtime.stack ��@�      �lo  �    �h�     &runtime.stack %�  !*runtime._panic ��  ��O      %runtime._panic 8��D�     �argp  �    �arg4�   �link �f�   �pc  �    �sp (�    �recovered 0K�   �aborted 1K�   �goexit 2K�    &runtime._panic ��  �interface {} y�  � �       &interface {} ��  %runtime.eface ��        �_type  !�   �data�     &runtime.eface F�  !*runtime._defer Y�  �@O      %runtime._defer H�@[�     �siz  ��   �started �K�   �heap �K�   �openDefer �K�   �sp�    �pc ��    �fn �m�   �_panic  f�   �link (��   �fd 0�    �varp 8�    �framepc @�     &runtime._defer ��  !*runtime.funcval ��  �@T      %runtime.funcval���      �fn  �     &runtime.funcval ��  %runtime.gobuf 8� @�     �sp  �    �pc�    �g �<�   �ctxt ��    �ret  �    �lr (�    �bp 0�     &runtime.gobuf ��  �runtime.guintpt@�      �int64 ��z      �runtime.puintptr`�      �runtime.muintptr �      #[]uint8 ��@r      ��  �array  p�   �len   �cap     !*[]runtime.ancestorInfo ��  �@J      #[]runtime.ancestorInfo �� l      ��  �array  ��   �len   �cap �    %runtime.ancestorInfo (� �      �pcs  ��   �goid �Y�   �gopc  �     &runtime.ancestorInfo P�  #[]uintptr ���r      �   �array  ��   �len   �cap     !*runtime.sudog ��  �@`      %runtime.sudog X��_�     �g  e�   �next��   �prev ���   �elem ��    �acquiretime  Y�   �releasetime (Y�   �ticket 0��   �isSelect 4K�   �success 5K�   �parent 8��   �waitlink @��   �waittail H��   �c P��    &runtime.sudo�  !*runtime.hchan ��  ���      %runtime.hchan `�@X�     �qcount  ��   �dataqsi��   �buf ��    �elemsize ���   �closed ���   �elemtype  !�   �sendx (��   �recvx 0��   �recvq 8 �   �sendq H �   �lock XN�    &runtime.hchan      �  %runtime.waitq ��@�      �first  ��   �las��    &runtime.waitq ��  %runtime.mutex���      �lockRankStruct  ��  ��key  �     &runtime.mutex ��  %runtime.lockRankStruct  ���       &runtime.lockRankStruct a�  !*runtime.timer 8   ��a      %runtime.timer H��F�     �pp  k�   �whenY�   �period �Y�   �f �}    �arg  4�   �seq 0�    �nextwhen 8Y�   �status @��    &runtime.timer ��  �func(interface {}, uintptr)� �      �4�  ��    &func(interface {}, uintptr) K   %runtime.gsignalStack  ���       &runtime.gsignalStack �   %runtime.sigset  � �       &runtime.sigset �   �[6]uintptr �   0���      ��   � �func()� s       &func() )!  �[2]uint32 �� �`�      ��   � !*runtime.cgoCallers �!  ��Q      �runtime.cgoCallers �   �����      ��     &runtime.cgoCallers �!  %runtime.note���      �key  �     &runtime.note �!  �[32]uintptr �   ��� �      ��     �func(*runtime.g, unsafe.Pointer) bool�`�      �e�  ��   �\�   &func(*runtime.g, unsafe.Pointer) bool %"  %struct { runtime.lock runtime.mutex; runtime.used uint32; runtime.fn func(bool) bool } ����      �lock  N�   �use��   �fn �<#    �func(bool) bool���      �K�  �\�   &func(bool) bool �#  %runtime.libcall 0� :�     �fn  �    �n�    �args ��    �r1 ��    �r2  �    �err (�     &runtime.libcall Q#  %runtime.dlogPerM  ���       &runtime.dlogPerM �#  %runtime.mOS 0� ;�     �threadLock  N�   �threa�    �waitsema ��    �resumesema ��    �highResTimer  �    �preemptExtLock (��    &runtime.mOS �#  �[10]runtime.heldLockInfo �$  �����      ��
         %runtime.heldLockInfo �� �      �lockAddr  �    �ran��    &runtime.heldLockInfo �$  !*runtime.guintptr <�  � �      !*runtime.muintptr ��  ���      %noalg.struct { F uintptr; runtime..autotmp_20 *runtime.g; runtime..autotmp_21 func(*runtime.g) } ����      �.F  �    �.autotmp_2e�   �.autotmp_21 �t&    &noalg.struct { F uintptr; runtime..autotmp_20 *runtime.g; runtime..autotmp_21 func(*runtime.g) } O%  �func(*runtime.g)��y      �e�   &func(*runtime.g) R&  %noalg.struct { F uintptr; runtime..autotmp_22 *bool } �� �      �.F  �    �.autotmp_2\�    &noalg.struct { F uintptr; runtime..autotmp_22 *bool } �&  !*runtime.moduledata �)  ��X      %runtime.moduledata ��� k�     �pcHeader  �)   �funcnametab��   �cutab  �*   �filetab 8��   �pctab P��   �pclntable h��   �ftab ��0+   �findfunctab ���    �minpc ���    �maxpc ���    �text ���    �etext ���    �noptrdata ���    �enoptrdata ���    �data ���    �edata ���    �bss ���    �ebss ���    �noptrbss ���    �enoptrbss ���    �end ���    �gcdata ���    �gcbss ���    �types ���    �etypes ���    �textsectmap ���+   �typelinks ��f,   �itablinks ���,   �ptab ���,   �pluginpath �   �pkghashes ��w-   �modulename ��   �modulehashes ��w-   �hasmain ����   �gcdatamask ��|.   �gcbssmask ��|.   �typemap ���.   �bad ��K�   �next ��$'    &runtime.moduledata F'  !*runtime.pcHeader �*  ��Z      %runtime.pcHeader @��\�     �magic  ��   �pad1 ���   �pad2 ���   �minLC ���   �ptrSize ��   �nfunc   �nfiles ���   �funcnameOffset ��    �cuOffset  �    �filetabOffset (�    �pctabOffset 0�    �pclnOffset 8�     &runtime.pcHeader �*  #[]uint32 ���q      ��  �array  ,<   �le   �cap     #[]runtime.functab ���l      �+  �array  :��  �l   �cap �    %runtime.functab ��`�      �entry  �    �funcoff�     &runtime.functab u+  #[]runtime.textsect ��@p      P,  �array  ���  �len   �cap     %runtime.textsect ��     �vaddr  �    �length�    �baseaddr ��     &runtime.textsec,  #[]int32 ���j      ��  �array  1�   �len   �cap     #[]*runtime.itab ���g      ��  �array  �D   �len   �cap     #[]runtime.ptabEntry �� o      `-  �array  ���  �len   �cap     %runtime.ptabEntr�`�      �name  ��   �typ ���    &runtime.ptabEntry +-  #[]runtime.modulehash ���m      �.  �array  ��   �len   �cap �    %runtime.modulehash (� ��     �modulename    �linktimehash �   �runtimehash  /.    &runtime.modulehash �-  !*string   ��c      %runtime.bitvector �� �      �n  ��   �bytedatp�    &runtime.bitvector E.   map[runtime.typeOff]*runtime._type ��� �@�      ��  !�  !*[]*runtime.moduledata �.  �        #[]*runtime.moduledata �� h      $'  �array  eG   �len   �cap �    %noalg.struct { F uintptr; runtime.src uintptr; runtime.dst *uintptr } ����      �.F  �    �src�    �dst ���    &noalg.struct { F uintptr; runtime.src uintptr; runtime.dst *uintptr } :/  %noalg.struct { F uintptr; runtime.typ *runtime._type; runtime.src unsafe.Pointer; runtime.off uintptr; runtime.size uintptr } (��%�     �.F  �    �typ!�   �src ��    �off ��    �size  �     &noalg.struct { F uintptr; runtime.typ *runtime._type; runtime.src unsafe.Pointer; runtime.off uintptr; runtime.size uintptr } �/  !*runtime.mspan �2  ��J�     %runtime.mspan ���@e�     �next  ;1   �pre;1   �list ��2   �startAddr ��    �npages  �    �manualFreeList (A3   �freeindex 0�    �nelems 8�    �allocCache @)
           �allocBits H_3   �gcmarkBits P_3   �sweepgen X��   �divMul \��   �allocCount `��   �spanclass b�   �state c�3   �needzero d��   �elemsize h�    �limit p�    �speciallock xN�   �specials ���3    &runtime.mspan X1  !*runtime.mSpanList *3  ���      %runtime.mSpanList ����      �first  ;1   �las;1    &runtime.mSpanList �2  �runtime.gclinkptr��      !*runtime.gcBits }3  � �      �runtime.gcBits  }      %runtime.mSpanStateBox ����      �s  �    &runtime.mSpanStateBox �3  !*runtime.special ?4  ��\      %runtime.special ��`�     �next  �3   �offset��   �kind
        ��    &runtime.special �3  %runtime.heapBits ��        �bitp  p�   �shift��   �arena ��   �last �p�    &runtime.heapBits T4  �runtime.arenaId �      !*runtime.heapArena �5  ��U      %runtime.heapArena �@A�     �bitmap  �5   �spans ��5   �pageInUse ��6   �pageMarks �6   �pageSpecials ��6   �checkmarks #6   �zeroedBase�     &runtime.heapArena �4  �[131072]uint8 ��  �� �      ��   � �[512]*runtime.mspan ;1  � � �      ��   �� �[64]uint8 ��  @� �      ��   @ !*runtime.checkmarksMap x6  ��R      �runtime.checkmarksMap ��  ���� �      ��   ��� &runtime.checkmarksMap H6  !*runtime.structfield ��  �        !*runtime.arraytype
        �  �        !*runtime.structtype ��  �        !*runtime.chantype }�  �        !**runtime.hchan ��  �        !**runtime.sudog ��  �        %noalg.struct { F uintptr; runtime.c *runtime.hchan } �� �      �.F  �    ���    &noalg.struct { F uintptr; runtime.c *runtime.hchan } U7  !*runtime.mutex N�  �@Y      !*runtime.waitq  �  ���      %runtime.gList� �      �head  <�    &runtime.gList �8  !*runtime.gList C8  �@�      %noalg.struct { F uintptr; runtime.pc *uintptr; runtime.ret *string } �� �      �.F  �    �pc��   �ret �/.    &noalg.struct { F uintptr; runtime.pc *uintptr; runtime.ret *string } s8  !*runtime._func �:  ��O      %runtime._func 0�@^�     �entry  �    �nameoff��   �args ��   �deferreturn ���   �pcsp ���   �pcfile ���   �pcln ���   �npcdata  ��   �cuOffset $��   �funcID (��   �flag )1�   �_ *+:   �nfuncdata +��    &runtime._func J9  �[1]uint8 ��  ����      ��   � %runtime.funcInfo �� ��     �_func  -9  ��datap$'    &runtime.funcInfo J:  %noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.callerpc uintptr; runtime.dispatch uintptr; runtime.lockedm *bool; runtime.lockedExt *uint32 } 0��3�     �.F  �    �gpe�   �callerpc ��    �dispatch ��    �lockedm  \�   �lockedExt (,<    &noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.callerpc uintptr; runtime.dispatch uintptr; runtime.lockedm *bool; runtime.lockedExt *uint32 } �:  !*uint32 ��  ��d      !*runtime.debugCallWrapArgs �<  ��R      %runtime.debugCallWrapArgs ����      �dispatch  �    �callingGe�    &runtime.debugCallWrapArgs k<  %noalg.struct { F uintptr; runtime.ok *bool } �� �      �.F  �    �ok\�    &noalg.struct { F uintptr; runtime.ok *bool } �<  �runtime.he�        !*runtime.context 9@  �        %runtime.context �        �        �p1home  )
           �p2home)
           �p3home �)
           �p4home �)
           �p5home  )
           �p6home ()
           �contextflags 0��   �mxcsr 4��   �segcs 8��   �segds :��   �seges <��   �segfs >��   �seggs @��   �segss B��   �eflags D��   �dr0 H)
           �dr1 P)
           �dr2 X)
           �dr3 `)
           �dr6 h)
           �dr7 p)
           �rax x)
           �rcx ��)
           �rdx ��)
           �rbx ��)
           �rsp ��)
           �rbp ��)
           �rsi ��)
           �rdi ��)
           �r8 ��)
           �r9 ��)
           �r10 ��)
           �r11 ��)
           �r12 ��)
           �r13 ��)
           �r14 ��)
           �r15 ��)
           �rip ��)
           �anon0 ��N@   �vectorregister ��q@   �vectorcontrol �        )
           �debugcontrol �      )
           �lastbranchtorip �   )
           �lastbranchfromrip � )
           �lastexceptiontorip �        )
           �lastexceptionfromrip �      )
            &runtime.context �=  �[512]uint8 ��  ���        ��   �� �[26]runtime.m128a �@  ���        ��   � %runtime.m128a ��        �low  )
           �highY�    &runtime.m128a �@  #[]string �� q        �array  /.   �len   �cap �    �[6]string  `�`�      ��   � !*runtime.TypeAssertionError �A  � �      %runtime.TypeAssertionError (����     �_interface  !�   �concrete!�   �asserted �!�   �missingMethod �    &runtime.TypeAssertionError dA  $runtime.errorString �� �      �str  p�   �len    %runtime.errorAddressString �� ��     �msg    �addr ��     &runtime.errorAddressString (B  $runtime.plainError ����      �str  p�   �len    %runtime.boundsError ��@5�     �x  Y�   �y   �signed �K�   �code ���    &runtime.boundsError �B  �[100]uint8 ��  d� `w      �float64 ��w      �complex64� w      �complex128 ����v      �[32]uint8 ��   �        ��     �[9]string   �����      ��       !*[8]uint8 �D  �        �[8]uint8 ���@�      ��   !*[4]uint8  �        !*runtime.itabTableType �D  � �      %runtime.itabTableType � ����     �size  �    �count�    �entries ��D    &runtime.itabTableType kD  �[512]*runtime.itab ��  � ���      ��   �� !**runtime.itab ��  �        %noalg.struct { F uintptr; R *runtime.itabTableType } ����      �F  �    �RFD    &noalg.struct { F uintptr; R *runtime.itabTableType } �E  !*runtime.imethod �  �@V      !*runtime.uncommontype JF  �        %runtime.uncommontype ��        �pkgpath  ��   �mcount ���   �xcount ���   �moff��   �_ ��    &runtime.uncommontype �E  #[]runtime.method ��        �F  �array  ���  �l   �cap     %runtime.method ��        �name  ��   �mtyp ���   �ifn�G   �tfn �G    &runtime.method �F  �runtime.textOff ���        #[]unsafe.Pointer ���r      �   �array  ��   �len   �cap �    !**runtime.moduledata $'  �        �func(*runtime.itab)�        ���   &func(*runtime.itab) �G  !*runtime.lfstack �G  �        �runtime.lfstack�        !*runtime.lfnode UH  ��V      %runtime.lfnode ��@�      �next  )
           �pushcnt�     &runtime.lfnode �H  !*runtime.note �!  � Z      !*runtime.arenaHint �H  ��P      %runtime.arenaHint ����      �addr  �    �downK�   �next ��H    &runtime.arenaHint �H  !*runtime.fixalloc �I  ���      %runtime.fixalloc H��K�     �size  �    �firs�I   �arg ��    �list �'J   �chunk  �    �nchunk (��   �inuse 0�    �stat 8}J   �zero @K�    &runtime.fixalloc �I  �func(unsafe.Pointer, unsafe.Pointer�@�      ��   ��    &func(unsafe.Pointer, unsafe.Pointer) �I  !*runtime.mlink jJ  ��X      %runtime.mlink� �      �next  'J    &runtime.mlink DJ  !*runtime.sysMemStat �J  � �      �runtime.sysMemStat��~      !**runtime.heapArena �4  �        !**[1048576]*runtime.heapArena K  �        !*[1048576]*runtime.heapArena 7K  ��I      �[1048576]*runtime.heapArena �4  �������      ��   ��@ !*runtime.mheap FN  ��S�     %runtime.mheap �����g�     �lock  N�   �page�O   �sweepgen �����   �sweepDrained �����   �sweepers �����   �allspans ����S   �_ Ђ���   �pagesInUse ؂�)
           �pagesSwept ���)
           �pagesSweptBasis ���)
           �sweepHeapLiveBasis ���)
           �sweepPagesPerByte ���uC   �scavengeGoal ���)
           �reclaimIndex ���)
           �reclaimCredit ����    �arenas ����S   �heapArenaAlloc ���XT   �arenaHints ����H   �arena ���XT   �allArenas ���qT   �sweepArenas ���qT   �markArenas ���qT   �curArena ����T   �_ �����   �central ����U   �spanalloc ����I   �cachealloc ����I   �specialfinalizeralloc ����I   �specialprofilealloc ����I   �specialReachableAlloc ����I   �speciallock ���N�   �arenaHintAlloc ����I   �unused ���%W    &runtime.mheap �K  %runtime.pageAlloc ����@N�     �summary  ,O   �chunks x�O   �searchAddr ���1Q   �start ���FQ   �end ���FQ   �inUse ����Q   �scav ���^R   �mheapLock ����7   �sysStat ���}J   �test ���K�    &runtime.pageAlloc YN  �[5][]runtime.pallocSum YO  x���      ��   � #[  �array  ψ   �len   �cap     �[8192]*[8192]runtime.pallocData �O  ���� �      ��   �@ !*[8192]runtime.pallocData �P  ��I      �[8192]runtime.pallocData vP  ��@���      ��   �@ %runtime.pallocData ��� �      �pallocBits  �P  ��scavenged @�P    &runtime.pallocData 3P  �runtime.pallocBits )
          @���      ��   &runtime.pallocBits �P  �runtime.pageBits )
          @���      ��   &runtime.pageBits �P  %runtime.offAdd� 8�     �a  �     &runtime.offAddr Q  �runtime.chunkIdx`�      %runtime.addrRanges (�`�      �ranges  �Q   �totalBytes ��    �sysStat  }J    &runtime.addrRanges cQ  #[]runtime.addrRange ���k      GR  �array  ��   �len   �cap     %runtime.addrRange ��`&�     �base  1Q   �limi1Q    &runtime.addrRange �R  %struct { runtime.inUse runtime.addrRanges; runtime.gen uint32; runtime.reservationBytes uintptr; runtime.released uintptr; runtime.scavLWM runtime.offAddr; runtime.freeHWM runtime.offAddr } P� 6�     �inUse  �Q   �gen (��   �reservationBytes 0�    �released 8�    �scavLWM @1Q   �freeHWM H1Q    #[]*runtime.mspan ��@h      ;1  �array  ��   �l   �cap     �[64]*[1048576]*runtime.heapArena K  ���@�      ��   @ %runtime.linearAlloc  ����     �next  �    �mappe�    �end ��    �mapMemory �K�    &runtime.linearAlloc �T  #[]runtime.arenaIdx ��@l      �4  �array  �k   �len   �cap     %struct { runtime.base uintptr; runtime.end uintptr } �� �      �base  �    �en�     �[136]struct { runtime.mcentral runtime.mcentral; runtime.pad [24]uint8 } qU  ������      ��   �� %struct { runtime.mcentral runtime.mcentral; runtime.pad [24]uint8 } �����      �mcentral  (V   �pad ���W    %runtime.mcentral ���`��     �spanclass  �   �partial>V   �full X>V    &runtime.mcentral �U  �[2]runtime.spanSet �V  P�@�      ��   � %runtime.spanSet (� 0�     �spineLock  N�   �spine�    �spineLen ��    �spineCap ��    �index  �V    &runtime.spanSet gV  �runtime.headTailInde� �      �[24]uint8 ��  ����      ��   � !*runtime.specialfinalizer �W  � ]      %runtime.specialfinalizer 0��0�     �special  ?4   �fn �m�   �nret ��    �fint  !�   �ot (�W    &runtime.specialfinalizer MW  !*runtime.ptrtype f�  � \      !*runtime.notInHeap )X  ���      %runtime.notInHeap  �@�       &runtime.notInHeap �X  !*runtime.mcache �X  �@�      %runtime.mcache �     ��E�     �nextSample  �    �scanAlloc�    �tiny ��    �tinyoffset ��    �tinyAllocs  �    �alloc (�Y   �stackcache >Y   �flushGen �     ��    &runtime.mcache ^X  �[136]*runtime.mspan ;1  �`�      ��   �� �[2]runtime.stackfreelist �Y   ���      ��   � %runtime.stackfreelist �� �      �list  A3   �size�     &runtime.stackfreelist mY  %runtime.gcTrigger ��        �kind  c   �nowY�   �n ���    &runtime.gcTrigger �Y  !*runtime.p q]  � �      %runtime.p �N�@o�     �id  ��   �status ���   �link�   �schedtick ���   �syscalltick ���   �sysmontick ��]   �m 8��   �mcache @@X   �pcache HB^   �raceprocctx `�    �deferpool hY^   �deferpoolbuf ���^   �goidcache ��)
           �goidcacheend ��)
           �runqhead ����   �runqtail ����   �runq ��"_   �runnext ��<�   �gFree ��P_   �sudogcache ���_   �sudogbuf ���_   �mspancache �$`   �tracebuf �,�`   �traceSweep �,K�   �traceSwept �,�    �traceReclaimed �,�    �palloc �,�`   �_ �,��   �timer0When �,)
           �timerModifiedEarliest �,)
           �gcAssistTime �,Y�   �gcFractionalMarkTime �-Y�   �gcMarkWorkerMode �-#   �gcMarkWorkerStartTime �-Y�   �gcw �-za   �wbBuf �-w�   �runSafePointFn �M��   �statsSeq �M��   �timersLock �MN�   �timers �Mpb   �numTimers �M��   �adjustTimers �M��   �deletedTimers �N��   �timerRaceCtx �N�    �preempt �NK�    &runtime.p 0Z  %runtime.sysmontick  ����     �schedtick  ��   �schedwheY�   �syscalltick ���   �syscallwhen �Y�    &runtime.sysmontick �]  %runtime.pageCache �����     �base  �    �cache)
           �scav �)
            &runtime.pageCache �]  �[5][]*runtime._defer �^  x� �      ��   � #[]*runtime._defer ���f      ��  �array  5�   �len   �cap �    �[5][32]*runtime._defer �^  �
        ���      ��   � �[32]*runtime._defer ��  �����      ��     �[256]runtime.guintptr <�  �����      ��   �� %struct { runtime.gList; runtime.n int32 } ����      �gList  C8  ����    #[]*runtime.sudog ���h      ��  �array  77   �l   �cap     �[128]*runtime.sudog ��  ���      ��   �� %struct { runtime.len int; runtime.buf [128]*runtime.mspan } � �      �len    �bufk`    �[128]*runtime.mspan ;1  �@�      ��   �� �runtime.traceBufPtr��      %runtime.persistentAlloc ����      �base  �W   �of�     &runtime.persistentAlloc �`  %runtime.gcWork (�`-�     �wbuf1  �a   �wbuf2�a   �bytesMarked �)
           �scanWork �Y�   �flushedWork  K�    &runtime.gcWork �a  !*runtime.workbuf �a  � �      %runtime.workbuf �����      �workbufhdr  3b  ��obj �Kb    &runtime.workbuf �a  %runtime.workbufhdr �� �      �node  UH   �nobj �    &runtime.workbufhdr �a  �[253]uintptr �   ���`�      ��   �� #[]*runtime.timer ���h      ��  �array  ��   �l   �cap �    %noalg.struct { F uintptr; runtime.size uintptr; runtime.align uintptr; runtime.sysStat *runtime.sysMemStat; runtime.p **runtime.notInHeap } (��#�     �.F  �    �size�    �align ��    �sysStat �}J   �p  �d    &noalg.struct { F uintptr; runtime.size uintptr; runtime.align uintptr; runtime.sysStat *runtime.sysMemStat; runtime.p **runtime.notInHeap } �b  !**runtime.notInHeap �W  ��H      !*runtime.persistentAlloc �`  ��[      !*runtime.linearAlloc XT  ���      !*runtime.hmap 9e  ����     %runtime.hmap 0� M�     �count     �flag��   �B    ��   �noverflow
        ��   �hash0 ��   �buckets ��    �oldbuckets ��    �nevacuate  �    �extra (Ke    &runtime.hmap �d  !*runtime.mapextra �e  �@X      %runtime.mapextra �����     �overflow  �e   �oldoverflow�e   �nextOverflow �7f    &runtime.mapextra ke  !*[]*runtime.bmap �e  � J      #[]*runtime.bmap ��@g      7f  �array  ���  �l   �cap �    !*runtime.bmap {f  � �      %runtime.bmap� �      �tophash  �D    &runtime.bmap Sf  !*runtime.maptype ��  �        �[2]runtime.evacDst �g  @���      ��   � %runtime.evacDst  ����     �b  7f   �i   �k ��    �e ��     &runtime.evacDst �f  !*internal/abi.IntArgRegBitmap t�  � �      !*runtime.mSpanStateBox �3  � �      �[40]uint8 ��  (�        ��   ( %noalg.struct { F uintptr; runtime.c **runtime.mcache } ����      �.F  �    �c-h    &noalg.struct { F uintptr; runtime.c **runtime.mcache } �g  !**runtime.mcache @X  �@H      %noalg.struct { F uintptr; runtime.c *runtime.mcache } ��        �.F  �    �c@X    &noalg.struct { F uintptr; runtime.c *runtime.mcache } Lh  �[1]uint64 )
         �        ��   � !*runtime.heapStatsDelta Nj  �        %runtime.heapStatsDelta �    �        �committed  Y�   �releaseY�   �inHeap �Y�   �inStacks �Y�   �inWorkBufs  Y�   �inPtrScalarBits (Y�   �tinyAllocCount 0�    �largeAlloc 8�    �largeAllocCount @�    �smallAllocCount Hjj   �largeFree ���    �largeFreeCount ���    �smallFreeCount ��jj   �_ �    �j    &runtime.heapStatsDelta "i  �[68]uintptr �   ���        ��   D �[0]uint32 ��   �        ��     !*runtime.mcentral (V  �@ �     %runtime.sweepLocker�        �sweepGen  ��   �blocking �K�    &runtime.sweepLocker �j  %runtime.sweepLocked�       k  �        !*runtime.arenaIdx �4  ���      %runtime.markBits �� 7�     �bytep  p�   �mask��   �index ��     &runtime.markBits �k  !*runtime.finblock �l  �        %runtime.finblock ���        �alllinkl   �nexl   �cnt ���   �_ ���   �fin ��l    &runtime.finblock (l  �[101]runtime.finalizer �m  ���        ��   e %runtime.finalizer (�        �fn  m�   �arg�    �nret ��    �fint �!�   �ot  �W    &runtime.finalizer �l  %noalg.struct { F uintptr; runtime.now *int64 } ����      �.F  �    �now�m    &noalg.struct { F uintptr; runtime.now *int64 } 0m  !*int64 Y�  ��L      !*runtime.timeHistogram 1n  �        %runtime.timeHistogram �-�        �counts  Ln   �underflow �-)
            &runtime.timeHistogram �m  �[720]uint64 )
          �-�        ��   �� %noalg.struct { F uintptr; runtime.restart *bool } ����      �.F  �    �restart\�    &noalg.struct { F uintptr; runtime.restart *bool } pn  !**runtime.p �Z  �        !*runtime.gcWork za  �@'�     �[3]int64 Y�  ��        ��   � �[5]int64 Y�  (�        ��   � %noalg.struct { F uintptr; runtime.startTime int64 } ��        �.F  �    �startTimeY�    &noalg.struct { F uintptr; runtime.startTime int64 } to  %noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.pp *runtime.p } �� �      �.F  �    �gpe�   �pp ��Z    &noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.pp *runtime.p }p  !*runtime.gcBgMarkWorkerNode 9q  ��T      %runtime.gcBgMarkWorkerNode  �`��     �node  UH   �gp �<�   �m ���    &runtime.gcBgMarkWorkerNode �p  !*runtime.gQueue �q  �        %runtime.gQueue ��        �head  <�   �tai<�    &runtime.gQueue wq  !*runtime.sweepClass �  �        %noalg.struct { F uintptr; runtime.i *int } ����      �.F  �    �iZr    &noalg.struct { F uintptr; runtime.i *int } �q  !*int   � L      %noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.gcw *runtime.gcWork } ��`�      �.F  �    �gpe�   �gcw ��o    &noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.gcw *runtime.gcWork } mr  %noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.scanWork int64 } ����      �.F  �    �gpe�   �scanWork �Y�    &noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.scanWork int64 } ;s  %noalg.struct { F uintptr; runtime.state *runtime.stackScanState; runtime.gcw *runtime.gcWork } �� �      �.F  �    �state�t   �gcw ��o    &noalg.struct { F uintptr; runtime.state *runtime.stackScanState; runtime.gcw *runtime.gcWork } �t  !*runtime.stackScanState �u  ���      %runtime.stackScanState ����O�     �cache  �v   �stack ��S�   �conservative ��K�   �buf ���w   �freeBuf ���w   �cbuf ���w   �head x   �tail x   �nobjs �   �root ���y    &runtime.stackScanState �u  %runtime.pcvalueCache ��� �      �entries  4v    &runtime.pcvalueCache �u  �[2][8]runtime.pcvalueCacheEnt iv  �����      ��   � �[8]runtime.pcvalueCacheEnt �v  �����      ��  %runtime.pcvalueCacheEnt �����     �targetpc  �    �of��   �val ��    &runtime.pcvalueCacheEnt �v  !*runtime.stackWorkBuf jw  �@^      %runtime.stackWorkBuf �����      �stackWorkBufHdr  �w  ��obj  �w    &runtime.stackWorkBuf &w  %runtime.stackWorkBufHdr  �`�      �workbufhdr  3b  ��next ��w    &runtime.stackWorkBufHdr �w  �[252]uintptr �   ��� �      ��   �� !*runtime.stackObjectBuf vx  ��]      %runtime.stackObjectBuf �����      �stackObjectBufHdr  �x  ��obj  �x    &runtime.stackObjectBuf .x  %runtime.stackObjectBufHdr  ���      �workbufhdr  3b  ��next x    &runtime.stackObjectBufHdr �x  �[63]runtime.stackObject �y  �����      ��   ? %runtime.stackObject  ��1�     �off  ��   �size ���   �typ!�   �left ��y   �right ��y    &runtime.stackObject $y  !*runtime.stackObject �y  �@�      �func(*runtime.stkframe, unsafe.Pointer) bool�        �5z  ��   �\�   &func(*runtime.stkframe, unsafe.Pointer) bool �y  !*runtime.stkframe �z  �@_      %runtime.stkframe X� Q�     �fn  �:   �pc ��    �continpc ��    �lr  �    �sp (�    �fp 0�    �varp 8�    �argp @�    �arglen H�    �argmap P�z    &runtime.stkframe Uz  !*runtime.bitvector |.  � �      !*runtime.stackObjectRecord �{  � ^      %runtime.stackObjectRecord �� �      �off     �ty!�    &runtime.stackObjectRecord H{  #[]runtime.stackObjectRecord ���o      �{  �array  �{   �len   �cap �    �func() bool�`x      �\�   &func() bool �{  !*runtime.gcControllerState ,~  �        %runtime.gcControllerState ���        �gcPercent  ��   �_ ���   �heapMinimum)
           �triggerRatio �uC   �trigger �)
           �heapGoal  )
           �lastHeapGoal ()
           �heapLive 0)
           �heapScan 8)
           �heapMarked @)
           �scanWork HY�   �bgScanCredit PY�   �assistTime XY�   �dedicatedMarkTime `Y�   �fractionalMarkTime hY�   �idleMarkTime pY�   �markStartTime xY�   �dedicatedMarkWorkersNeeded ��Y�   �assistWorkPerByte ��)
           �assistBytesPerWork ��)
           �fractionalUtilizationGoal ��uC   �_ ��z~    &runtime.gcControllerState I|  %internal/cpu.CacheLinePad @�        �_  �6    &internal/cpu.CacheLinePad K~  %noalg.struct { F uintptr; runtime.released *uintptr; runtime.crit *float64 } �� �      �.F  �    �released��   �crit �j�    &noalg.struct { F uintptr; runtime.released *uintptr; runtime.crit *float64 } �~  !*float64 uC  ��K      !*runtime.pageAlloc �O  ��Y�     !*runtime.addrRanges �Q  �`��     %noalg.struct { F uintptr; runtime.p *runtime.pageAlloc; runtime.minPages uintptr } ����      �.F  �    �p��   �minPages ��     &noalg.struct { F uintptr; runtime.p *runtime.pageAlloc; runtime.minPages uintptr } ��  �func(runtime.addrRange) (runtime.chunkIdx, bool)�        �GR  � �  �\�   &func(runtime.addrRange) (runtime.chunkIdx, bool) ��  !*runtime.chunkIdx FQ   ��      !*runtime.pallocData vP  ��C�     !**runtime.stackWorkBuf �w  �@I      �[2]**runtime.stackWorkBuf b�  �� �      ��   � %runtime.specialsIter ����      �pprev �   �s�3    &runtime.specialsIter ��  !**runtime.special �3  � I      !*runtime.sweepLocked Sk  �        !*runtime.specialsIter ��  � �      !*runtime.markBits �k  ���      %noalg.struct { F uintptr; runtime.s **runtime.mspan } ����      �.F  �    �s��    &noalg.struct { F uintptr; runtime.s **runtime.mspan } ��  !**runtime.mspan ;1  ��H      %noalg.struct { F uintptr; runtime.preemptible bool } ��        �.F  �    �preemptibleK�    &noalg.struct { F uintptr; runtime.preemptible bool } =�  %noalg.struct { F uintptr; runtime.h *runtime.mheap; runtime.npages uintptr; runtime.spanclass runtime.spanClass; runtime.s **runtime.mspan } (� "�     �.F  �    �hnK   �npages ��    �spanclass ��   �s  ��    &noalg.struct { F uintptr; runtime.h *runtime.mheap; runtime.npages uintptr; runtime.spanclass runtime.spanClass; runtime.s **runtime.mspan } Ճ  !*runtime.mSpanState �  � X      !*runtime.pageCache B^  ���      %noalg.struct { F uintptr; runtime.h *runtime.mheap; runtime.s *runtime.mspan } ��        �.F  �    �hnK   �s �;1    &noalg.struct { F uintptr; runtime.h *runtime.mheap; runtime.s *runtime.mspan } ~�  !*runtime.specialprofile ��  �        %runtime.specialprofile ��        �special  ?4   �b �Ɔ    &runtime.specialprofile o�  !*runtime.bucket I�  ���      %runtime.buc   �hash ��    �size  �    �nstk (�     &runtime.bucket ��  !*runtime.specialReachable և  �        %runtime.specialReachable ��        �special  ?4   �done �K�   �reachable �K�    &runtime.specialReachable ��  !*runtime.gcBitsArena ]�  �        %runtime.gcBitsArena ����        �free  �    �next��   �bits �v�    &runtime.gcBitsArena ��  �[65520]runtime.gcBits }3  ����        ��   ���      %noalg.struct { F uintptr; runtime.firstFree *struct { runtime.base runtime.offAddr; runtime.bound runtime.offAddr } } ����      �.F  �    �firstFre�    &noalg.struct { F uintptr; runtime.firstFree *struct { runtime.base runtime.offAddr; runtime.bound runtime.offAddr } } ��  !*struct { runtime.base runtime.offAddr; runtime.bound runtime.offAddr } ^�  ��c      %struct { runtime.base runtime.offAddr; runtime.bound runtime.offAddr } ����      �base  1Q   �bound1Q    �func(runtime.offAddr, uintptr)�        �1Q  ��    &func(runtime.offAddr, uintptr) ʊ  !*runtime.pallocBits �P  � <�     !*runtime.pageBits �P  � )�     �[5]uint ��  (�        ��   � %runtime.notInHeapSlice ��        �array  �W   �l   �cap     &runtime.notInHeapSlice ��  %noalg.struct { F uintptr; runtime.addrRangeToSummaryRange func(int, runtime.addrRange) (int, int); runtime.summaryRangeToSumAddrRange func(int, int, int) runtime.addrRange } �� �      �.F  �    �addrRangeToSummaryRange��   �summaryRangeToSumAddrRange �X�    &noalg.struct { F uintptr; runtime.addrRangeToSummaryRange func(int, runtime.addrRange) (int, int); runtime.summaryRangeToSumAddrRange func(int, int, int) runtime.addrRange } ��  �func(int, runtime.addrRange) (int, int)�`�      �  �GR  �Zr  �Zr   &func(int, runtime.addrRange) (int, int) ��  �func(int, int, int) runtime.addrRang� �      �  �  �  ���   &func(int, int, int) runtime.addrRange ��  !*runtime.addrRange GR   ��      %noalg.struct { F uintptr; runtime.p *runtime.pageAlloc } ����      �.F  �    ���    &noalg.struct { F uintptr; runtime.p *runtime.pageAlloc } ��  �func(int, runtime.addrRange) runtime.addrRange�        �  �GR  ���   &func(int, runtime.addrRange) runtime.addrRange :�  !*[32]uintptr �"  �        !*runtime.memRecord /�  �        %runtime.memRecord ���        �active  ��   �future  Ő    &runtime.memRecord ��  %runtime.memRecordCycle  �        �allocs  �    �free�    �alloc_bytes ��    �free_bytes ��     &runtime.memRecordCycle F�  �[3]runtime.memRecordCycle ��  `�        ��   � !*runtime.memRecordCycle ��  �        %noalg.struct { F uintptr; runtime.p unsafe.Pointer; runtime.b *runtime.bucket } ����      �.F  �    �p�    �b �Ɔ    &noalg.struct { F uintptr; runtime.p unsafe.Pointer; runtime.b *runtime.bucket } ��  %noalg.struct { F uintptr; runtime.pc uintptr; runtime.sp uintptr; runtime.gp *runtime.g }  �`��     �.F  �    �pc�    �sp ��    �gp �e�    &noalg.struct { F uintptr; runtime.pc uintptr; runtime.sp uintptr; runtime.gp *runtime.g } ��  !*runtime.spanSetBlock D�  �        %runtime.spanSetBlock � �        �lfnode  UH  ��popped ���   �spans ��5    &runtime.spanSetBlock ��  !*runtime.spanSet �V  ���      !*runtime.headTailIndex �V  � (�     !*runtime.spanSetBlockAlloc ��  �        %runtime.spanSetBlockAlloc�        �stack  �G    &runtime.spanSetBlockAlloc ˓  !**runtime.spanSetBlock Ւ  �        !*runtime.consistentHeapStats ��  �        %runtime.consistentHeapStats ���        �stats  ��   �gen ����   �noPLock ��N�    &runtime.consistentHeapStats m�  �[3]runtime.heapStatsDelta Nj  ���        ��   � !*runtime.pollDesc ��  ���      %runtime.pollDesc ����a�     �link  ��   �locN�   �fd ��    �closing �K�   �everr �K�   �user ���   �rseq  �    �rg (�    �rt 08    �rd xY�   �wseq ���    �wg ���    �wt ��8    �wd ��Y�   �self ����    &runtime.pollDesc 1�  !*runtime.net_op ��  ��Y      %runtime.net_op 8� /�     �o  ��   �pd  ��   �mode (��   �errno ,��   �qty 0��    &runtime.net_op 2�  %runtime.overlapped  �`��     �internal  �    �internalhigh�    �anon0 ��D   �hevent �p�    &runtime.overlapped ��  �[64]runtime.overlappedEntry ��  �����      ��   @ %runtime.overlappedEntry  � ��     �key  �    �op��   �internal ��    �qty ���    &runtime.overlappedEntry A�  �[10]uint8 ��
        �        ��
 �[14]uint8 ��  ��        ��   � �[15]uint8 ��  ��        ��   � �[16]uint8 ��  ��        ��   � �[17]uint8 ��  ��        ��   � �[18]uint8 ��  ��        ��   � �[22]uint8 ��  ��        ��   � �[23]uint8 ��  ��        ��   � �[27]uint8 ��  ��        ��   � �[39]uint8 ��  '�        ��   ' %runtime._DEVICE_NOTIFY_SUBSCRIBE_PARAMETERS·2 ��        �callback  �    �contex�     &runtime._DEVICE_NOTIFY_SUBSCRIBE_PARAMETERS·2 2�  "runtime.stdFunction %runtime.systeminfo 0�`R�     �anon0    �dwpagesize ���   �lpminimumapplicationaddressp�   �lpmaximumapplicationaddress �p�   �dwactiveprocessormask ��    �dwnumberofprocessors  ��   �dwprocessortype $��   �dwallocationgranularity (��   �wprocessorlevel ,��   �wprocessorrevision .��    &runtime.systeminfo ՙ  �[26]uint8 ��  ��        ��   � !*int32 ��  ��L      !*uint16 ��  ��d      #[]uint16 ��        ��  �array  F�   �len   �cap �    %noalg.struct { F uintptr; runtime.result uintptr } ��        �.F  �    �result�     &noalg.struct { F uintptr; runtime.result uintptr } ��  %runtime.memoryBasicInformation 0�        �baseAddress  �    �allocationBase�    �allocationProtect ���   �regionSize ��    �state  ��   �protect $��   �type_ (��    &runtime.memoryBasicInformation '�  %noalg.struct { F uintptr; runtime.us uint32 } ��        �.F  �    �us��    &noalg.struct { F uintptr; runtime.us uint32 } ��  �[1247]uint8 ��  �  �        ��   �  �[5]int32 ��  ��        ��   � %noalg.struct { F uintptr; runtime.pp *runtime.p; runtime.sc uintptr } ����      �.F  �    �p�Z   �sc ��     &noalg.struct { F uintptr; runtime.pp *runtime.p; runtime.sc uintptr } ��  %noalg.struct { F uintptr; runtime.siz int32; runtime.d **runtime._defer } ����      �.F  �    �si��   �d �5�    &noalg.struct { F uintptr; runtime.siz int32; runtime.d **runtime._defer } r�  !**runtime._defer ��  ��G      !*[]*runtime._defer �^  �        �error Ɵ  ���       &error u�  %runtime.iface ��        �tab  ��   �dat�     &runtime.iface ��  �runtime.stringer Ɵ  � �       &runtime.stringer ٟ  %noalg.struct { F uintptr; runtime.pc uintptr; runtime.sp unsafe.Pointer; runtime.gp *runtime.g; runtime.prevDefer *runtime._defer } (��"�     �.F  �    �pc�    �sp ��    �gp �e�   �prevDefer  ��    &noalg.struct { F uintptr; runtime.pc uintptr; runtime.sp unsafe.Pointer; runtime.gp *runtime.g; runtime.prevDefer *runtime._defer } ��  %noalg.struct { F uintptr; runtime.prevDefer *runtime._defer; runtime.gp *runtime.g } ��`�      �.F  �    �prevDefe��   �gp �e�    &noalg.struct { F uintptr; runtime.prevDefer *runtime._defer; runtime.gp *runtime.g } a�  !**runtime.funcval m�  ��G      !**runtime._panic f�  �        %noalg.struct { F uintptr; runtime.s string } �� �      �.F  �        &noalg.struct { F uintptr; runtime.s string } ��  %noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.pc uintptr; runtime.sp uintptr }  � ��     �.F  �    �ge�   �pc ��    �sp ��     &noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.pc uintptr; runtime.sp uintptr } ��  %noalg.struct { F uintptr; runtime.msgs *runtime._panic; runtime.gp *runtime.g; runtime.pc uintptr; runtime.sp uintptr; runtime.docrash *bool } 0�`4�     �.F  �    �msgf�   �gp �e�   �pc ��    �sp  �    �docrash (\�    &noalg.struct { F uintptr; runtime.msgs *runtime._panic; runtime.gp *runtime.g; runtime.pc uintptr; runtime.sp uintptr; runtime.docrash *bool } ��  %runtime.suspendGState ��        �g  e�   �deaK�   �stopped    K�    &runtime.suspendGState ]�  !*[1048576]runtime.inlinedCall ��  �        �[1048576]runtime.inlinedCall ��  ���
        �        ��   ��@ %runtime.inlinedCall ��        �parent  ��   �funcID ���   �_ ���   �file ���   �lin��   �func_ ��   �parentPc ���    &runtime.inlinedCall "�  �int16 ��� z      !*runtime.slice !�  �        %runtime.slice ��        �array  �    �l   �cap �    &runtime.slice ��  �func(uintptr) uint8�        ��   �p�   &func(uintptr) uint8 4�  %noalg.struct { F uintptr; runtime.needUnlock *bool } �� �      �.F  �    �needUnloc\�    &noalg.struct { F uintptr; runtime.needUnlock *bool } w�  %noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.traceskip int } ��@�      �.F  �    �gpe�   �traceskip     &noalg.struct { F uintptr; runtime.gp *runtime.g; runtime.traceskip int } ��  !*[]*runtime.sudog �_  �        !**runtime.g e�  �        %noalg.struct { F uintptr; runtime.oldval uint32; runtime.newval uint32 } ��        �.F  �    �oldva��   �newval ��    &noalg.struct { F uintptr; runtime.oldval uint32; runtime.newval uint32 } ��  !*runtime.puintptr k�  � �      !**runtime.m ��  � H      �func(*runtime.p�        ��Z   &func(*runtime.p) ��  %noalg.struct { F uintptr; runtime.freem **runtime.m } �� �      �.F  �    �freem��    &noalg.struct { F uintptr; runtime.freem **runtime.m } K�  %runtime.cgothreadstart ��� �     �g  <�   �tls<�   �fn ��     &runtime.cgothreadstart ߪ  !*uint64 )
          � e      #runtime.pMask ��        ��  �array  ,<   �l   �cap     &runtime.pMask R�  #[]*runtime.p ��        �Z  �array  �n   �len   �cap     %runtime.randomEnum ��        �i  ��   �count ���   �po��   �inc ��    &runtime.randomEnum ��  !*runtime.randomOrder ��  �        %runtime.randomOrder  �        �count  ��   �coprimes�*    &runtime.randomOrder l�  !*runtime.randomEnum 1�  �        %noalg.struct { F uintptr; runtime._g_ *runtime.g } ����      �.F  �    �_g_e�    &noalg.struct { F uintptr; runtime._g_ *runtime.g } ��  %noalg.struct { F uintptr; runtime.sp uintptr; runtime._g_ *runtime.g } ��@�      �.F  �    �sp�    �_g_ �e�    &noalg.struct { F uintptr; runtime.sp uintptr; runtime._g_ *runtime.g } p�  %noalg.struct { F uintptr; runtime.sp1 uintptr; runtime.sp2 uintptr; runtime.sp3 uintptr; runtime._g_ *runtime.g } (��$�     �.F  �    �sp1�    �sp2 ��    �sp3 ��    �_g_  e�    &noalg.struct { F uintptr; runtime.sp1 uintptr; runtime.sp2 uintptr; runtime.sp3 uintptr; runtime._g_ *runtime.g } .�  %noalg.struct { F uintptr; runtime.ok *bool; runtime.oldp *runtime.p; runtime._g_ *runtime.g }  ����     �.F  �    �ok\�   �oldp ��Z   �_g_ �e�    &noalg.struct { F uintptr; runtime.ok *bool; runtime.oldp *runtime.p; runtime._g_ *runtime.g } Y�  %noalg.struct { F uintptr; runtime.stacksize int32; runtime.newg *runtime.g } ����      �.F  �    �stacksize��   �newg �e�    &noalg.struct { F uintptr; runtime.stacksize int32; runtime.newg *runtime.g } Q�  %noalg.struct { F uintptr; runtime.fn **runtime.funcval; runtime.argp unsafe.Pointer; runtime.siz int32; runtime.gp *runtime.g; runtime.pc uintptr } 0��2�     �.F  �    �fnA�   �argp ��    �siz ���   �gp  e�   �pc (�     &noalg.struct { F uintptr; runtime.fn **runtime.funcval; runtime.argp unsafe.Pointer; runtime.siz int32; runtime.gp *runtime.g; runtime.pc uintptr } #�  !*runtime.gobuf )�  �@U      !*runtime.ancestorInfo ��  ��P      �[100]uintptr �   ���        ��   d %noalg.struct { F uintptr; runtime.gp *runtime.g } ����      �.F  �    �gpe�    &noalg.struct { F uintptr; runtime.gp *runtime.g }  �  %noalg.struct { F uintptr; runtime.pp *runtime.p } �� �      �.F  �    �pp�Z    &noalg.struct { F uintptr; runtime.pp *runtime.p } ��  %noalg.struct { F uintptr; runtime.grunning *int } �� �      �.F  �    �grunningZr    &noalg.struct { F uintptr; runtime.grunning *int } ��  �[129]*runtime.g e�  �        ��   �� !*[256]runtime.guintptr "_  �        !*runtime.initTask Q�  �        %runtime.initTask ��        �state  �    �ndep�    �nfns ��     &runtime.initTask ��  %runtime.tracestat  �        �active  K�   �iY�   �allocs �)
           �bytes �)
            &runtime.tracestat g�  !**uint8 p�  �        !*runtime.dbgVar 7�  �        %runtime.dbgVar ��        �name     �value �1�    &runtime.dbgVar ��  %noalg.struct { F uintptr; runtime.rw *runtime.rwmutex } �� �      �.F  �    �rw��    &noalg.struct { F uintptr; runtime.rw *runtime.rwmutex } K�  !*runtime.rwmutex ��  � �      %runtime.rwmutex 0�`B�     �rLock  N�   �reader��   �readerPass ���   �wLock �N�   �writer  ��   �readerCount (��   �readerWait ,��    &runtime.rwmutex ��  !*runtime.semaRoot ��  �        %runtime.semaRoot ��        �lock  N�   �trea��   �nwait ���    &runtime.semaRoot ��  !*runtime.exceptionrecord ��  �        %runtime.exceptionrecord ���        �exceptioncode  ��   �exceptionflags ���   �exceptionrecord��   �exceptionaddress �p�   �numberparameters ���   �exceptioninformation  ��    &runtime.exceptionrecord >�  �[15]uintptr �   x�        ��   � !*runtime.adjustinfo ��  �        %runtime.adjustinfo ���        �old  S�   �delta ��    �cache ��v   �sghi ���     &runtime.adjustinfo R�  !*runtime.pcvalueCache �v  � [      !*runtime.stackmap B�  �        %runtime.stackmap �        �n  ��   �nbit ���   �bytedata+:    &runtime.stackmap  �  !*runtime.tmpBuf ��  �        �runtime.tmpBuf ��   �        ��     &runtime.tmpBuf v�  �[2]string    � �      ��   � �[3]string   0���      ��   � �[4]string  @���      ��   � !*[32]int32 (�  �        �[32]int32 ��  ���        ��     !*[70368744177663]uint16 o�  �        �[70368744177663]uint16 ��  ��������        ��   ������� !*runtime.modulehash �.  � Y      !*runtime.findfuncbucket 0�  �        %runtime.findfuncbucket ��        �idx  ��   �subbuckets �R�    &runtime.findfuncbucket ��  !*[8]runtime.pcvalueCacheEnt iv  �        !*runtime.abiPart �  ���      %runtime.abiPart (��,�     �kind  N�   �srcStackOffset�    �dstStackOffset ��    �dstRegister    �len  �     &runtime.abiPart ��  #[]runtime.abiPart ���k      �  �array  v�   �len   �cap �    !*runtime.abiDesc ��  ���      %runtime.abiDesc 8��+�     �parts  ��   �srcStackSize ��    �dstStackSize  �    �dstRegisters (   �retOffset 0�     &runtime.abiDesc ��  %runtime.winCallback H��     �     �fn  m�   �retPop�    �abiMap ���    &runtime.winCallback ��  !*runtime.callbackArgs ��  �        %runtime.callbackArgs  �        �index  �    �args�    �result ��    �retPop ��     &runtime.callbackArgs ��  !**runtime.timer ��  �        #[]uint64 �� r      )
          �array  <�   �len   �cap     !*runtime.traceBufPtr �`  ���      !*runtime.traceBuf ܿ  �        %runtime.traceBuf ����        �traceBufHeader  I�  ��arr ��    &runtime.traceBuf ��  %runtime.traceBufHeader �        �link  �`   �lastTicks)
           �pos �   �stk �e�    &runtime.traceBufHeader ��  �[128]uintptr �   �        ��   �� �[64488]uint8 ��  ����        ��   ��� !*runtime.traceStack (�  �        %runtime.traceStack (�        �link  @�   �hash�    �id ���   �n    �stk  b�    &runtime.traceStack ��  �runtime.traceStackPt        �[0]uintptr �    �        ��     !*runtime.traceStackTable ��  �        %runtime.traceStackTable ����        �lock  N�   �seq��   �mem �P�   �tab  ��    &runtime.traceStackTable ��  %runtime.traceAlloc ��        �head  h�   �of�     &runtime.traceAlloc ��  �runtime.traceAllocBlockPt        �[8192]runtime.traceStackPtr @�  ����        ��   �@ !*runtime.traceAlloc P�  �        !*runtime.traceAllocBlock K�  �        %runtime.traceAllocBlock ����        �ne�  �[65528]uint8 ��  ����        ��   ��� !*runtime.traceAllocBlockPtr h�  �        �[2]uint64 )
          ��        ��   � �[3]uint64 )
          ��        ��   � !*runtime.inlinedCall ��  �        %noalg.struct { F uintptr; runtime.argp unsafe.Pointer } �� �      �.F  �    �arg�     &noalg.struct { F uintptr; runtime.argp unsafe.Pointer } ��  !*[171]uint8 ��  �        �[171]uint8 ��  ���        ��   �� �func(uint8, uint8)�        ���  ���   &func(uint8, uint8) ��  !*runtime.reflectMethodValue ��  �        %runtime.reflectMethodValue ��        �fn  �    �stack�z   �argLen ��     &runtime.reflectMethodValue [�  %runtime.cgoSymbolizerArg 8� ?�     �pc  �    �filp�   �lineno ��    �funcName �p�   �entry  �    �more (�    �data 0�     &runtime.cgoSymbolizerArg ��  %noalg.struct { F uintptr; runtime.pc uintptr; runtime.sp uintptr; runtime.gp *runtime.g; runtime.skip int; runtime.pcbuf []uintptr; runtime.n *int } H� =�     �.F  �    �pc�    �sp ��    �gp �e�   �skip     �pcbuf (��   �n @Zr    &noalg.struct { F uintptr; runtime.pc uintptr; runtime.sp uintptr; runtime.gp *runtime.g; runtime.skip int; runtime.pcbuf []uintptr; runtime.n *int } c�  %noalg.struct { F uintptr; runtime.me *runtime.g; runtime.curgp *runtime.g; runtime.level int32 }  ����     �.F  �    �mee�   �curgp �e�   �level ���    &noalg.struct { F uintptr; runtime.me *runtime.g; runtime.curgp *runtime.g; runtime.level int32 } ��  %noalg.struct { F uintptr; runtime.frame *runtime.stkframe; runtime.bad uintptr } ����      �.F  �    �frame5z   �bad ��     &noalg.struct { F uintptr; runtime.frame *runtime.stkframe; runtime.bad uintptr } ��  !*runtime.cgoSymbolizerArg E�  ��Q      �func(unsafe.Pointer, unsafe.Pointer) int32�        ��   ��   �1�   &func(unsafe.Pointer, unsafe.Pointer) int32 ��  %runtime.cgoTracebackArg  �`��     �context  �    �sigContext�    �buf ���   �max ��     &runtime.cgoTracebackArg \�  !**runtime._type !�  �        %noalg.map.bucket[runtime._typePai�     �topbits  �D   �keys��   �elems ��\�   �overflow ���     &noalg.map.bucket[runtime._typePair]struct {} ��  �noalg.[8]runtime._typePair ��  �����      ��  &noalg.[8]runtime._typePair ��  %runtime._typePair ����      �t1  !�   �t2!�    &runtime._typePair ��  �noalg.[8]struct {} t�   � �      ��   &noalg.[8]struct {} 3�  %struct {}  ���       %noalg.map.bucket[uint32][]*runtime._type ���`��     �topbits  �D   �keysM�   �elems (��   �overflow ���     &noalg.map.bucket[uint32][]*runtime._type ��  �noalg.[8]uint32 ��   �`�      ��  &noalg.[8]uint32 '�  �noalg.[8][]*runtime._type ��  ��� �      ��   &noalg.[8][]*runtime._type b�  #[]*runtime._type �� g      !�  �array  ��   �le   �cap     %noalg.map.hdr[runtime._typePair]struct {} 0� H�     �count    �flag��   �B    ��   �noverflow
        ��   �hash0 ��   �buckets ���   �oldbuckets ���   �nevacuate  �    �extra (�     &noalg.map.hdr[runtime._typePair]struct {} ��  !*map.bucket[runtime._typePair]struct {} f�  ��N      %noalg.map.hdr[uint32][]*runtime._type 0�@I�     �count     �flags��   �B    ��   �noverflow
        ��   �hash0 ��   �buckets ���   �oldbuckets ���   �nevacuate  �    �extra (�     &noalg.map.hdr[uint32][]*runtime._type ��  !*map.bucket[uint32][]*runtime._type ��  � O       map[uint32][]*runtime._type ��� �`�      ��  ��   map[runtime._typePair]struct {} ��� ���      ��  t�  !*runtime.functype ��  �        !*runtime.slicetype ��  �        %runtime.winCallbackKey ����      �fn  m�   �cdeclK�    &runtime.winCallbackKey ��  !*runtime.libcall �#  � W      %struct { runtime.lpFileName *uint16; runtime.hFile uintptr; runtime.flags uint32 } ��`�      �lpFileName  F�   �hFile�    �flags ���    !*runtime.errorString �A  � �      !*runtime.boundsError C  � �      !*runtime.sysmontick �]  ��`      !*struct { runtime.gList; runtime.n int32 } P_  �`�      !*runtime.mOS w$  ��W      !*runtime.lockRank ��  � �      !*runtime.wai  ���      !*runtime.errorAddressString fB  � �      !*struct { runtime.mcentral runtime.mcentral; runtime.pad [24]uint8 } qU  �        !*[136]struct { runtime.mcentral runtime.mcentral; runtime.pad [24]uint8 } �U  �        !*runtime.overlappedEntry ��  ��Z      !*runtime.plainError �B  ���      !*runtime.winCallbackKey ��  � c      !*[2]string ��  �        !*[3]string Ϻ  �        !*[4]string ��  �        !*[64]runtime.overlappedEntry ��  �        !*[9]string �C  �        !*struct { runtime.lpFileName *uint16; runtime.hFile uintptr; runtime.flags uint32 } >�  �        !*sync.Mutex F�  � �      %sync.Mutex���      �state  ��   �sema ���    &sync.Mutex ��  !*sync.Pool ��  �        %sync.Pool (�        �noCopy  ��   �local  �    �localSiz�    �victim ��    �victimSize ��    �New  9�    &sync.Pool o�  %sync.noCopy  �         &sync.noCopy ��  �func() interface {}�        �R�   &func() interface {} ��  !*interface {} 4�           !*struct { runtime/cgo.cstr *uint8 } ��  �        %struct { runtime/cgo.cstr *uint8 �        �cstr  p�    �internal/reflectlite.Kind��      �internal/reflectlite.tflag `{      �internal/reflectlite.chanDir�        �internal/reflectlite.flag        !*internal/reflectlite.rtype ^�  �`c�     %internal/reflectlite.rtype 0�@U�     �size  �    �ptrdat�    �hash ���   �tflag ���   �align ���   �fieldAlign ���   �kind ���   �equal ���   �gcdata  p�   �str (~�   �ptrToThis ,��    &internal/reflectlite.rtype ��  �internal/reflectlite.nameOff ��� {      �internal/reflectlite.typeOff ����{      %internal/reflectlite.name�        �bytes  p�    &internal/reflectlite.name ��  !*internal/unsafeheader.String ��  �        %internal/unsafeheader.String ��        �Data  �    �Len    &internal/unsafeheader.String N�  !*internal/reflectlite.arrayType :�  �        %internal/reflectlite.arrayType H�        �rtype  ^�  ��elem 0w�   �slice 8w�   �len @�     &internal/reflectlite.arrayType ��  !*internal/reflectlite.chanType ��  �        %internal/reflectlite.chanType @�        �rtype  ^�  ��elem 0w�   �dir 8�     &internal/reflectlite.chanType ��  !*internal/reflectlite.mapType ��  �        %internal/reflectlite.mapType X�        �rtype  ^�  ��key 0w�   �elem 8w�   �bucket @w�   �hasher H��   �keysize P��   �valuesize Q��   �bucketsize R��   �flags T��    &internal/reflectlite.mapType (�  !*internal/reflectlite.ptrType `�  �        %internal/reflectlite.ptrType 8�        �rtype  ^�  ��elem 0w�    &internal/reflectlite.ptrType ��  !*internal/reflectlite.sliceType ��  �        %internal/reflectlite.sliceType 8�        �rtype  ^�  ��elem 0w�    &internal/reflectlite.sliceType ��  !*internal/reflectlite.Kind ��  �@�      !*internal/reflectlite.uncommonType ��  ���      %internal/reflectlite.uncommonType ���*�     �pkgPath  ~�   �mcount ���   �xcount ���   �moff��   �_ ��    &internal/reflectlite.uncommonType r�  !*errors.errorString Q�  ���      %errors.errorString ����      �s     &errors.errorString )�  %internal/reflectlite.emptyInterface ��        �typ  w�   �wor�     &internal/reflectlite.emptyInterface i�  �syscall.Handle`�      �syscall.Errno �      �syscall.Signal �        !*[6]string �A  �        !*syscall.DLLError ��  ���      %syscall.DLLError 0�`
        �     �Err  ��   �ObjName    �Msg     &syscall.DLLError b�  !*syscall.DLL �  � �      %syscall.DLL ��`�      �Name     �Handle ���    &syscall.DLL ��  !*syscall.Proc p�  � �      %syscall.Proc  ����     �Dll  ��   �Nam   �addr ��     &syscall.Proc 4�  %noalg.struct { F uintptr; syscall..autotmp_7 *sync.Mutex } ����      �.F  �    �.autotmp_��    &noalg.struct { F uintptr; syscall..autotmp_7 *sync.Mutex } ��  !*syscall.LazyDLL ��  �@�      %syscall.LazyDLL  � ��     �mu  F�   �dl��   �Name     &syscall.LazyDLL D�  !*syscall.LazyProc ��  ���      %syscall.LazyProc (����     �mu  F�   �Na   �l �%�   �proc  ��    &syscall.LazyProc ��  �[300]uint16 ��  ���        ��   �� !*syscall.Errno ��  ���      !*struct { main.r0 int } {�  �        %struct { main.r0 int }�        �r0      �[131]string  ���        ��   �� #[]*sync.Pool ��        V�  �array  ���  �l   �cap �     map[string]bool S�� � �        K�  �[128]uint8 ��  ���        ��   �� �[4]uintptr �    �        ��   � �[8]string  ���        ��  �[33]float64 uC  ���        ��   ! �[256]uint64 )
          ���        ��   �� %struct { runtime.mutex; runtime.persistentAlloc } ��        �mutex  N�  ��persistentAllo�`  � �[1024]uint8 ��  �        ��   �[5]uint8 ��  ��        ��   � �chan int �#� ��v       %struct { runtime.enabled bool; runtime.pad [3]uint8; runtime.needed bool; runtime.cgo bool; runtime.alignme uint64 } ��        �enabled  K�   �pad �a�   �needed �K�   �cgo �K�   �alignm)
            �[3]uint8 ��  ��        ��   � %struct { runtime.full runtime.lfstack; runtime.empty runtime.lfstack; runtime.pad0 internal/cpu.CacheLinePad; runtime.wbufSpans struct { runtime.lock runtime.mutex; runtime.free runtime.mSpanList; runtime.busy runtime.mSpanList }; _ uint32; runtime.bytesMarked uint64; runtime.markrootNext uint32; runtime.markrootJobs uint32; runtime.nproc uint32; runtime.tstart int64; runtime.nwait uint32; runtime.nDataRoots int; runtime.nBSSRoots int; runtime.nSpanRoots int; runtime.nStackRoots int; runtime.baseData uint32; runtime.baseBSS uint32; runtime.baseSpans uint32; runtime.baseStacks uint32; runtime.baseEnd uint32; runtime.startSema uint32; runtime.markDoneSema uint32; runtime.bgMarkReady runtime.note; runtime.bgMarkDone uint32; runtime.mode runtime.gcMode; runtime.userForced bool; runtime.totaltime int64; runtime.initialHeapLive uint64; runtime.assistQueue struct { runtime.lock runtime.mutex; runtime.q runtime.gQueue }; runtime.sweepWaiters struct { runtime.lock runtime.mutex; runtime.list runtime.gList }; runtime.cycles uint32; runtime.stwprocs int32; runtime.maxprocs int32; runtime.tSweepTerm int64; runtime.tMark int64; runtime.tMarkTerm int64; runtime.tEnd int64; runtime.pauseNS int64; runtime.pauseSFAILURE: No signal received
FAIL
FAIL    runtime 99.834s
FAIL

c:\Users\Alex\dev\go\src>

Alex

Loading

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Apr 28, 2021

Another avenue to explore here (I don't have time anymore) is to try to run os/signal.TestCtrlBreak to see if it breaks too. Similarly, you should not run it with -counter parameter, because test build Go executable first. So just modify the test to build executable once and then run it many times in a loop.

If Go control handler code is broken, then this test might break too.

Alex

Loading

@toothrot
Copy link
Contributor

@toothrot toothrot commented Apr 29, 2021

/cc @bufflig

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 30, 2021

Ah!! I can explain the garbage, too.

Turns out that we do actually check for this case! There's a check at the beginning of needm that checks if we initialized far enough yet.

Unfortunately, what that does then is print the earlycgocallback message... but the runtime globals haven't been initialized yet! Hence all sorts of garbage is printed.

Loading

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Apr 30, 2021

Cute. Nice analysis.

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 30, 2021

Actually, I'm less certain now about where the garbage data is coming from. earlycgocallback won't be used (needm won't be called) before osinit, and osinit uses exactly the same kind of global variable on Linux (I've confirmed it works). I still feel like it's that print that's problematic, but I don't understand why earlycgocallback would be invalid at that point.

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 30, 2021

Ah, this may be a PE vs. ELF thing? I don't see Windows runtime code relying on a variable being initialized anywhere.

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Apr 30, 2021

Yeah... If I print the length of earlycgocallback in osinit it works fine.

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented May 3, 2021

Coming back to this... I think my theory is subtly wrong.

According to the MSDN, control handlers are always called on a new thread (https://docs.microsoft.com/en-us/windows/console/console-control-handlers). That thread should be distinct from the runtime initialization thread, and so shouldn't impede progress in the runtime, presumably. That thread will just spin in needm until initialization gets far enough.

The only thing preventing progress though, is that check of cgoHasExtraM at the beginning of needm. If that somehow cascades into a bunch of failures, that would explain why the test fails in such a strange manner (though, admittedly, I still don't understand how it would do so).

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented May 3, 2021

Well, hey I learned one new thing. There should be a runtime.abort call after runtime.badsignal2 on Windows. That probably explains the cascading failures (there's no way to proceed, and yet we do!).

I'm going to try to see where that signal is coming from. I have access to the instruction pointer in that context, it's just a question of how to actually surface it.

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented May 4, 2021

The PC that triggers the failure isn't one that's known to Go (and it's consistently the same one). If I were to guess, it's probably a Windows DLL (win32?).

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented May 4, 2021

Change https://golang.org/cl/316809 mentions this issue: runtime: abort after emitting message for bad signal on Windows

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented May 4, 2021

The saga continues. I fixed the garbage-getting-printed issue, but there's still that pesky signal firing (FWIW it's an access violation signal, so the equivalent of a segfault, and to catch you up, the PC that triggered it is NOT in the Go binary). The good news is, I've narrowed the source a little further. Namely, if I eliminate the call to SetConsoleCtrlHandler, but keep the compileCallback invocation (printing the PC so the call stays around... also I made sure it still reproduces with everything the same, just the print added in). Doing this does not reproduce.

Now what I did was I kept the print and the compileCallback in, but rather than use the result of compileCallback as the handler, I made my own little handler in assembly that basically just returns FALSE (so the next handler runs). And with that I can still reproduce. So, there's something wrong about where we're calling SetConsoleCtrlHandler specifically. It doesn't seem to involve the callback, or needm, or the cgocallback path at all.

I'm thinking now that there's, very simply (back to the very very beginning of this thread), a race between SetConsoleCtrlHandler and when the control signal lands. I do not understand what the problem could be. MSDN does not discuss any concurrency guarantees (e.g. is it safe to call SetConsoleCtrlHandler concurrently? (Note that in this case it is not being called concurrently, I just mean that as an example)).

At this point I have no idea how to proceed. My CL above is a reasonable workaround. I can also make it so that the test binary calls Dummy in the Go DLL before indicating readiness. That seems to fix it, too.

@ianlancetaylor @alexbrainman WDYT?

Loading

gopherbot pushed a commit that referenced this issue May 4, 2021
Currently if a signal lands on a non-Go thread that's handled by the Go
handler, Go will emit a message. However, unlike everywhere else in the
runtime, Go will not abort the process after, and the signal handler
will try to continue executing.

This leads to cascading failures and possibly even memory corruption.

For #45638.

Change-Id: I546f4e82f339d555bed295528d819ac883b92bc6
Reviewed-on: https://go-review.googlesource.com/c/go/+/316809
Trust: Michael Knyszek <mknyszek@google.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented May 6, 2021

The saga continues. I fixed the garbage-getting-printed issue, but there's still that pesky signal firing (FWIW it's an access violation signal, so the equivalent of a segfault, and to catch you up, the PC that triggered it is NOT in the Go binary).

I tried running test on top of 1108cbe and test failed once and I still see garbage printed:

c:\Users\Alex\dev\go\test>go test -count=10 -run=CtrlH runtime
--- FAIL: TestLibraryCtrlHandler (19.45s)
    signal_windows_test.go:211: Program exited with error: exit status 1
        runtime: signal received on thread not created by Go.
                                                                                  ,�ݪY-�A��Z�R��"on�5�M����.R�F�x58D�%��q�ч]�6I#�9�2�g�.i保�C��ߞc�T;:����8���Av���8�*�Z���oZ���۞�X����Ո��~��ӛ��k��O��                                                                                                                                                                                                                                                                runtime: signal received on thread not created by Go.
                                                                                  ,�ݪY-�A��Z�R��"on�5�M����.R�F�x58D�%��q�ч]�6I#�9�2�g�.i保�C��ߞc�T;:����8���Av���8�*�Z���oZ���۞�X����Ո��~��ӛ��k��O��        ���     ���                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             � ��������������                                                                                runtime: signal received on thread not created by Go.
                                                                                  ,�ݪY-�A��Z�R��"on�5�M����.R�F�x58D�%��q�ч]�6I#�9�2�g�.i保�C��ߞc�T;:����8���Av���8�*�Z���oZ���۞�X����Ո��~��ӛ��k��O��        ���     ���     
... more garbage ...                                                           

Is this garbage expected?

I also noticed that this message would only be printed, if runtime.sigtramp called and it could not find g. The fact that runtime.sigtramp is called means we received an exception. It would be nice to see stack trace printed. Unfortunately runtime.sigtramp end-up calling runtime.badsignal2 instead.

Perhaps we can disable runtime.badsignal2 check and hope stack trace is printed.

Another option would be run test program inside a debugger. Debugger will stop when exception. Maybe we can see the program state then. It would be nice to see exactly where exception occurs.

Perhaps your https://go-review.googlesource.com/c/go/+/315830/ fixes this error, and is good enough.

The good news is, I've narrowed the source a little further. Namely, if I eliminate the call to SetConsoleCtrlHandler, but keep the compileCallback invocation (printing the PC so the call stays around... also I made sure it still reproduces with everything the same, just the print added in). Doing this does not reproduce.

That means that your CL 315830 could not be a solution here. Correct?

I'm thinking now that there's, very simply (back to the very very beginning of this thread), a race between SetConsoleCtrlHandler and when the control signal lands. I do not understand what the problem could be. MSDN does not discuss any concurrency guarantees (e.g. is it safe to call SetConsoleCtrlHandler concurrently? (Note that in this case it is not being called concurrently, I just mean that as an example)).

What particular race are you thinking about? You are allowed to call SetConsoleCtrlHandler API many times - that is how the API is designed. In fact I am pretty sure our test calls SetConsoleCtrlHandler twice - Go calls it, and C runtime also calls it.

Perhaps the problem is that Go DLL does not remove SetConsoleCtrlHandler handler before it exits? And Windows calls the handler address when Go runtime is already off.

Similarly SetConsoleCtrlHandler is supposed to handle CTRL_CLOSE_EVENT. Perhaps we get that message (I don't see how that happens) after Go DLL runtime is off, but Windows SetConsoleCtrlHandler handler still points to Go code.

I plan to play with this more on the weekend. Let's delay the decision.

Also, perhaps @zx2c4 can help debug this, if you have time. Thank you.

Alex

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented May 6, 2021

The saga continues. I fixed the garbage-getting-printed issue, but there's still that pesky signal firing (FWIW it's an access violation signal, so the equivalent of a segfault, and to catch you up, the PC that triggered it is NOT in the Go binary).

I tried running test on top of 1108cbe and test failed once and I still see garbage printed:

c:\Users\Alex\dev\go\test>go test -count=10 -run=CtrlH runtime
--- FAIL: TestLibraryCtrlHandler (19.45s)
    signal_windows_test.go:211: Program exited with error: exit status 1
        runtime: signal received on thread not created by Go.
                                                                                  ,�ݪY-�A��Z�R��"on�5�M����.R�F�x58D�%��q�ч]�6I#�9�2�g�.i保�C��ߞc�T;:����8���Av���8�*�Z���oZ���۞�X����Ո��~��ӛ��k��O��                                                                                                                                                                                                                                                                runtime: signal received on thread not created by Go.
                                                                                  ,�ݪY-�A��Z�R��"on�5�M����.R�F�x58D�%��q�ч]�6I#�9�2�g�.i保�C��ߞc�T;:����8���Av���8�*�Z���oZ���۞�X����Ո��~��ӛ��k��O��        ���     ���                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             � ��������������                                                                                runtime: signal received on thread not created by Go.
                                                                                  ,�ݪY-�A��Z�R��"on�5�M����.R�F�x58D�%��q�ч]�6I#�9�2�g�.i保�C��ߞc�T;:����8���Av���8�*�Z���oZ���۞�X����Ո��~��ӛ��k��O��        ���     ���     
... more garbage ...                                                           

Is this garbage expected?

It's not, that's very surprising. It's also surprising that you landed in badsignal2 more than once -- all the failures I got after that CL were clean. I'll try to reproduce this again.

I also noticed that this message would only be printed, if runtime.sigtramp called and it could not find g. The fact that runtime.sigtramp is called means we received an exception. It would be nice to see stack trace printed. Unfortunately runtime.sigtramp end-up calling runtime.badsignal2 instead.

Perhaps we can disable runtime.badsignal2 check and hope stack trace is printed.

Unfortunately I don't think that'll work. There's a lot of stuff that depends on having some kind of G (including a g0); I'm not even positive we can throw in that circumstance.

Another option would be run test program inside a debugger. Debugger will stop when exception. Maybe we can see the program state then. It would be nice to see exactly where exception occurs.

I've been putting this aside because I don't have an environment where I can easily attach a debugger and follow subprocesses. The Windows world feels alien to me. I don't have high confidence in Delve handling this very specific situation very well, but I can try.

Perhaps your https://go-review.googlesource.com/c/go/+/315830/ fixes this error, and is good enough.

The good news is, I've narrowed the source a little further. Namely, if I eliminate the call to SetConsoleCtrlHandler, but keep the compileCallback invocation (printing the PC so the call stays around... also I made sure it still reproduces with everything the same, just the print added in). Doing this does not reproduce.

That means that your CL 315830 could not be a solution here. Correct?

What the commit message of that CL describes as the problem is incorrect, yes. However, it's clear that calling SetConsoleControlHandler later in initialization resolves the problem. Hence my theory about some kind of race condition.

I'm thinking now that there's, very simply (back to the very very beginning of this thread), a race between SetConsoleCtrlHandler and when the control signal lands. I do not understand what the problem could be. MSDN does not discuss any concurrency guarantees (e.g. is it safe to call SetConsoleCtrlHandler concurrently? (Note that in this case it is not being called concurrently, I just mean that as an example)).

What particular race are you thinking about? You are allowed to call SetConsoleCtrlHandler API many times - that is how the API is designed. In fact I am pretty sure our test calls SetConsoleCtrlHandler twice - Go calls it, and C runtime also calls it.

I don't have a specific one in mind, because I don't have any documentation on SetConsoleCtrlHandler's requirements. But I think it's clear to me that there's some kind of race going on. Runtime initialization is happening on a separate thread in parallel with the rest of the C code in the test. In particular, it's happening in parallel with the CTRL_BREAK landing in the process. I think there's some kind of invariant in the Windows APIs that we're violating. The sigtramp is coming from some PC that I can't identify, so my default assumption is that it's some shared library in the address space (could be wrong though).

Perhaps the problem is that Go DLL does not remove SetConsoleCtrlHandler handler before it exits? And Windows calls the handler address when Go runtime is already off.

Similarly SetConsoleCtrlHandler is supposed to handle CTRL_CLOSE_EVENT. Perhaps we get that message (I don't see how that happens) after Go DLL runtime is off, but Windows SetConsoleCtrlHandler handler still points to Go code.

Interesting, I hadn't thought about a situation where runtime initialization is complete. What do you mean by when the Go runtime is already "off"? Do you mean the rest of the process is already exiting?

I plan to play with this more on the weekend. Let's delay the decision.

Also, perhaps @zx2c4 can help debug this, if you have time. Thank you.

Alex

Thank you so much for taking the time to look at this!

Loading

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented May 9, 2021

It's not, that's very surprising. It's also surprising that you landed in badsignal2 more than once -- all the failures I got after that CL were clean. I'll try to reproduce this again.

badsignal2 calls abort at the end now. And abort starts with INT $3. And INT $3 will raise exception again (we just came from badsignal2 and sigtramp), and will call sigtramp again (I did not check that, I am just guessing).

I managed to reproduce this error just twice. But I did not have debugger ready at that time. I will try again next week.

Alex

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented May 10, 2021

Huh. I figured abort meant something closer to SIGABRT but yeah I guess I would expect INT $3 to call back into the runtime.

Should this just ExitProcess then? It's not hard to arrange, I was doing that in some of my testing (I surfaced the crashing PC and SP via the exit value).

Loading

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented May 11, 2021

Huh. I figured abort meant something closer to SIGABRT

I don't know enough about SIGABRT to comment.

but yeah I guess I would expect INT $3 to call back into the runtime.

INT $3 raises an exception on Windows. So our exception handler is called when that happens. I don't remember what happens when exception is raised when exception handler is run, but, I think, it will just call exception handler recursively.

Should this just ExitProcess then?

I think we should do that.

We should also print more exception details, if possible. See winthrow function. Exception code and exception address might help users to debug. If it is not hard to do.

Alex

Loading

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented May 13, 2021

I found a computer (with Windows 7) where this issue reproduces much more easily than on my Windows 10 computer.

Unfortunately I can reproduce this issue even on top of

https://go-review.googlesource.com/c/go/+/315830/

I will try to understand where the issue occurs, as promised. No success guraneteed.

Alex

Loading

@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented May 20, 2021

@mknyszek Checking in from a release meeting, are there updates here? Thanks.

Loading

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented May 21, 2021

@dmitshur I have been working on this issue.

I think the problem here is with the runtime.TestLibraryCtrlHandler. The test is racy and broken.

For example, the test was introduced together with the fix in CL 211139. But the test still passes, if I revert to commit before CL 211139.

c:\Users\Alex\dev\go\src\runtime>git status
HEAD detached at 5756808ce8
nothing to commit, working tree clean

c:\Users\Alex\dev\go\src\runtime>go test -v -count=1 -run=yCtrlH
=== RUN   TestLibraryCtrlHandler
--- PASS: TestLibraryCtrlHandler (2.98s)
PASS
ok      runtime 3.569s

c:\Users\Alex\dev\go\src\runtime>git checkout HEAD~ os_windows.go
Updated 1 path from 2e2e0c0cf8

c:\Users\Alex\dev\go\src\runtime>git diff --staged
diff --git a/src/runtime/os_windows.go b/src/runtime/os_windows.go
index 7576565599..bddc25729a 100644
--- a/src/runtime/os_windows.go
+++ b/src/runtime/os_windows.go
@@ -1031,11 +1031,7 @@ func ctrlhandler1(_type uint32) uint32 {
        if sigsend(s) {
                return 1
        }
-       if !islibrary && !isarchive {
-               // Only exit the program if we don't have a DLL.
-               // See https://golang.org/issues/35965.
-               exit(2) // SIGINT, SIGTERM, etc
-       }
+       exit(2) // SIGINT, SIGTERM, etc
        return 0
 }


c:\Users\Alex\dev\go\src\runtime>go test -v -count=1 -run=yCtrlH
=== RUN   TestLibraryCtrlHandler
--- PASS: TestLibraryCtrlHandler (3.86s)
PASS
ok      runtime 4.270s

c:\Users\Alex\dev\go\src\runtime>

The runtime.TestLibraryCtrlHandler assumes that by the time LoadLibrary returns that Go runtime has finished initialising (including calling SetConsoleCtrlHandler in runtime)

HMODULE dummyDll = LoadLibrary("dummy.dll");
if (!dummyDll) {
fprintf(stderr, "ERROR: Could not load dummy.dll");
return 1;
}
printf("ready\n");
fflush(stdout);

But there is nothing in the test to verify that SetConsoleCtrlHandler was called or not. In fact, if I comment out LoadLibrary call in the test, the test still passes.

So we need to replace the test first. And then we should be able to tell, if we have a problem or not.

It is pretty hard to build proper test here. But I think I am making progress. I will report more when I am ready with the test.

I hope it helps.

Alex

Loading

@zx2c4
Copy link
Contributor

@zx2c4 zx2c4 commented May 21, 2021

But there is nothing in the test to verify that SetConsoleCtrlHandler was called or not. In fact, if I comment out LoadLibrary call in the test, the test still passes.

This should be very easy to fix. It's true that Go initialization is done asynchronously on another thread, so this test races. (Side note: even if it races, it shouldn't spew garbage like above.) But every cgo entry point calls _cgo_wait_runtime_init_done, which properly waits for initialization to complete. So an easy fix to that is to just call the dummy function from dummy.go before printing "ready". I'll submit a patch for that.

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented May 21, 2021

Change https://golang.org/cl/321769 mentions this issue: runtime: wait for Go runtime to initialize in Windows signal test

Loading

@zx2c4
Copy link
Contributor

@zx2c4 zx2c4 commented May 21, 2021

@alexbrainman https://go-review.googlesource.com/c/go/+/321769 passes tests and should now fix the races with the tests you mentioned. I haven't read all of this thread in detail, so I'm not sure if this is all that's required to close the issue. But at the very least, you can +2 that and we'll at least have a fixed test.

However, the garbage text you're getting is a bit disturbing, and maybe we should look more deeply into that after you review my test fix. Specifically: if a signal arrives before the Go runtime is finished initializing, why are we spewing garbage rather than doing something polite? Do we have better options?

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented May 21, 2021

I added the call to Dummy in an earlier iteration of investigating this, but @alexbrainman brings up a good point that the test isn't properly checking the condition it's supposed to check (i.e. it's racy).

Maybe the path forward here is to fix the test, and then file a separate issue for the garbage being spewed due to the race between the signal landing and initialization completing. Then again, the garbage spewing might just be fixed by calling ExitProcess directly instead of runtime.abort in badsignal2. I'll send a CL for that.

Loading

@zx2c4
Copy link
Contributor

@zx2c4 zx2c4 commented May 21, 2021

I added the call to Dummy in an earlier iteration of investigating this, but @alexbrainman brings up a good point that the test isn't properly checking the condition it's supposed to check (i.e. it's racy).

Adding a call to Dummy DOES make the test properly check for the condition that it's supposed to check and FIXES the race. Why does it do this? Because the call to Dummy implies a call to _cgo_wait_runtime_init_done, which waits for initialization to complete, and hence waits for the signal handler to be installed.

then file a separate issue for the garbage being spewed due to the race between the signal landing and initialization completing. Then again, the garbage spewing might just be fixed by calling ExitProcess directly instead of runtime.abort in badsignal2. I'll send a CL for that.

Or you could just re-title this issue if you wanted? AFAICT, https://go-review.googlesource.com/c/go/+/321769 is the correct fix for the actual test failing, but indeed, I agree with you, that the garbage spewing is the lingering issue that definitely deserves attention.

Loading

gopherbot pushed a commit that referenced this issue May 21, 2021
The test harness waits for "ready" as a sign that the Go runtime has
installed its signal handler and is ready to be tested. But actually,
while LoadLibrary starts the loading of the Go runtime, it does so
asynchronously, so the "ready" sign is potentially premature and
certainly racy. However, all exported cgo entry points make a call to
_cgo_wait_runtime_init_done which waits for that asynchronous
initialization to complete. Therefore, this commit fixes the test to
call into the exported "Dummy" cgo function before emitting the "ready"
sign, so that we're sure the Go runtime is actually loaded.

Updates #45638.

Change-Id: I9b12b172d45bdcc09d54dd301de3a3e499544834
Reviewed-on: https://go-review.googlesource.com/c/go/+/321769
Trust: Jason A. Donenfeld <Jason@zx2c4.com>
Run-TryBot: Jason A. Donenfeld <Jason@zx2c4.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented May 21, 2021

Change https://golang.org/cl/321789 mentions this issue: runtime: exit harder in badsignal2

Loading

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented May 24, 2021

I think we can close this issue because

https://go-review.googlesource.com/c/go/+/321769

fixes the runtime.TestLibraryCtrlHandler test properly.

The test fails if I revert

https://go-review.googlesource.com/c/go/+/211139/

But on current tip I could not make test fail in any way on few different systems.

Alex

Loading

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented May 24, 2021

@alexbrainman Sounds good. I'll file a new issue for the garbage we saw getting generated.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants