Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/cgo: "function symbol table not sorted by program counter" crash when using multiple shared libraries linked with -buildmode=c-archive #30822

Closed
mewmew opened this issue Mar 14, 2019 · 7 comments

Comments

@mewmew
Copy link
Contributor

commented Mar 14, 2019

What version of Go are you using (go version)?

$ go version
go version devel +04f1b65cc6 Wed Mar 13 15:11:37 2019 +0000 linux/amd64

I also tried to run this on the Go 1.12 release, and the same crash happens there.

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/u/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/u/goget:/home/u/Desktop/go"
GOPROXY=""
GORACE=""
GOROOT="/home/u/go1.13"
GOTMPDIR=""
GOTOOLDIR="/home/u/go1.13/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build238606776=/tmp/go-build -gno-record-gcc-switches"

What did you do?

For a minimal reproducible example code see: https://github.com/mewspring/go-cgo-issue

$ git clone https://github.com/mewspring/go-cgo-issue
$ cd go-cgo-issue/foo
$ make -B
$ ./foo
...
fatal error: cgo callback before cgo call

In essence, I have a minimal C program that invokes multiple shared libraries, each one written in C and linked with Go code using -buildmode=c-archive.

The program works great when invoking Go code from C this way when using one library. However, when you invoke Go functions from more than one library, the program crashes with the following error message.

u@x1 ~/D/f/foo> ./foo
function symbol table not sorted by program counter: 0xf7ef1f00 _cgo_topofstack > 0xf7dcbf20 runtime.goexit
    0xf7d80780 __x86.get_pc_thunk.ax
    0xf7d80790 __x86.get_pc_thunk.cx
..
    0xf7d8c840 runtime.evacuate_fast32
    0xf7d8cc30 runtime.mapassign_faststr
a1
       args: ./foo
b1
c1
fatal error: cgo callback before cgo call

Note: if we simply uncomment one line (https://github.com/mewspring/go-cgo-issue/blob/d03e251e54cc0a6821bee5fb15c52d69e90c08d7/c/c.c#L12), the invocation of the Go function FooBar in the c.so library, then the program works as expected.

What did you expect to see?

No crash. In particular, I expected to see the output:

a1
       args: ./foo
b1
c1
       foobar: ./foo

And a status code of 0.

What did you see instead?

The crash "fatal error: cgo callback before cgo call" crash.

@mewmew mewmew changed the title cmd/cgo: Minimal program crash with "fatal error: cgo callback before cgo call" when invoking functions of multiple shared libraries build using -buildmode=c-archive cmd/cgo: Minimal program crash with "fatal error: cgo callback before cgo call" when invoking functions of multiple shared libraries built using -buildmode=c-archive Mar 14, 2019

@mewmew

This comment has been minimized.

Copy link
Contributor Author

commented Mar 16, 2019

Is there any further information I could provide to help investigate the cause of this issue? Also, is this a previously known issue or limitation of Cgo interaction with Go?

The scenario is as follows.

We have a C program foo and three shared libraries written in C, a.so, b.so and c.so.

  1. The first C library (a.so) links against a go-a.a archive produced with -buildmode=c-archive. The go-a.a archive contains one exported Go function that is invoked from C.
  2. The second C library (b.so) links against a go-b.a archive produced with -buildmode=c-archive. The go-b.a archive contains no exported Go functions.
  3. The third C library (c.so) links against a go-c.a archive produced with -buildmode=c-archive. The go-c.a archive contains one exported Go function that is invoked from C.

Given this setup, the resulting binary foo crashes with fatal error: cgo callback before cgo call.

If we change the scenario to not invoke any Go function in go-c.a, then we can successfully run the foo program, without experiencing any crashes.

So, from what I can tell, it seems that only one shared library may have Cgo code in it (produced with -buildmode=c-archive)? It this a correct understanding of the issue and is it a known limitation?

Cheers,
Robin

@mewmew

This comment has been minimized.

Copy link
Contributor Author

commented Mar 16, 2019

A quick update. @acln0 helped me do some initial trouble shooting. We've now reproduced this issue on two machines (and on both 32-bit and 64-bit builds).

@mewmew mewmew changed the title cmd/cgo: Minimal program crash with "fatal error: cgo callback before cgo call" when invoking functions of multiple shared libraries built using -buildmode=c-archive cmd/cgo: "function symbol table not sorted by program counter" crash when using multiple shared libraries linked with -buildmode=c-archive Mar 16, 2019

@mewmew

This comment has been minimized.

Copy link
Contributor Author

commented Mar 16, 2019

We could further conclude that the issue is with the function symbol table not sorted by program counter error (and not the cgo callback before cgo call error).

One way of doing so was to insert a call to sleep before invoking the Go function FooBar in c.c. With the added call to sleep, the ./foo program crashes instead with the error message panic before malloc heap initialized, preceded by function symbol table not sorted by program counter.

crash error message Output
u@x1 ~/D/f/g/foo> ./foo
function symbol table not sorted by program counter: 0x7f814ca73920 _cgo_topofstack > 0x7f814c8f0970 runtime.goexit
	0x7f814c8a3730 sync/atomic.StoreUint32
	0x7f814c8a3740 internal/cpu.Initialize
	0x7f814c8a3790 internal/cpu.processOptions
	0x7f814c8a3e20 internal/cpu.indexByte
	0x7f814c8a3e60 internal/cpu.doinit
	0x7f814c8a4270 internal/cpu.cpuid
	0x7f814c8a4290 internal/cpu.xgetbv
	0x7f814c8a42b0 type..hash.internal/cpu.CacheLinePad
	0x7f814c8a42c0 type..eq.internal/cpu.CacheLinePad
	0x7f814c8a42d0 type..hash.internal/cpu.arm64
	0x7f814c8a4330 type..eq.internal/cpu.arm64
	0x7f814c8a43a0 type..hash.internal/cpu.option
	0x7f814c8a4420 type..eq.internal/cpu.option
	0x7f814c8a44d0 type..hash.internal/cpu.x86
	0x7f814c8a4530 type..eq.internal/cpu.x86
	0x7f814c8a4590 type..hash.[15]internal/cpu.option
	0x7f814c8a4600 type..eq.[15]internal/cpu.option
	0x7f814c8a4690 runtime/internal/atomic.Store
	0x7f814c8a46a0 internal/bytealg.init.0
	0x7f814c8a46d0 internal/bytealg.init
	0x7f814c8a4730 runtime.cmpstring
	0x7f814c8a4750 cmpbody
	0x7f814c8a4990 runtime.memequal
	0x7f814c8a49c0 runtime.memequal_varlen
	0x7f814c8a49f0 memeqbody
	0x7f814c8a4b30 internal/bytealg.IndexByteString
	0x7f814c8a4b50 indexbytebody
	0x7f814c8a4c70 runtime.memhash0
	0x7f814c8a4c80 runtime.memhash8
	0x7f814c8a4ce0 runtime.memhash16
	0x7f814c8a4d40 runtime.memhash128
	0x7f814c8a4da0 runtime.memhash_varlen
	0x7f814c8a4df0 runtime.strhash
	0x7f814c8a4e50 runtime.f32hash
	0x7f814c8a4f70 runtime.f64hash
	0x7f814c8a5090 runtime.c64hash
	0x7f814c8a5100 runtime.c128hash
	0x7f814c8a5170 runtime.interhash
	0x7f814c8a52e0 runtime.nilinterhash
	0x7f814c8a5450 runtime.memequal0
	0x7f814c8a5460 runtime.memequal8
	0x7f814c8a5480 runtime.memequal16
	0x7f814c8a54a0 runtime.memequal32
	0x7f814c8a54c0 runtime.memequal64
	0x7f814c8a54e0 runtime.memequal128
	0x7f814c8a5510 runtime.f32equal
	0x7f814c8a5540 runtime.f64equal
	0x7f814c8a5570 runtime.c64equal
	0x7f814c8a55b0 runtime.c128equal
	0x7f814c8a55f0 runtime.strequal
	0x7f814c8a5660 runtime.interequal
	0x7f814c8a56d0 runtime.nilinterequal
	0x7f814c8a5740 runtime.efaceeq
	0x7f814c8a5860 runtime.ifaceeq
	0x7f814c8a5980 runtime.alginit
	0x7f814c8a5a30 runtime.initAlgAES
	0x7f814c8a5b00 runtime.atomicwb
	0x7f814c8a5b80 runtime.atomicstorep
	0x7f814c8a5bd0 runtime.mmap
	0x7f814c8a5d50 runtime.munmap
	0x7f814c8a5e00 runtime.sigaction
	0x7f814c8a5f60 runtime.cgocall
	0x7f814c8a6050 runtime.cgocallbackg
	0x7f814c8a61c0 runtime.cgocallbackg1
	0x7f814c8a6430 runtime.unwindm
	0x7f814c8a64f0 runtime.cgoCheckArg
	0x7f814c8a6c40 runtime.cgoCheckUnknownPointer
	0x7f814c8a6f90 runtime.cgoIsGoPointer
	0x7f814c8a7080 runtime.cgoCheckResult
	0x7f814c8a7110 runtime._cgo_panic_internal
	0x7f814c8a7190 runtime.cgoCheckWriteBarrier
	0x7f814c8a7290 runtime.cgoCheckMemmove
	0x7f814c8a7330 runtime.cgoCheckSliceCopy
	0x7f814c8a7410 runtime.cgoCheckTypedBlock
	0x7f814c8a77a0 runtime.cgoCheckBits
	0x7f814c8a7890 runtime.cgoCheckUsingType
	0x7f814c8a7af0 runtime.makechan
	0x7f814c8a7d30 runtime.chansend1
	0x7f814c8a7d70 runtime.chansend
	0x7f814c8a83b0 runtime.send
	0x7f814c8a84d0 runtime.sendDirect
	0x7f814c8a8560 runtime.recvDirect
	0x7f814c8a85f0 runtime.closechan
	0x7f814c8a8870 runtime.chanrecv1
	0x7f814c8a88b0 runtime.chanrecv
	0x7f814c8a8f70 runtime.recv
	0x7f814c8a9130 runtime.(*waitq).dequeue
	0x7f814c8a9230 runtime.init.0
	0x7f814c8a9290 runtime.(*cpuProfile).add
	0x7f814c8a93b0 runtime.(*cpuProfile).addNonGo
	0x7f814c8a94b0 runtime.(*cpuProfile).addExtra
	0x7f814c8a96b0 runtime.debugCallCheck
	0x7f814c8a9790 runtime.debugCallWrap
	0x7f814c8a9840 runtime.gogetenv
	0x7f814c8a9990 runtime.(*TypeAssertionError).Error
	0x7f814c8a9de0 runtime.errorString.Error
	0x7f814c8a9e60 runtime.plainError.Error
	0x7f814c8a9e80 runtime.printany
	0x7f814c8aa680 runtime.panicwrap
	0x7f814c8aaa70 runtime.Callers
	0x7f814c8aaaf0 runtime.GOROOT
	0x7f814c8aab80 runtime.float64frombits
	0x7f814c8aab90 runtime.memhash
	0x7f814c8aaea0 runtime.memhash32
	0x7f814c8aaf10 runtime.memhash64
	0x7f814c8aaf80 runtime.getitab
	0x7f814c8ab310 runtime.(*itabTableType).find
	0x7f814c8ab370 runtime.itabAdd
	0x7f814c8ab4d0 runtime.(*itabTableType).add
	0x7f814c8ab530 runtime.(*itab).init
	0x7f814c8ab9a0 runtime.itabsinit
	0x7f814c8aba90 runtime.panicdottypeI
	0x7f814c8abba0 runtime.convT64
	0x7f814c8abc20 runtime.convTstring
	0x7f814c8abcc0 runtime.assertE2I
	0x7f814c8abda0 runtime.assertE2I2
	0x7f814c8abe50 runtime.iterate_itabs
	0x7f814c8abed0 runtime.(*lfstack).push
	0x7f814c8ac030 runtime.(*lfstack).pop
	0x7f814c8ac070 runtime.lfnodeValidate
	0x7f814c8ac130 runtime.lock
	0x7f814c8ac2e0 runtime.unlock
	0x7f814c8ac3b0 runtime.notewakeup
	0x7f814c8ac480 runtime.notesleep
	0x7f814c8ac580 runtime.notetsleep_internal
	0x7f814c8ac740 runtime.notetsleep
	0x7f814c8ac7d0 runtime.notetsleepg
	0x7f814c8ac870 runtime.mallocinit
	0x7f814c8acad0 runtime.(*mheap).sysAlloc
	0x7f814c8ad2d0 runtime.sysReserveAligned
	0x7f814c8ad440 runtime.(*mcache).nextFree
	0x7f814c8ad680 runtime.mallocgc
	0x7f814c8ae020 runtime.largeAlloc
	0x7f814c8ae1c0 runtime.newobject
	0x7f814c8ae220 runtime.newarray
	0x7f814c8ae2f0 runtime.profilealloc
	0x7f814c8ae360 runtime.fastexprand
	0x7f814c8ae4e0 runtime.persistentalloc
	0x7f814c8ae590 runtime.persistentalloc1
	0x7f814c8ae8c0 runtime.inPersistentAlloc
	0x7f814c8ae900 runtime.(*linearAlloc).alloc
	0x7f814c8ae9e0 runtime.(*hmap).incrnoverflow
	0x7f814c8aea70 runtime.(*hmap).newoverflow
	0x7f814c8aed50 runtime.makemap_small
	0x7f814c8aedf0 runtime.makemap
	0x7f814c8aefd0 runtime.makeBucketArray
	0x7f814c8af210 runtime.mapaccess2
	0x7f814c8af4b0 runtime.mapassign
	0x7f814c8afac0 runtime.hashGrow
	0x7f814c8afcf0 runtime.growWork
	0x7f814c8afda0 runtime.evacuate
	0x7f814c8b03e0 runtime.advanceEvacuationMark
	0x7f814c8b04d0 runtime.mapaccess1_fast32
	0x7f814c8b0690 runtime.mapaccess2_fast32
	0x7f814c8b0860 runtime.mapassign_fast32
	0x7f814c8b0bf0 runtime.growWork_fast32
	0x7f814c8b0ca0 runtime.evacuate_fast32
	0x7f814c8b10d0 runtime.mapassign_faststr
	0x7f814c8b1530 runtime.growWork_faststr
	0x7f814c8b15e0 runtime.evacuate_faststr
	0x7f814c8b1a60 runtime.typedmemmove
	0x7f814c8b1b30 runtime.reflectcallmove
	0x7f814c8b1bd0 runtime.typedslicecopy
	0x7f814c8b1d10 runtime.typedmemclr
	0x7f814c8b1d80 runtime.memclrHasPointers
	0x7f814c8b1de0 runtime.(*mspan).refillAllocCache
	0x7f814c8b1e00 runtime.(*mspan).nextFreeIndex
	0x7f814c8b1f90 runtime.markBitsForAddr
	0x7f814c8b2080 runtime.findObject
	0x7f814c8b2450 runtime.heapBits.nextArena
	0x7f814c8b24c0 runtime.heapBits.forward
	0x7f814c8b2580 runtime.heapBits.forwardOrBoundary
	0x7f814c8b2640 runtime.bulkBarrierPreWrite
	0x7f814c8b2b00 runtime.bulkBarrierPreWriteSrcOnly
	0x7f814c8b2d00 runtime.bulkBarrierBitmap
	0x7f814c8b2ea0 runtime.typeBitsBulkBarrier
	0x7f814c8b3160 runtime.heapBits.initSpan
	0x7f814c8b3370 runtime.heapBits.initCheckmarkSpan
	0x7f814c8b34b0 runtime.heapBits.clearCheckmarkSpan
	0x7f814c8b3560 runtime.(*mspan).countAlloc
	0x7f814c8b35e0 runtime.heapBitsSetType
	0x7f814c8b4040 runtime.heapBitsSetTypeGCProg
	0x7f814c8b43b0 runtime.progToPointerMask
	0x7f814c8b44d0 runtime.runGCProg
	0x7f814c8b4b30 runtime.materializeGCProg
	0x7f814c8b4be0 runtime.allocmcache
	0x7f814c8b4ca0 runtime.freemcache
	0x7f814c8b4d00 runtime.(*mcache).refill
	0x7f814c8b4e40 runtime.(*mcache).releaseAll
	0x7f814c8b4ee0 runtime.(*mcache).prepareForSweep
	0x7f814c8b4ff0 runtime.(*mcentral).cacheSpan
	0x7f814c8b5450 runtime.(*mcentral).uncacheSpan
	0x7f814c8b55b0 runtime.(*mcentral).freeSpan
	0x7f814c8b5750 runtime.(*mcentral).grow
	0x7f814c8b58d0 runtime.sysAlloc
	0x7f814c8b59e0 runtime.sysUnused
	0x7f814c8b5bd0 runtime.sysUsed
	0x7f814c8b5c50 runtime.sysFree
	0x7f814c8b5ca0 runtime.sysMap
	0x7f814c8b5d80 runtime.queuefinalizer
	0x7f814c8b5fd0 runtime.wakefing
	0x7f814c8b6060 runtime.createfing
	0x7f814c8b60d0 runtime.runfinq
	0x7f814c8b6500 runtime.SetFinalizer
	0x7f814c8b6d10 runtime.(*fixalloc).alloc
	0x7f814c8b6e70 runtime.gcinit
	0x7f814c8b6f50 runtime.readgogc
	0x7f814c8b7020 runtime.gcenable
	0x7f814c8b70b0 runtime/debug.setGCPercent
	0x7f814c8b7180 runtime.(*gcControllerState).startCycle
	0x7f814c8b7440 runtime.(*gcControllerState).revise
	0x7f814c8b7550 runtime.(*gcControllerState).endCycle
	0x7f814c8b7a60 runtime.(*gcControllerState).enlistWorker
	0x7f814c8b7bf0 runtime.(*gcControllerState).findRunnableGCWorker
	0x7f814c8b7e50 runtime.pollFractionalWorkerExit
	0x7f814c8b7f10 runtime.gcSetTriggerRatio
	0x7f814c8b82e0 runtime.gcWaitOnMark
	0x7f814c8b83b0 runtime.gcStart
	0x7f814c8b8a40 runtime.gcMarkDone
	0x7f814c8b8d10 runtime.gcMarkTermination
	0x7f814c8b9840 runtime.gcBgMarkStartWorkers
	0x7f814c8b9910 runtime.gcBgMarkWorker
	0x7f814c8b9de0 runtime.gcMark
	0x7f814c8ba360 runtime.gcSweep
	0x7f814c8ba550 runtime.gcResetMarkState
	0x7f814c8ba6a0 sync.runtime_registerPoolCleanup
	0x7f814c8ba700 runtime.clearpools
	0x7f814c8ba870 runtime.itoaDiv
	0x7f814c8ba950 runtime.fmtNSAsMS
	0x7f814c8baae0 runtime.(*treapNode).pred
	0x7f814c8babf0 runtime.(*treapNode).succ
	0x7f814c8bad00 runtime.treapIter.prev
	0x7f814c8bad50 runtime.(*mTreap).start
	0x7f814c8bad80 runtime.(*mTreap).end
	0x7f814c8badb0 runtime.(*mTreap).insert
	0x7f814c8bb080 runtime.(*mTreap).removeNode
	0x7f814c8bb1a0 runtime.(*mTreap).find
	0x7f814c8bb250 runtime.(*mTreap).removeSpan
	0x7f814c8bb2e0 runtime.(*mTreap).erase
	0x7f814c8bb330 runtime.(*mTreap).rotateLeft
	0x7f814c8bb3f0 runtime.(*mTreap).rotateRight
	0x7f814c8bb4b0 runtime.gcMarkRootPrepare
	0x7f814c8bb600 runtime.gcMarkRootCheck
	0x7f814c8bb850 runtime.markroot
	0x7f814c8bbbc0 runtime.markrootBlock
	0x7f814c8bbc80 runtime.markrootFreeGStacks
	0x7f814c8bbdb0 runtime.markrootSpans
	0x7f814c8bbfe0 runtime.gcAssistAlloc
	0x7f814c8bc2b0 runtime.gcAssistAlloc1
	0x7f814c8bc600 runtime.gcWakeAllAssists
	0x7f814c8bc680 runtime.gcParkAssist
	0x7f814c8bc7d0 runtime.gcFlushBgCredit
	0x7f814c8bc960 runtime.scanstack
	0x7f814c8bd030 runtime.scanframeworker
	0x7f814c8bd210 runtime.gcDrain
	0x7f814c8bd650 runtime.gcDrainN
	0x7f814c8bd890 runtime.scanblock
	0x7f814c8bda10 runtime.scanobject
	0x7f814c8bddc0 runtime.shade
	0x7f814c8bde70 runtime.greyobject
	0x7f814c8be430 runtime.gcDumpObject
	0x7f814c8be8d0 runtime.gcmarknewobject
	0x7f814c8be960 runtime.gcMarkTinyAllocs
	0x7f814c8bea50 runtime.initCheckmarks
	0x7f814c8beb90 runtime.clearCheckmarks
	0x7f814c8becd0 runtime.init.1
	0x7f814c8bece0 runtime.(*stackScanState).putPtr
	0x7f814c8bee00 runtime.(*stackScanState).getPtr
	0x7f814c8bef10 runtime.(*stackScanState).addObject
	0x7f814c8bf080 runtime.binarySearchTree
	0x7f814c8bf1e0 runtime.(*stackScanState).findObject
	0x7f814c8bf230 runtime.finishsweep_m
	0x7f814c8bf280 runtime.bgsweep
	0x7f814c8bf3f0 runtime.sweepone
	0x7f814c8bf7e0 runtime.(*mspan).ensureSwept
	0x7f814c8bf8e0 runtime.(*mspan).sweep
	0x7f814c8c0290 runtime.deductSweepCredit
	0x7f814c8c03f0 runtime.clobberfree
	0x7f814c8c0410 runtime.(*gcSweepBuf).push
	0x7f814c8c05d0 runtime.(*gcSweepBuf).block
	0x7f814c8c06b0 runtime.init.2
	0x7f814c8c06c0 runtime.(*gcWork).init
	0x7f814c8c0730 runtime.(*gcWork).put
	0x7f814c8c0840 runtime.(*gcWork).putBatch
	0x7f814c8c0a10 runtime.(*gcWork).tryGet
	0x7f814c8c0b00 runtime.(*gcWork).dispose
	0x7f814c8c0bf0 runtime.(*gcWork).balance
	0x7f814c8c0cf0 runtime.(*workbuf).checknonempty
	0x7f814c8c0d50 runtime.(*workbuf).checkempty
	0x7f814c8c0db0 runtime.getempty
	0x7f814c8c0fe0 runtime.putempty
	0x7f814c8c1040 runtime.putfull
	0x7f814c8c10a0 runtime.trygetfull
	0x7f814c8c1120 runtime.handoff
	0x7f814c8c11e0 runtime.prepareFreeWorkbufs
	0x7f814c8c1290 runtime.freeSomeWbufs
	0x7f814c8c1370 runtime.(*mheap).coalesce
	0x7f814c8c15e0 runtime.(*mspan).scavenge
	0x7f814c8c16a0 runtime.recordspan
	0x7f814c8c1850 runtime.inHeapOrStack
	0x7f814c8c18e0 runtime.spanOfHeap
	0x7f814c8c1960 runtime.(*mheap).init
	0x7f814c8c1dc0 runtime.(*mheap).reclaim
	0x7f814c8c2020 runtime.(*mheap).reclaimChunk
	0x7f814c8c2360 runtime.(*mheap).alloc_m
	0x7f814c8c2670 runtime.(*mheap).alloc
	0x7f814c8c2760 runtime.(*mheap).allocManual
	0x7f814c8c2840 runtime.(*mheap).setSpans
	0x7f814c8c2910 runtime.(*mheap).pickFreeSpan
	0x7f814c8c2a50 runtime.(*mheap).allocSpanLocked
	0x7f814c8c2ed0 runtime.(*mheap).grow
	0x7f814c8c30e0 runtime.(*mheap).freeSpan
	0x7f814c8c3160 runtime.(*mheap).freeManual
	0x7f814c8c3210 runtime.(*mheap).freeSpanLocked
	0x7f814c8c3500 runtime.(*mheap).scavengeLargest
	0x7f814c8c3690 runtime.(*mheap).scavengeAll
	0x7f814c8c37e0 runtime.(*mheap).scavenge
	0x7f814c8c3a80 runtime.(*mSpanList).remove
	0x7f814c8c3c10 runtime.(*mSpanList).insert
	0x7f814c8c3d30 runtime.(*mSpanList).insertBack
	0x7f814c8c3e50 runtime.(*mSpanList).takeAll
	0x7f814c8c3eb0 runtime.addspecial
	0x7f814c8c4080 runtime.removespecial
	0x7f814c8c4230 runtime.addfinalizer
	0x7f814c8c44a0 runtime.removefinalizer
	0x7f814c8c4550 runtime.setprofilebucket
	0x7f814c8c4610 runtime.freespecial
	0x7f814c8c4760 runtime.newMarkBits
	0x7f814c8c4aa0 runtime.nextMarkBitArenaEpoch
	0x7f814c8c4b60 runtime.newArenaMayUnlock
	0x7f814c8c4c50 runtime.newBucket
	0x7f814c8c4d20 runtime.(*bucket).mp
	0x7f814c8c4d90 runtime.(*bucket).bp
	0x7f814c8c4e10 runtime.stkbucket
	0x7f814c8c5150 runtime.eqslice
	0x7f814c8c51a0 runtime.mProf_NextCycle
	0x7f814c8c5220 runtime.mProf_Flush
	0x7f814c8c5290 runtime.mProf_FlushLocked
	0x7f814c8c5370 runtime.mProf_Malloc
	0x7f814c8c5540 runtime.mProf_Free
	0x7f814c8c5600 runtime.blockevent
	0x7f814c8c5680 runtime.blocksampled
	0x7f814c8c5710 runtime.saveblockevent
	0x7f814c8c58c0 sync.event
	0x7f814c8c59a0 runtime.tracealloc
	0x7f814c8c5c20 runtime.tracefree
	0x7f814c8c5d80 runtime.tracegc
	0x7f814c8c5e70 runtime.init.3
	0x7f814c8c5f00 runtime.cachestats
	0x7f814c8c5f80 runtime.flushmcache
	0x7f814c8c6000 runtime.purgecachedstats
	0x7f814c8c60a0 runtime.mSysStatInc
	0x7f814c8c6160 runtime.mSysStatDec
	0x7f814c8c6220 runtime.(*wbBuf).reset
	0x7f814c8c62b0 runtime.wbBufFlush
	0x7f814c8c63b0 runtime.wbBufFlush1
	0x7f814c8c65f0 internal/poll.runtime_pollServerInit
	0x7f814c8c6630 internal/poll.runtime_pollOpen
	0x7f814c8c6770 internal/poll.runtime_pollClose
	0x7f814c8c6890 runtime.(*pollCache).free
	0x7f814c8c68f0 internal/poll.runtime_pollUnblock
	0x7f814c8c6ae0 runtime.netpollready
	0x7f814c8c6bd0 runtime.netpollunblock
	0x7f814c8c6c70 runtime.(*pollCache).alloc
	0x7f814c8c6d40 runtime.netpollinit
	0x7f814c8c6e20 runtime.netpollopen
	0x7f814c8c6eb0 runtime.netpoll
	0x7f814c8c70f0 runtime.futexsleep
	0x7f814c8c71c0 runtime.futexwakeup
	0x7f814c8c7260 runtime.getproccount
	0x7f814c8c7360 runtime.newosproc
	0x7f814c8c7530 runtime.newosproc0
	0x7f814c8c7630 runtime.sysargs
	0x7f814c8c78b0 runtime.sysauxv
	0x7f814c8c79c0 runtime.osinit
	0x7f814c8c7a00 runtime.getRandomData
	0x7f814c8c7b60 runtime.libpreinit
	0x7f814c8c7b90 runtime.minit
	0x7f814c8c7be0 runtime.setsig
	0x7f814c8c7ca0 runtime.setsigstack
	0x7f814c8c7d30 runtime.sysSigaction
	0x7f814c8c7da0 runtime.panicCheckMalloc
	0x7f814c8c7e50 runtime.panicindex
	0x7f814c8c7f80 runtime.panicslice
	0x7f814c8c80b0 runtime.panicdivide
	0x7f814c8c8120 runtime.throwinit
	0x7f814c8c8170 runtime.deferproc
	0x7f814c8c82a0 runtime.testdefersizes
	0x7f814c8c84f0 runtime.init.4
	0x7f814c8c8570 runtime.newdefer
	0x7f814c8c8790 runtime.freedefer
	0x7f814c8c89a0 runtime.freedeferpanic
	0x7f814c8c89f0 runtime.freedeferfn
	0x7f814c8c8a40 runtime.deferreturn
	0x7f814c8c8b60 runtime.preprintpanics
	0x7f814c8c8d30 runtime.printpanics
	0x7f814c8c8e30 runtime.gopanic
	0x7f814c8c93a0 runtime.getargp
	0x7f814c8c93b0 runtime.gorecover
	0x7f814c8c9400 sync.throw
	0x7f814c8c9440 runtime.throw
	0x7f814c8c94d0 runtime.recovery
	0x7f814c8c9630 runtime.fatalthrow
	0x7f814c8c96a0 runtime.fatalpanic
	0x7f814c8c9750 runtime.startpanic_m
	0x7f814c8c9900 runtime.dopanic_m
	0x7f814c8c9c90 runtime.shouldPushSigpanic
	0x7f814c8c9d50 runtime.recordForPanic
	0x7f814c8c9ea0 runtime.printlock
	0x7f814c8c9f20 runtime.printunlock
	0x7f814c8c9f90 runtime.gwrite
	0x7f814c8ca0e0 runtime.printsp
	0x7f814c8ca130 runtime.printnl
	0x7f814c8ca180 runtime.printbool
	0x7f814c8ca1f0 runtime.printfloat
	0x7f814c8ca490 runtime.printcomplex
	0x7f814c8ca520 runtime.printuint
	0x7f814c8ca620 runtime.printint
	0x7f814c8ca690 runtime.printhex
	0x7f814c8ca7a0 runtime.printpointer
	0x7f814c8ca7e0 runtime.printstring
	0x7f814c8ca870 runtime.printslice
	0x7f814c8ca940 runtime.printeface
	0x7f814c8ca9f0 runtime.hexdumpWords
	0x7f814c8cac40 runtime.main
	0x7f814c8cafd0 runtime.init.5
	0x7f814c8cb020 runtime.forcegchelper
	0x7f814c8cb150 runtime.gopark
	0x7f814c8cb2a0 runtime.goready
	0x7f814c8cb310 runtime.acquireSudog
	0x7f814c8cb6a0 runtime.releaseSudog
	0x7f814c8cba30 runtime.funcPC
	0x7f814c8cba40 runtime.badmcall
	0x7f814c8cba90 runtime.badmcall2
	0x7f814c8cbae0 runtime.badreflectcall
	0x7f814c8cbb30 runtime.badmorestackg0
	0x7f814c8cbb70 runtime.badmorestackgsignal
	0x7f814c8cbbb0 runtime.badctxt
	0x7f814c8cbbe0 runtime.allgadd
	0x7f814c8cbd30 runtime.cpuinit
	0x7f814c8cbec0 runtime.schedinit
	0x7f814c8cc120 runtime.checkmcount
	0x7f814c8cc1e0 runtime.mcommoninit
	0x7f814c8cc410 runtime.ready
	0x7f814c8cc6d0 runtime.freezetheworld
	0x7f814c8cc780 runtime.casfrom_Gscanstatus
	0x7f814c8ccbe0 runtime.castogscanstatus
	0x7f814c8cccc0 runtime.casgstatus
	0x7f814c8ccec0 runtime.scang
	0x7f814c8cd250 runtime.restartg
	0x7f814c8cd430 runtime.stopTheWorldWithSema
	0x7f814c8cd6f0 runtime.startTheWorldWithSema
	0x7f814c8cd970 runtime.mstart
	0x7f814c8cda00 runtime.mstart1
	0x7f814c8cdb10 runtime.mstartm0
	0x7f814c8cdb70 runtime.mexit
	0x7f814c8cdde0 runtime.forEachP
	0x7f814c8ce170 runtime.runSafePointFn
	0x7f814c8ce240 runtime.allocm
	0x7f814c8ce4f0 runtime.needm
	0x7f814c8ce680 runtime.newextram
	0x7f814c8ce710 runtime.oneNewExtraM
	0x7f814c8ce8c0 runtime.dropm
	0x7f814c8cea00 runtime.lockextra
	0x7f814c8ceac0 runtime.newm
	0x7f814c8cec10 runtime.newm1
	0x7f814c8ced30 runtime.templateThread
	0x7f814c8cee50 runtime.stopm
	0x7f814c8cefc0 runtime.mspinning
	0x7f814c8cefe0 runtime.startm
	0x7f814c8cf200 runtime.handoffp
	0x7f814c8cf510 runtime.stoplockedm
	0x7f814c8cf7d0 runtime.startlockedm
	0x7f814c8cf8b0 runtime.gcstopm
	0x7f814c8cf9d0 runtime.execute
	0x7f814c8cfb40 runtime.findrunnable
	0x7f814c8d0740 runtime.pollWork
	0x7f814c8d0830 runtime.resetspinning
	0x7f814c8d0910 runtime.injectglist
	0x7f814c8d0aa0 runtime.schedule
	0x7f814c8d0ef0 runtime.parkunlock_c
	0x7f814c8d0f40 runtime.park_m
	0x7f814c8d1110 runtime.goschedImpl
	0x7f814c8d1380 runtime.gosched_m
	0x7f814c8d1410 runtime.gopreempt_m
	0x7f814c8d14a0 runtime.goexit1
	0x7f814c8d1510 runtime.goexit0
	0x7f814c8d17e0 runtime.save
	0x7f814c8d1830 runtime.reentersyscall
	0x7f814c8d1a70 runtime.entersyscall
	0x7f814c8d1aa0 runtime.entersyscall_sysmon
	0x7f814c8d1b20 runtime.entersyscall_gcwait
	0x7f814c8d1c20 runtime.entersyscallblock
	0x7f814c8d1dd0 runtime.entersyscallblock_handoff
	0x7f814c8d1e60 runtime.exitsyscall
	0x7f814c8d20f0 runtime.exitsyscallfast
	0x7f814c8d21f0 runtime.exitsyscallfast_reacquired
	0x7f814c8d2280 runtime.exitsyscallfast_pidle
	0x7f814c8d2370 runtime.exitsyscall0
	0x7f814c8d2560 runtime.malg
	0x7f814c8d2630 runtime.newproc
	0x7f814c8d26b0 runtime.newproc1
	0x7f814c8d2bc0 runtime.saveAncestors
	0x7f814c8d2f10 runtime.gfput
	0x7f814c8d3080 runtime.gfget
	0x7f814c8d3210 runtime.gfpurge
	0x7f814c8d32e0 runtime.unlockOSThread
	0x7f814c8d3370 runtime.badunlockosthread
	0x7f814c8d33c0 runtime._System
	0x7f814c8d3400 runtime._ExternalCode
	0x7f814c8d3440 runtime._LostExternalCode
	0x7f814c8d3480 runtime._GC
	0x7f814c8d34c0 runtime._VDSO
	0x7f814c8d3500 runtime.sigprof
	0x7f814c8d39f0 runtime.sigprofNonGo
	0x7f814c8d3a80 runtime.sigprofNonGoPC
	0x7f814c8d3b00 runtime.setsSP
	0x7f814c8d3b90 runtime.procresize
	0x7f814c8d4600 runtime.acquirep
	0x7f814c8d4660 runtime.wirep
	0x7f814c8d47d0 runtime.releasep
	0x7f814c8d4a00 runtime.incidlelocked
	0x7f814c8d4a70 runtime.checkdead
	0x7f814c8d4ef0 runtime.sysmon
	0x7f814c8d5410 runtime.retake
	0x7f814c8d5650 runtime.preemptall
	0x7f814c8d56d0 runtime.schedtrace
	0x7f814c8d61b0 runtime.schedEnableUser
	0x7f814c8d6300 runtime.schedEnabled
	0x7f814c8d6370 runtime.globrunqget
	0x7f814c8d64c0 runtime.pidleput
	0x7f814c8d6550 runtime.runqempty
	0x7f814c8d6590 runtime.runqput
	0x7f814c8d6690 runtime.runqputslow
	0x7f814c8d68c0 runtime.runqget
	0x7f814c8d6970 runtime.runqgrab
	0x7f814c8d6ad0 runtime.runqsteal
	0x7f814c8d6bd0 sync.runtime_canSpin
	0x7f814c8d6c60 sync.runtime_doSpin
	0x7f814c8d6c90 runtime.(*randomOrder).reset
	0x7f814c8d6db0 runtime.gcd
	0x7f814c8d6dd0 runtime.(*profBuf).takeOverflow
	0x7f814c8d6e40 runtime.(*profBuf).incrementOverflow
	0x7f814c8d6e90 runtime.(*profBuf).canWriteRecord
	0x7f814c8d6f40 runtime.(*profBuf).canWriteTwoRecords
	0x7f814c8d7040 runtime.(*profBuf).write
	0x7f814c8d74c0 runtime.(*profBuf).wakeupExtra
	0x7f814c8d7540 syscall.runtime_envs
	0x7f814c8d7640 os.runtime_args
	0x7f814c8d7740 runtime.args
	0x7f814c8d77b0 runtime.goargs
	0x7f814c8d7900 runtime.goenvs_unix
	0x7f814c8d7a50 runtime.testAtomic64
	0x7f814c8d7cb0 runtime.check
	0x7f814c8d8050 runtime.parsedebugvars
	0x7f814c8d83c0 runtime/debug.SetTraceback
	0x7f814c8d8570 runtime.timediv
	0x7f814c8d85d0 internal/reflectlite.resolveNameOff
	0x7f814c8d8620 internal/reflectlite.resolveTypeOff
	0x7f814c8d8680 runtime.extendRandom
	0x7f814c8d8770 runtime.waitReason.String
	0x7f814c8d87e0 runtime.(*rwmutex).rlock
	0x7f814c8d8870 runtime.(*rwmutex).runlock
	0x7f814c8d8970 internal/poll.runtime_Semacquire
	0x7f814c8d89d0 sync.runtime_Semrelease
	0x7f814c8d8a30 sync.runtime_SemacquireMutex
	0x7f814c8d8a90 internal/poll.runtime_Semrelease
	0x7f814c8d8ae0 runtime.readyWithTime
	0x7f814c8d8b50 runtime.semacquire1
	0x7f814c8d8e30 runtime.semrelease1
	0x7f814c8d9010 runtime.cansemacquire
	0x7f814c8d9040 runtime.(*semaRoot).queue
	0x7f814c8d9490 runtime.(*semaRoot).dequeue
	0x7f814c8d9820 runtime.(*semaRoot).rotateLeft
	0x7f814c8d9a30 runtime.(*semaRoot).rotateRight
	0x7f814c8d9c20 sync.runtime_notifyListCheck
	0x7f814c8d9ce0 sync.runtime_nanotime
	0x7f814c8d9d20 runtime.dumpregs
	0x7f814c8da410 runtime.(*sigctxt).preparePanic
	0x7f814c8da4e0 runtime.sighandler
	0x7f814c8dac40 runtime.init.6
	0x7f814c8dac50 runtime.initsig
	0x7f814c8dae50 runtime.setThreadCPUProfiler
	0x7f814c8daf30 runtime.sigtrampgo
	0x7f814c8db3d0 runtime.sigpanic
	0x7f814c8db890 runtime.dieFromSignal
	0x7f814c8db930 runtime.raisebadsignal
	0x7f814c8dba50 runtime.crash
	0x7f814c8dba80 runtime.noSignalStack
	0x7f814c8dbb10 runtime.sigNotOnStack
	0x7f814c8dbba0 runtime.signalDuringFork
	0x7f814c8dbc30 runtime.badsignal
	0x7f814c8dbc90 runtime.sigfwdgo
	0x7f814c8dbe50 runtime.unblocksig
	0x7f814c8dbee0 runtime.minitSignals
	0x7f814c8dbf20 runtime.minitSignalStack
	0x7f814c8dc080 runtime.minitSignalMask
	0x7f814c8dc190 runtime.unminitSignals
	0x7f814c8dc230 runtime.signalstack
	0x7f814c8dc290 runtime.sigsend
	0x7f814c8dc420 runtime.makeslice
	0x7f814c8dc520 runtime.growslice
	0x7f814c8dcb60 runtime.stackinit
	0x7f814c8dcbc0 runtime.stacklog2
	0x7f814c8dcbe0 runtime.stackpoolalloc
	0x7f814c8dcdc0 runtime.stackpoolfree
	0x7f814c8dcf50 runtime.stackcacherefill
	0x7f814c8dd040 runtime.stackcacherelease
	0x7f814c8dd160 runtime.stackcache_clear
	0x7f814c8dd240 runtime.stackalloc
	0x7f814c8dd560 runtime.stackfree
	0x7f814c8dd8b0 runtime.adjustpointers
	0x7f814c8ddaf0 runtime.adjustframe
	0x7f814c8ddf50 runtime.adjustctxt
	0x7f814c8ddfa0 runtime.adjustdefers
	0x7f814c8de070 runtime.adjustsudogs
	0x7f814c8de0b0 runtime.findsghi
	0x7f814c8de100 runtime.syncadjustsudogs
	0x7f814c8de250 runtime.copystack
	0x7f814c8de580 runtime.round2
	0x7f814c8de5b0 runtime.newstack
	0x7f814c8df2e0 runtime.shrinkstack
	0x7f814c8df460 runtime.freeStackSpans
	0x7f814c8df600 runtime.getStackMap
	0x7f814c8dfeb0 runtime.morestackc
	0x7f814c8dfee0 runtime.concatstrings
	0x7f814c8e01c0 runtime.concatstring2
	0x7f814c8e0230 runtime.concatstring3
	0x7f814c8e02b0 runtime.concatstring4
	0x7f814c8e0330 runtime.concatstring5
	0x7f814c8e03b0 runtime.slicebytetostring
	0x7f814c8e04a0 runtime.rawstringtmp
	0x7f814c8e0550 runtime.rawstring
	0x7f814c8e05e0 runtime.gostring
	0x7f814c8e06b0 runtime.index
	0x7f814c8e07b0 runtime.atoi
	0x7f814c8e08e0 runtime.findnull
	0x7f814c8e09a0 runtime.badsystemstack
	0x7f814c8e09e0 runtime.fastrand
	0x7f814c8e0a30 runtime.modulesinit
	0x7f814c8e0cb0 runtime.moduledataverify
	0x7f814c8e0d10 runtime.moduledataverify1
	0x7f814c8e13a0 runtime.findmoduledatap
	0x7f814c8e13e0 runtime.findfunc
	0x7f814c8e1560 runtime.pcvalue
	0x7f814c8e1b20 runtime.funcname
	0x7f814c8e1bd0 runtime.funcnameFromNameoff
	0x7f814c8e1c80 runtime.funcline1
	0x7f814c8e1e50 runtime.funcline
	0x7f814c8e1ed0 runtime.funcspdelta
	0x7f814c8e2040 runtime.pcdatavalue
	0x7f814c8e20f0 runtime.funcdata
	0x7f814c8e21d0 runtime.step
	0x7f814c8e2380 runtime.readvarint
	0x7f814c8e23e0 runtime.deltimer
	0x7f814c8e24b0 runtime.(*timersBucket).deltimerLocked
	0x7f814c8e2650 runtime.timejump
	0x7f814c8e2720 runtime.timejumpLocked
	0x7f814c8e27c0 runtime.siftupTimer
	0x7f814c8e28e0 runtime.siftdownTimer
	0x7f814c8e2ab0 time.now
	0x7f814c8e2b20 runtime.traceReader
	0x7f814c8e2c20 runtime.traceProcFree
	0x7f814c8e2ce0 runtime.traceEvent
	0x7f814c8e2de0 runtime.traceEventLocked
	0x7f814c8e3090 runtime.traceStackID
	0x7f814c8e31d0 runtime.traceAcquireBuffer
	0x7f814c8e3290 runtime.traceReleaseBuffer
	0x7f814c8e3310 runtime.traceFlush
	0x7f814c8e3500 runtime.(*traceBuf).varint
	0x7f814c8e3580 runtime.(*traceStackTable).put
	0x7f814c8e3780 runtime.(*traceStackTable).find
	0x7f814c8e3830 runtime.(*traceStackTable).newStack
	0x7f814c8e3890 runtime.(*traceAlloc).alloc
	0x7f814c8e39b0 runtime.traceProcStart
	0x7f814c8e3a40 runtime.traceProcStop
	0x7f814c8e3b20 runtime.traceGCSweepStart
	0x7f814c8e3ba0 runtime.traceGCSweepSpan
	0x7f814c8e3c40 runtime.traceGCSweepDone
	0x7f814c8e3d20 runtime.traceGoCreate
	0x7f814c8e3e20 runtime.traceGoStart
	0x7f814c8e3fc0 runtime.traceGoPark
	0x7f814c8e4050 runtime.traceGoUnpark
	0x7f814c8e4150 runtime.traceGoSysCall
	0x7f814c8e41b0 runtime.traceGoSysExit
	0x7f814c8e4290 runtime.traceGoSysBlock
	0x7f814c8e4370 runtime.traceNextGC
	0x7f814c8e4430 runtime.tracebackdefers
	0x7f814c8e4660 runtime.gentraceback
	0x7f814c8e63a0 runtime.getArgInfo
	0x7f814c8e6570 runtime.tracebackCgoContext
	0x7f814c8e67e0 runtime.printcreatedby
	0x7f814c8e68b0 runtime.printcreatedby1
	0x7f814c8e6a70 runtime.traceback
	0x7f814c8e6ae0 runtime.tracebacktrap
	0x7f814c8e6ba0 runtime.traceback1
	0x7f814c8e6ec0 runtime.printAncestorTraceback
	0x7f814c8e70c0 runtime.printAncestorTracebackFuncInfo
	0x7f814c8e7400 runtime.callers
	0x7f814c8e74e0 runtime.gcallers
	0x7f814c8e7590 runtime.showframe
	0x7f814c8e7660 runtime.showfuncinfo
	0x7f814c8e7870 runtime.goroutineheader
	0x7f814c8e7b50 runtime.tracebackothers
	0x7f814c8e7e00 runtime.tracebackHexdump
	0x7f814c8e7ff0 runtime.isSystemGoroutine
	0x7f814c8e8100 runtime.printCgoTraceback
	0x7f814c8e8240 runtime.printOneCgoTraceback
	0x7f814c8e8480 runtime.callCgoSymbolizer
	0x7f814c8e8500 runtime.cgoContextPCs
	0x7f814c8e85e0 runtime.(*_type).string
	0x7f814c8e86a0 runtime.(*_type).uncommon
	0x7f814c8e8750 runtime.(*_type).pkgpath
	0x7f814c8e8860 runtime.resolveNameOff
	0x7f814c8e8b60 runtime.resolveTypeOff
	0x7f814c8e8eb0 runtime.(*_type).textOff
	0x7f814c8e91f0 runtime.name.tagLen
	0x7f814c8e9250 runtime.name.name
	0x7f814c8e92a0 runtime.name.tag
	0x7f814c8e9330 runtime.name.pkgPath
	0x7f814c8e9430 runtime.typelinksinit
	0x7f814c8e9ad0 runtime.typesEqual
	0x7f814c8eaa10 runtime.vdsoInitFromSysinfoEhdr
	0x7f814c8eae60 runtime.vdsoFindVersion
	0x7f814c8eafa0 runtime.vdsoParseSymbols
	0x7f814c8eb330 runtime.vdsoauxv
	0x7f814c8eb410 runtime.inVDSOPage
	0x7f814c8eb480 runtime.mmap.func1
	0x7f814c8eb500 runtime.munmap.func1
	0x7f814c8eb550 runtime.sigaction.func1
	0x7f814c8eb5c0 runtime.cgocallbackg1.func1
	0x7f814c8eb5d0 runtime.cgoCheckWriteBarrier.func1
	0x7f814c8eb690 runtime.cgoCheckTypedBlock.func1
	0x7f814c8eb6f0 runtime.chansend.func1
	0x7f814c8eb740 runtime.chanrecv.func1
	0x7f814c8eb790 runtime.debugCallCheck.func1
	0x7f814c8eb990 runtime.debugCallWrap.func1
	0x7f814c8eba00 runtime.mallocgc.func1
	0x7f814c8eba70 runtime.persistentalloc.func1
	0x7f814c8ebae0 runtime.freemcache.func1
	0x7f814c8ebba0 runtime.SetFinalizer.func1
	0x7f814c8ebbf0 runtime.SetFinalizer.func2
	0x7f814c8ebc80 runtime.gcStart.func1
	0x7f814c8ebcc0 runtime.gcStart.func2
	0x7f814c8ebd30 runtime.gcMarkDone.func1.1
	0x7f814c8ebdb0 runtime.gcMarkDone.func1
	0x7f814c8ebe50 runtime.gcMarkDone.func2
	0x7f814c8ebf30 runtime.gcMarkDone.func3
	0x7f814c8ebf80 runtime.gcMarkTermination.func1
	0x7f814c8ebfc0 runtime.gcMarkTermination.func2
	0x7f814c8ec0e0 runtime.gcMarkTermination.func3
	0x7f814c8ec120 runtime.gcMarkTermination.func4.1
	0x7f814c8ec170 runtime.gcMarkTermination.func4
	0x7f814c8ec1b0 runtime.gcBgMarkWorker.func1
	0x7f814c8ec230 runtime.gcBgMarkWorker.func2
	0x7f814c8ec3e0 runtime.markroot.func1
	0x7f814c8ec4d0 runtime.gcAssistAlloc.func1
	0x7f814c8ec520 runtime.scanstack.func1
	0x7f814c8ec580 runtime.getempty.func1
	0x7f814c8ec5f0 runtime.freeSomeWbufs.func1
	0x7f814c8ec6e0 runtime.(*mheap).coalesce.func1
	0x7f814c8ec900 runtime.(*mheap).coalesce.func2
	0x7f814c8ecae0 runtime.(*mheap).alloc.func1
	0x7f814c8ecb50 runtime.(*mheap).freeSpan.func1
	0x7f814c8ecc80 runtime.mProf_Malloc.func1
	0x7f814c8eccd0 runtime.tracealloc.func1
	0x7f814c8ecd30 runtime.tracefree.func1
	0x7f814c8ecd90 runtime.wbBufFlush.func1
	0x7f814c8ecde0 runtime.futexwakeup.func1
	0x7f814c8ece90 runtime.sysSigaction.func1
	0x7f814c8ecee0 runtime.newdefer.func1
	0x7f814c8ed0b0 runtime.newdefer.func2
	0x7f814c8ed200 runtime.freedefer.func1
	0x7f814c8ed380 runtime.preprintpanics.func1
	0x7f814c8ed3f0 runtime.throw.func1
	0x7f814c8ed480 runtime.fatalthrow.func1
	0x7f814c8ed510 runtime.fatalpanic.func1
	0x7f814c8ed5e0 runtime.fatalpanic.func2
	0x7f814c8ed620 runtime.hexdumpWords.func1
	0x7f814c8ed6c0 runtime.main.func1
	0x7f814c8ed710 runtime.main.func2
	0x7f814c8ed750 runtime.goready.func1
	0x7f814c8ed7a0 runtime.casgstatus.func1
	0x7f814c8ed860 runtime.casgstatus.func2
	0x7f814c8ed960 runtime.reentersyscall.func1
	0x7f814c8eda60 runtime.entersyscallblock.func1
	0x7f814c8edba0 runtime.entersyscallblock.func2
	0x7f814c8edce0 runtime.exitsyscallfast.func1
	0x7f814c8edd90 runtime.exitsyscallfast_reacquired.func1
	0x7f814c8eddf0 runtime.malg.func1
	0x7f814c8ede50 runtime.newproc.func1
	0x7f814c8edec0 runtime.gfget.func1
	0x7f814c8edf20 runtime.(*rwmutex).rlock.func1
	0x7f814c8edfe0 runtime.callers.func1
	0x7f814c8ee090 runtime.tracebackHexdump.func1
	0x7f814c8ee0d0 runtime.vdsoParseSymbols.func1
	0x7f814c8ee270 runtime.init.ializers
	0x7f814c8ee3f0 runtime.init
	0x7f814c8ee480 runtime.skipPleaseUseCallersFrames
	0x7f814c8ee580 _rt0_amd64_lib
	0x7f814c8ee630 _rt0_amd64_lib_go
	0x7f814c8ee650 runtime.rt0_go
	0x7f814c8ee7a0 runtime.asminit
	0x7f814c8ee7b0 runtime.gogo
	0x7f814c8ee810 runtime.mcall
	0x7f814c8ee870 runtime.systemstack_switch
	0x7f814c8ee880 runtime.systemstack
	0x7f814c8ee920 runtime.morestack
	0x7f814c8ee9b0 runtime.morestack_noctxt
	0x7f814c8ee9c0 runtime.reflectcall
	0x7f814c8eeba0 callRet
	0x7f814c8eebd0 runtime.call32
	0x7f814c8eec50 runtime.call64
	0x7f814c8eecd0 runtime.call128
	0x7f814c8eed80 runtime.call256
	0x7f814c8eee40 runtime.call512
	0x7f814c8eef00 runtime.call1024
	0x7f814c8eefc0 runtime.call2048
	0x7f814c8ef080 runtime.call4096
	0x7f814c8ef150 runtime.call8192
	0x7f814c8ef220 runtime.call16384
	0x7f814c8ef2f0 runtime.call32768
	0x7f814c8ef3c0 runtime.call65536
	0x7f814c8ef490 runtime.call131072
	0x7f814c8ef560 runtime.call262144
	0x7f814c8ef630 runtime.call524288
	0x7f814c8ef700 runtime.call1048576
	0x7f814c8ef7d0 runtime.call2097152
	0x7f814c8ef8a0 runtime.call4194304
	0x7f814c8ef970 runtime.call8388608
	0x7f814c8efa40 runtime.call16777216
	0x7f814c8efb10 runtime.call33554432
	0x7f814c8efbe0 runtime.call67108864
	0x7f814c8efcb0 runtime.call134217728
	0x7f814c8efd80 runtime.call268435456
	0x7f814c8efe50 runtime.call536870912
	0x7f814c8eff20 runtime.call1073741824
	0x7f814c8efff0 runtime.procyield
	0x7f814c8f0000 runtime.publicationBarrier
	0x7f814c8f0010 runtime.jmpdefer
	0x7f814c8f0030 gosave
	0x7f814c8f0070 runtime.asmcgocall
	0x7f814c8f0130 runtime.cgocallback
	0x7f814c8f0180 runtime.cgocallback_gofunc
	0x7f814c8f0280 runtime.setg
	0x7f814c8f02a0 setg_gcc
	0x7f814c8f02b0 runtime.abort
	0x7f814c8f02c0 runtime.stackcheck
	0x7f814c8f02f0 runtime.cputicks
	0x7f814c8f0310 runtime.aeshash
	0x7f814c8f0330 runtime.aeshashstr
	0x7f814c8f0350 runtime.aeshashbody
	0x7f814c8f08a0 runtime.aeshash32
	0x7f814c8f08e0 runtime.aeshash64
	0x7f814c8f0920 runtime.checkASM
	0x7f814c8f0940 runtime.return0
	0x7f814ca73920 _cgo_topofstack
fatal error: invalid runtime symbol table
runtime: panic before malloc heap initialized

runtime stack:
runtime.throw(0x7f814c90774d, 0x1c)
/home/u/go1.13/src/runtime/panic.go:627 +0x74 fp=0x7f814c61fd50 sp=0x7f814c61fd20 pc=0x7f814c8c94b4
runtime.moduledataverify1(0x7f814c9a9360)
/home/u/go1.13/src/runtime/symtab.go:448 +0x5a0 fp=0x7f814c61fe58 sp=0x7f814c61fd50 pc=0x7f814c8e12b0
runtime.moduledataverify()
/home/u/go1.13/src/runtime/symtab.go:416 +0x36 fp=0x7f814c61fe78 sp=0x7f814c61fe58 pc=0x7f814c8e0ce6
runtime.schedinit()
/home/u/go1.13/src/runtime/proc.go:537 +0x5c fp=0x7f814c61fed0 sp=0x7f814c61fe78 pc=0x7f814c8cbf1c
runtime.rt0_go(0x7ffe2435c0d8, 0x1, 0x7ffe2435c0d8, 0x7f814c620700, 0x7f814c62aa9d, 0x0, 0x7f814c620700, 0x7f814c620700, 0x9d89969ea2fc8472, 0x7ffe2435beee, ...)
/home/u/go1.13/src/runtime/asm_amd64.s:195 +0x11e fp=0x7f814c61fed8 sp=0x7f814c61fed0 pc=0x7f814c8ee76e

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Mar 18, 2019

With the current implementation it's not going to work to use -buildmode=c-archive multiple times and put the results into multiple shared libraries. As you've discovered. The essential problem is that there has to be only one Go runtime, but you have multiple runtimes. When using -buildmode=c-archive there's no way to isolate the different runtimes. I don't see any simple way to make this work.

@mewmew

This comment has been minimized.

Copy link
Contributor Author

commented Mar 18, 2019

The essential problem is that there has to be only one Go runtime, but you have multiple runtimes. When using -buildmode=c-archive there's no way to isolate the different runtimes. I don't see any simple way to make this work.

Hi @ianlancetaylor. Thanks for the input. One question, when we generate multiple shared libraries using -buildmode=c-shared, does each one contain the Go runtime? If so, how do shared libraries created with -buildmode=c-archive differ from those produced with -buildmode=c-shared?

I would assume there are cinit functions run in c-shared that are not run in c-archive. If so, are there any theoretical restrictions that make it impossible to implement support for the same initialization in shared libraries produced with c-archived as those produced with c-shared?

If there is a design document, or specific parts of the runtime source code I could attend to to gain further insight into this, I'd be very curious to learn more.

Kindly,
Robin

@katiehockman katiehockman added this to the Unplanned milestone Mar 19, 2019

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Mar 19, 2019

The -buildmode=c-shared libraries differ in that they are built with -Bsymbolic which forces all their local references to be local. The effect is that we have multiple Go runtimes, but they don't refer to each other so there is no confusion.

I guess you could do the same with your shared libraries, if your C code doesn't mind being linked with -Bsymbolic.

There is a design doc but I'm not sure it helps with this level of detail: https://golang.org/s/execmodes.

mewmew added a commit to mewspring/go-cgo-issue that referenced this issue Mar 19, 2019
add -Wl,-Bsymbolic to link c-archive in the same way as c-shared
Thank you @ianlancetaylor for the suggestion and insight!

This resolved the issue outlined in golang/go#30822.
@mewmew

This comment has been minimized.

Copy link
Contributor Author

commented Mar 19, 2019

Thank you so much @ianlancetaylor! Your insight is invaluable.

Indeed, adding -Wl,-Bsymbolic to the build of each shared library that included Go code in c-archive form resolved the issue.

Thanks once more :)

Cheerful regards,
Robin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.