runtime: unexpected return pc for os/signal.loop called from 0x0

[avinash-getmega]

What version of Go are you using (go version)?

go version go1.13 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build556391308=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I'm building a module which is supposed to work with another program (https://github.com/heroiclabs/nakama). The program runs fine as long as I don't load the module and crashes with below error while loading the module.
The program even starts loading the module (I know it because I can see the logs from module) but crashes instantly.

What did you expect to see?

Not crash

What did you see instead?

{"level":"warn","ts":"2019-10-19T06:03:25.728Z","msg":"WARNING: insecure default parameter value, change this for production!","param":"console.signing_key"}
{"level":"info","ts":"2019-10-19T06:03:25.730Z","msg":"Nakama starting"}
{"level":"info","ts":"2019-10-19T06:03:25.731Z","msg":"Node","name":"nakama-node-1","version":"2.7.0+d5e0b5bb","runtime":"go1.13","cpu":1,"proc":1}
{"level":"info","ts":"2019-10-19T06:03:25.731Z","msg":"Data directory","path":"/nakama/data"}
{"level":"info","ts":"2019-10-19T06:03:25.732Z","msg":"Database connections","dsns":["nakama:nakama@10.89.192.3:5432/nakama"]}
{"level":"debug","ts":"2019-10-19T06:03:25.733Z","msg":"Complete database connection URL","raw_url":"postgresql://nakama:nakama@10.89.192.3:5432/nakama?sslmode=disable"}
{"level":"info","ts":"2019-10-19T06:03:25.744Z","msg":"Database information","version":"PostgreSQL 11.5 on x86_64-pc-linux-gnu, compiled by gcc (Debian 7.3.0-5) 7.3.0, 64-bit"}
{"level":"info","ts":"2019-10-19T06:03:25.750Z","msg":"Initializing leaderboard rank cache"}
{"level":"info","ts":"2019-10-19T06:03:25.751Z","msg":"Leaderboard rank cache initialization completed successfully","cached":[],"skipped":[]}
{"level":"info","ts":"2019-10-19T06:03:25.752Z","msg":"Initialising runtime","path":"/nakama/data/modules"}
{"level":"info","ts":"2019-10-19T06:03:25.752Z","msg":"Initialising runtime event queue processor"}
{"level":"info","ts":"2019-10-19T06:03:25.753Z","msg":"Runtime event queue processor started","size":8192,"workers":8}
{"level":"info","ts":"2019-10-19T06:03:25.754Z","msg":"Initialising Go runtime provider","path":"/nakama/data/modules"}
{"level":"info","ts":"2019-10-19T06:03:25.794Z","msg":"My module"}
{"level":"info","ts":"2019-10-19T06:03:25.795Z","msg":"env variables%!(EXTRA map[string]string=map[env:staging])"}
runtime: unexpected return pc for os/signal.loop called from 0x0
stack: frame={sp:0xc00005e7c0, fp:0xc00005e7e0} stack=[0xc00005e000,0xc00005e800)
000000c00005e6c0:  0000000000000000  0000000000000000
000000c00005e6d0:  0000000000000000  0000000000a04046 <runtime.futexsleep+70>
000000c00005e6e0:  00000000029a7c00  0000000000000080
000000c00005e6f0:  0000000000000000  0000000000000000
000000c00005e700:  0000000000000000  000000c00005e768
000000c00005e710:  000000c000001980  0000000000000000
000000c00005e720:  000000c00005e758  00000000009e3286 <runtime.notetsleep_internal+134>
000000c00005e730:  00000000029a7c00  0000000000000000
000000c00005e740:  0000000000000000  0000000000000000
000000c00005e750:  0000000000000000  0000000000000000
000000c00005e760:  0000000000000000  0000000000000000
000000c00005e770:  0000000000000000  0000000000000000
000000c00005e780:  0000000000000000  0000000000000000
000000c00005e790:  0000000000a1d57c <os/signal.signal_recv+156>  00000000029a7c00
000000c00005e7a0:  ffffffffffffffff  0000000000000000
000000c00005e7b0:  000000c00005e7d0  0000000000d0f0e2 <os/signal.loop+34>
000000c00005e7c0: <0000000000000000  0000000000000000
000000c00005e7d0:  0000000000000000 !0000000000000000
000000c00005e7e0: >0000000000000000  0000000000000000
000000c00005e7f0:  0000000000000000  0000000000000000
fatal error: unknown caller pc

runtime stack:
runtime.throw(0x1ac7719, 0x11)
        runtime/panic.go:774 +0x72
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc000001980, 0x0, 0x0, 0x7fffffff, 0x7ffdfd5a6340, 0x0, 0x0, ...)
        runtime/traceback.go:273 +0x19e5
runtime.scanstack(0xc000001980, 0xc00004b270)
        runtime/mgcmark.go:711 +0x15d
runtime.scang(0xc000001980, 0xc00004b270)
        runtime/proc.go:886 +0x1e3
runtime.markroot.func1()
        runtime/mgcmark.go:221 +0x6d
runtime.markroot(0xc00004b270, 0xb)
        runtime/mgcmark.go:202 +0x2f3
runtime.gcDrain(0xc00004b270, 0xb)
        runtime/mgcmark.go:915 +0x110
runtime.gcBgMarkWorker.func2()
        runtime/mgc.go:1925 +0x185
runtime.systemstack(0xa332a4)
        runtime/asm_amd64.s:370 +0x66
runtime.mstart()
        runtime/proc.go:1146

goroutine 9 [GC worker (idle)]:
runtime.systemstack_switch()
        runtime/asm_amd64.s:330 fp=0xc00005ff60 sp=0xc00005ff58 pc=0xa333a0
runtime.gcBgMarkWorker(0xc00004a000)
        runtime/mgc.go:1891 +0x1bd fp=0xc00005ffd8 sp=0xc00005ff60 pc=0x9f472d
runtime.goexit()
        runtime/asm_amd64.s:1357 +0x1 fp=0xc00005ffe0 sp=0xc00005ffd8 pc=0xa35471
created by runtime.gcBgMarkStartWorkers
        runtime/mgc.go:1785 +0x77

goroutine 1 [runnable]:
bytes.makeSlice(0x902, 0x0, 0x0, 0x0)
        bytes/buffer.go:229 +0x77
bytes.(*Buffer).grow(0xc0007321e0, 0x902, 0x1a0c740)
        bytes/buffer.go:142 +0x15b
bytes.(*Buffer).Write(0xc0007321e0, 0xc000723500, 0x902, 0x903, 0x18dce00, 0x1a0c740, 0x1)
        bytes/buffer.go:172 +0xdd
bytes.(*Reader).WriteTo(0xc0007321b0, 0x1fbc5e0, 0xc0007321e0, 0x7f097c77fba0, 0xc0007321b0, 0x7f097b294601)
        bytes/reader.go:144 +0xa1
io.copyBuffer(0x1fbc5e0, 0xc0007321e0, 0x1fbc620, 0xc0007321b0, 0x0, 0x0, 0x0, 0xc02f80ac1f, 0x2f80ac1f7b65d9bc, 0xc0003ba898)
        io/io.go:384 +0x34f
io.Copy(...)
        io/io.go:364
github.com/gobuffalo/packd.(*virtualFile).write(0xc000725880, 0x1a0c740, 0xc0007321b0, 0x0, 0xc0003ba8f8, 0x7f097b294d9a)
        github.com/gobuffalo/packd@v0.3.0/file.go:76 +0x2e5
github.com/gobuffalo/packd.buildFile(0xc0003867e0, 0x13, 0x1fbc620, 0xc0007321b0, 0xc000620d80, 0xc000732150, 0xc000723500)
        github.com/gobuffalo/packd@v0.3.0/file.go:121 +0xef
github.com/gobuffalo/packd.NewFile(...)
        github.com/gobuffalo/packd@v0.3.0/file.go:97
github.com/gobuffalo/packr.Box.find(0x7f097ba75f64, 0x6, 0xc000398c00, 0x30, 0xc000620d80, 0xc000732150, 0xc0003867e0, 0x13, 0xc0003baa90, 0x7f097b364e66, ...)
        github.com/gobuffalo/packr@v1.30.1/box.go:160 +0x2c6
github.com/gobuffalo/packr.Box.Find(0x7f097ba75f64, 0x6, 0xc000398c00, 0x30, 0xc000620d80, 0x0, 0xc0003867e0, 0x13, 0xc0003bab50, 0x7f097b75fa00, ...)
        github.com/gobuffalo/packr@v1.30.1/box.go:108 +0x76
getmega.com/multiplay/pkg/thirdparty/google.readFile(...)
        getmega.com/multiplay@/pkg/thirdparty/google/default_application_credentials.go:15
getmega.com/multiplay/pkg/thirdparty/google.SetupDefaultCredentials(0xc000619440)
        getmega.com/multiplay@/pkg/thirdparty/google/default_application_credentials.go:22 +0x105
plugin/unnamed-979dcbb6301160304f8e5e9973563ccbab186d50.InitModule(0x1feb900, 0xc000620600, 0x20001c0, 0xc000010b20, 0xc00003e540, 0x200fe20, 0xc0005680e0, 0x20145c0, 0xc00003e6c0, 0xc0006d4380, ...)
        getmega.com/multiplay@/init_module.go:35 +0xbc
github.com/heroiclabs/nakama/v2/server.NewRuntimeProviderGo(0xc0000fd860, 0xc0000fd860, 0xc00003e540, 0x28bfb80, 0x2001fe0, 0xc0005fe1e0, 0xc000562410, 0x1ffc760, 0xc00001fd40, 0x1ff5bc0, ...)
        github.com/heroiclabs/nakama/v2@/server/runtime_go.go:1830 +0xc11
github.com/heroiclabs/nakama/v2/server.NewRuntime(0xc0000fd860, 0xc0000fd860, 0xc00003e540, 0x28bfb80, 0x2981af0, 0x2001fe0, 0xc0005fe1e0, 0xc000562410, 0x1ffc760, 0xc00001fd40, ...)
        github.com/heroiclabs/nakama/v2@/server/runtime.go:442 +0x6a8
main.main()
        main.go:130 +0x1356

goroutine 6 [syscall (scan)]:
os/signal.signal_recv(0x0)
        runtime/sigqueue.go:147 +0x9c
runtime: unexpected return pc for os/signal.loop called from 0x0
stack: frame={sp:0xc00005e7c0, fp:0xc00005e7e0} stack=[0xc00005e000,0xc00005e800)
000000c00005e6c0:  0000000000000000  0000000000000000
000000c00005e6d0:  0000000000000000  0000000000a04046 <runtime.futexsleep+70>
000000c00005e6e0:  00000000029a7c00  0000000000000080
000000c00005e6f0:  0000000000000000  0000000000000000
000000c00005e700:  0000000000000000  000000c00005e768
000000c00005e710:  000000c000001980  0000000000000000
000000c00005e720:  000000c00005e758  00000000009e3286 <runtime.notetsleep_internal+134>
000000c00005e730:  00000000029a7c00  0000000000000000
000000c00005e740:  0000000000000000  0000000000000000
000000c00005e750:  0000000000000000  0000000000000000
000000c00005e760:  0000000000000000  0000000000000000
000000c00005e770:  0000000000000000  0000000000000000
000000c00005e780:  0000000000000000  0000000000000000
000000c00005e790:  0000000000a1d57c <os/signal.signal_recv+156>  00000000029a7c00
000000c00005e7a0:  ffffffffffffffff  0000000000000000
000000c00005e7b0:  000000c00005e7d0  0000000000d0f0e2 <os/signal.loop+34>
000000c00005e7c0: <0000000000000000  0000000000000000
000000c00005e7d0:  0000000000000000 !0000000000000000
000000c00005e7e0: >0000000000000000  0000000000000000
000000c00005e7f0:  0000000000000000  0000000000000000
os/signal.loop()
        os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
        os/signal/signal_unix.go:29 +0x41

goroutine 8 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc00003ca50)
        go.opencensus.io@v0.22.1/stats/view/worker.go:154 +0x100
created by go.opencensus.io/stats/view.init.0
        go.opencensus.io@v0.22.1/stats/view/worker.go:32 +0x57

goroutine 30 [select]:
github.com/heroiclabs/nakama/v2/server.NewRuntimeEventQueue.func1(0xc0006204e0)
        github.com/heroiclabs/nakama/v2@/server/runtime_event.go:43 +0xda
created by github.com/heroiclabs/nakama/v2/server.NewRuntimeEventQueue
        github.com/heroiclabs/nakama/v2@/server/runtime_event.go:41 +0x100
goroutine 31 [select]:
github.com/heroiclabs/nakama/v2/server.NewRuntimeEventQueue.func1(0xc0006204e0)
        github.com/heroiclabs/nakama/v2@/server/runtime_event.go:43 +0xda
created by github.com/heroiclabs/nakama/v2/server.NewRuntimeEventQueue
        github.com/heroiclabs/nakama/v2@/server/runtime_event.go:41 +0x100
goroutine 25 [select]:
database/sql.(*DB).connectionCleaner(0xc00003e540, 0x34630b8a000)
        database/sql/sql.go:950 +0x370
created by database/sql.(*DB).startCleanerLocked
        database/sql/sql.go:937 +0xa7
goroutine 29 [select]:
github.com/heroiclabs/nakama/v2/server.StartLocalTracker.func1(0xc0005dbdd0)
        github.com/heroiclabs/nakama/v2@/server/tracker.go:191 +0xeb
created by github.com/heroiclabs/nakama/v2/server.StartLocalTracker
        github.com/heroiclabs/nakama/v2@/server/tracker.go:188 +0x1cb
goroutine 15 [select]:
github.com/blevesearch/bleve/index.AnalysisWorker(0xc000396c60, 0xc000396cc0)
        github.com/blevesearch/bleve@v0.0.0-20190613153557-26d2099fa968/index/analysis.go:102 +0x102
created by github.com/blevesearch/bleve/index.NewAnalysisQueue
        github.com/blevesearch/bleve@v0.0.0-20190613153557-26d2099fa968/index/analysis.go:94 +0xc8
goroutine 16 [select]:
github.com/blevesearch/bleve/index.AnalysisWorker(0xc000396c60, 0xc000396cc0)
        github.com/blevesearch/bleve@v0.0.0-20190613153557-26d2099fa968/index/analysis.go:102 +0x102
created by github.com/blevesearch/bleve/index.NewAnalysisQueue
        github.com/blevesearch/bleve@v0.0.0-20190613153557-26d2099fa968/index/analysis.go:94 +0xc8
goroutine 18 [select]:
github.com/blevesearch/bleve/index.AnalysisWorker(0xc000396c60, 0xc000396cc0)
        github.com/blevesearch/bleve@v0.0.0-20190613153557-26d2099fa968/index/analysis.go:102 +0x102
created by github.com/blevesearch/bleve/index.NewAnalysisQueue
        github.com/blevesearch/bleve@v0.0.0-20190613153557-26d2099fa968/index/analysis.go:94 +0xc8
goroutine 19 [select]:
github.com/blevesearch/bleve/index.AnalysisWorker(0xc000396c60, 0xc000396cc0)
        github.com/blevesearch/bleve@v0.0.0-20190613153557-26d2099fa968/index/analysis.go:102 +0x102
created by github.com/blevesearch/bleve/index.NewAnalysisQueue
        github.com/blevesearch/bleve@v0.0.0-20190613153557-26d2099fa968/index/analysis.go:94 +0xc8
goroutine 22 [select]:
database/sql.(*DB).connectionOpener(0xc00003e540, 0x1feb840, 0xc000618a00)
        database/sql/sql.go:1052 +0xe8
created by database/sql.OpenDB
        database/sql/sql.go:722 +0x15d

goroutine 23 [select]:
database/sql.(*DB).connectionResetter(0xc00003e540, 0x1feb840, 0xc000618a00)
        database/sql/sql.go:1065 +0xfb
created by database/sql.OpenDB
        database/sql/sql.go:723 +0x193
goroutine 32 [select]:
github.com/heroiclabs/nakama/v2/server.NewRuntimeEventQueue.func1(0xc0006204e0)
        github.com/heroiclabs/nakama/v2@/server/runtime_event.go:43 +0xda
created by github.com/heroiclabs/nakama/v2/server.NewRuntimeEventQueue
        github.com/heroiclabs/nakama/v2@/server/runtime_event.go:41 +0x100
goroutine 33 [select]:
github.com/heroiclabs/nakama/v2/server.NewRuntimeEventQueue.func1(0xc0006204e0)
        github.com/heroiclabs/nakama/v2@/server/runtime_event.go:43 +0xda
created by github.com/heroiclabs/nakama/v2/server.NewRuntimeEventQueue
        github.com/heroiclabs/nakama/v2@/server/runtime_event.go:41 +0x100
goroutine 34 [select]:
github.com/heroiclabs/nakama/v2/server.NewRuntimeEventQueue.func1(0xc0006204e0)
        github.com/heroiclabs/nakama/v2@/server/runtime_event.go:43 +0xda
created by github.com/heroiclabs/nakama/v2/server.NewRuntimeEventQueue
        github.com/heroiclabs/nakama/v2@/server/runtime_event.go:41 +0x100
goroutine 35 [select]:
github.com/heroiclabs/nakama/v2/server.NewRuntimeEventQueue.func1(0xc0006204e0)
        github.com/heroiclabs/nakama/v2@/server/runtime_event.go:43 +0xda
created by github.com/heroiclabs/nakama/v2/server.NewRuntimeEventQueue
        github.com/heroiclabs/nakama/v2@/server/runtime_event.go:41 +0x100
goroutine 36 [select]:
github.com/heroiclabs/nakama/v2/server.NewRuntimeEventQueue.func1(0xc0006204e0)
        github.com/heroiclabs/nakama/v2@/server/runtime_event.go:43 +0xda
created by github.com/heroiclabs/nakama/v2/server.NewRuntimeEventQueue
        github.com/heroiclabs/nakama/v2@/server/runtime_event.go:41 +0x100
goroutine 37 [select]:
github.com/heroiclabs/nakama/v2/server.NewRuntimeEventQueue.func1(0xc0006204e0)
        github.com/heroiclabs/nakama/v2@/server/runtime_event.go:43 +0xda
created by github.com/heroiclabs/nakama/v2/server.NewRuntimeEventQueue
        github.com/heroiclabs/nakama/v2@/server/runtime_event.go:41 +0x100

[dmitshur]

Hello, thanks for the report.

It will be very helpful to try to create a smaller snippet that can reproduce this issue in order to find the root problem. Are you able to narrow it down further?

If not, can you share more complete steps to reproduce the problem, if possible? What commands did you run in order to get the crash?

[novabyte]

@dmitshur There's more information on the Nakama issue tracker under this ticket:
heroiclabs/nakama#388

Here's what we've been able to determine with our investigation so far:

• The problem occurs with Alpine 3.10 when a Go plugin is loaded by the binary built with Go 1.13.0+. We've not tested with earlier versions of Go because we use --trimpath to ensure that the module based builds don't get confused with their dependencies.
• The problem does not occur under Debian-based container base images.
• I've been able to observe the problem on Ubuntu and Windows as the host environments running the latest version of Docker engine.
• I cannot reproduce the issue at all on macOS (10.14.5 Mojave).
• The issue does not occur with an earlier release of Alpine (3.9.4).

The current theory is that there's an issue with how shared objects are loaded by the Go runtime in containers that use the most recent version of musl libc within Alpine Linux. Though only when the host OS is Windows or Ubuntu (only Linux distro I tried). 🤷‍♂

Does this provide more context? I don't know whether the Go team considers musl libc a no-go with the Go runtime. Please advise on what the recommended course of action is for Go projects that are packaged with Alpine Linux or other non-GNU libc core libraries.

[joshgarnett]

@dmitshur I just put together a simple project that reproduces this at https://github.com/joshgarnett/nakama-2.7.0-go-1.13-crash

[dmitshur]

Thanks for doing that.

/cc @aclements @randall77

[randall77]

I can't get this to crash using the given repro. Could you give more explicit steps? A list of things you typed at the command line + results, perhaps?

I don't think we officially support non-GNU libc, at least with linux. I don't think that's intentional, just that we can only really support what we test with, and all our linux platforms are GNU libc (I think).

[novabyte]

@randall77 I ran these steps on macOS with Docker 2.1.0.5:

$> git clone "https://github.com/joshgarnett/nakama-2.7.0-go-1.13-crash"
$> cd "nakama-2.7.0-go-1.13-crash"
$> docker-compose -f docker-compose.yml build nakama
[snip]
Successfully built c3c1ed2a29b6
Successfully tagged nakama-270-go-113-crash_nakama:latest
$> docker-compose -f docker-compose.yml up
[snip]
nakama-crash | {"level":"info","ts":"2019-12-10T12:56:13.808Z","msg":"Starting API server for gRPC requests","port":7349}
nakama-crash | {"level":"info","ts":"2019-12-10T12:56:13.815Z","msg":"Starting API server gateway for HTTP requests","port":7350}
nakama-crash | {"level":"info","ts":"2019-12-10T12:56:14.838Z","msg":"Startup done"}

I'm no longer able to reproduce this issue either. 🤔
I wonder whether it was due to a problem with an earlier version of Docker? @joshgarnett

[joshgarnett]

@novabyte the issue doesn't appear on OSX. Only on Windows.

I just re-ran it on Windows 10 (1803) with Docker for Desktop 2.1.0.5 and saw the same crash. I also did a docker system prune -a and restarted my machine for good measure before running. Below is the output:

PS C:\Users\joshg\git\nakama-2.7.0-go-1.13-crash> ls


    Directory: C:\Users\joshg\git\nakama-2.7.0-go-1.13-crash


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----       10/24/2019   9:22 AM                src
-a----       10/24/2019  10:46 AM            788 docker-compose.alpine310.yml
-a----       10/24/2019  10:40 AM            785 docker-compose.alpine39.yml
-a----       10/24/2019  11:05 AM            779 docker-compose.buster.yml
-a----       10/24/2019   9:23 AM            712 docker-compose.yml
-a----       10/24/2019  10:36 AM            401 Dockerfile
-a----       10/24/2019  11:10 AM           3347 Dockerfile.alpine310
-a----       10/24/2019  10:39 AM           3348 Dockerfile.alpine39
-a----       10/24/2019  11:08 AM           3385 Dockerfile.buster
-a----       10/24/2019   9:13 AM             77 go.mod
-a----       10/24/2019   9:21 AM           1092 LICENSE
-a----       10/24/2019  11:06 AM            899 README.md


PS C:\Users\joshg\git\nakama-2.7.0-go-1.13-crash> docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

PS C:\Users\joshg\git\nakama-2.7.0-go-1.13-crash> docker-compose -f docker-compose.yml build nakama

Building nakama
Step 1/11 : FROM golang:1.13.0-alpine3.10 as builder
1.13.0-alpine3.10: Pulling from library/golang
9d48c3bd43c5: Pull complete
7f94eaf8af20: Pull complete
9fe9984849c1: Pull complete
43c4d47973b8: Pull complete
9aaad45c37b5: Pull complete
Digest: sha256:84349ee862d8bafff35e0d2bfd539da565b536b4dfce654773fc21a1db2da6d7
Status: Downloaded newer image for golang:1.13.0-alpine3.10
 ---> 33fbbc2ca571
Step 2/11 : ENV GOOS linux
 ---> Running in c947ee77a435
Removing intermediate container c947ee77a435
 ---> f40ebdc0be1d
Step 3/11 : ENV GOARCH amd64
 ---> Running in 5c1eaa1b90f4
Removing intermediate container 5c1eaa1b90f4
 ---> c353270a2f7a
Step 4/11 : ENV CGO_ENABLED 1
 ---> Running in a05a3ace4ccc
Removing intermediate container a05a3ace4ccc
 ---> 54360f69c947
Step 5/11 : RUN apk --no-cache add ca-certificates gcc musl-dev git
 ---> Running in fdf564476702
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/16) Installing binutils (2.32-r0)
(2/16) Installing gmp (6.1.2-r1)
(3/16) Installing isl (0.18-r0)
(4/16) Installing libgomp (8.3.0-r0)
(5/16) Installing libatomic (8.3.0-r0)
(6/16) Installing libgcc (8.3.0-r0)
(7/16) Installing mpfr3 (3.1.5-r1)
(8/16) Installing mpc1 (1.1.0-r0)
(9/16) Installing libstdc++ (8.3.0-r0)
(10/16) Installing gcc (8.3.0-r0)
(11/16) Installing nghttp2-libs (1.39.2-r0)
(12/16) Installing libcurl (7.66.0-r0)
(13/16) Installing expat (2.2.8-r0)
(14/16) Installing pcre2 (10.33-r0)
(15/16) Installing git (2.22.2-r0)
(16/16) Installing musl-dev (1.1.22-r3)
Executing busybox-1.30.1-r2.trigger
OK: 118 MiB in 31 packages
Removing intermediate container fdf564476702
 ---> efb1907f7d38
Step 6/11 : WORKDIR /builder
 ---> Running in 11d60deeffc3
Removing intermediate container 11d60deeffc3
 ---> 92cad302eec2
Step 7/11 : COPY src src
 ---> 4e20baffe8e7
Step 8/11 : COPY go.mod go.mod
 ---> fe5568a2f56d
Step 9/11 : RUN go build --buildmode=plugin -trimpath -o ./src/modules/main.so ./src/main.go
 ---> Running in 9afed9a74169
go: downloading github.com/heroiclabs/nakama-common v1.0.0
go: extracting github.com/heroiclabs/nakama-common v1.0.0
go: downloading github.com/golang/protobuf v1.3.2
go: extracting github.com/golang/protobuf v1.3.2
go: finding github.com/heroiclabs/nakama-common v1.0.0
9d48c3bd43c5: Already exists
Removing intermediate container 9afed9a74169
 ---> 52d1a491e582
Step 10/11 : FROM heroiclabs/nakama:2.7.0
2.7.0: Pulling from heroiclabs/nakama
e4cb6aa05ce0: Pull complete
7a3facfc428a: Pull complete
Digest: sha256:9ddbf650eab4d1ee01efbd695340ec6ad2c683dcab7c619dece3f18e0d207cce
Status: Downloaded newer image for heroiclabs/nakama:2.7.0
 ---> ef23b2996f07
Step 11/11 : COPY --from=builder /builder/src/modules/main.so /nakama/data/modules
 ---> 372d9ca45d21
Successfully built 372d9ca45d21
Successfully tagged nakama-270-go-113-crash_nakama:latest

PS C:\Users\joshg\git\nakama-2.7.0-go-1.13-crash> docker-compose -f docker-compose.yml up

Creating network "nakama-270-go-113-crash_default" with the default driver
Pulling postgres (postgres:11.3)...
11.3: Pulling from library/postgres
fc7181108d40: Pull complete
81cfa12d39e9: Pull complete
793d305ca761: Pull complete
41e3ced3a2aa: Pull complete
a300bc9d5405: Pull complete
3c6a5c3830ed: Pull complete
fb8c79b24338: Pull complete
fcda1144379f: Pull complete
476a22a819cc: Pull complete
78b36b49bb24: Pull complete
6a096a28591f: Pull complete
c0cb89b5217b: Pull complete
778f1469a309: Pull complete
7c4413fcad87: Pull complete
Digest: sha256:1518027f4aaee49b836c5cf4ece1b4a16bdcd820af873402e19e1cc181c1aff2
Status: Downloaded newer image for postgres:11.3
Creating nakama-crash-postgres ... done
Creating nakama-crash          ... done
Attaching to nakama-crash-postgres, nakama-crash
nakama-crash-postgres | The files belonging to this database system will be owned by user "postgres".
nakama-crash-postgres | This user must also own the server process.
nakama-crash-postgres |
nakama-crash-postgres | The database cluster will be initialized with locale "en_US.utf8".
nakama-crash-postgres | The default database encoding has accordingly been set to "UTF8".
nakama-crash-postgres | The default text search configuration will be set to "english".
nakama-crash-postgres |
nakama-crash-postgres | Data page checksums are disabled.
nakama-crash-postgres |
nakama-crash-postgres | fixing permissions on existing directory /var/lib/postgresql/data ... ok
nakama-crash-postgres | creating subdirectories ... ok
nakama-crash-postgres | selecting default max_connections ... 100
nakama-crash-postgres | selecting default shared_buffers ... 128MB
nakama-crash-postgres | selecting dynamic shared memory implementation ... posix
nakama-crash-postgres | creating configuration files ... ok
nakama-crash | + /nakama/nakama migrate up --database.address postgres:password@postgres:5432/nakama
nakama-crash-postgres | running bootstrap script ... ok
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:01.832Z","msg":"Database connection","dsn":"postgres:password@postgres:5432/nakama"}
nakama-crash | {"level":"fatal","ts":"2019-12-10T23:04:01.834Z","msg":"Error pinging database","error":"dial tcp 172.18.0.2:5432: connect: connection refused","stacktrace":"github.com/heroiclabs/nakama/v2/migrate.Parse\n\tgithub.com/heroiclabs/nakama/v2@/migrate/migrate.go:147\nmain.main\n\tmain.go:79\nruntime.main\n\truntime/proc.go:203"}
nakama-crash-postgres | performing post-bootstrap initialization ... ok
nakama-crash-postgres | syncing data to disk ...
nakama-crash-postgres | WARNING: enabling "trust" authentication for local connections
nakama-crash-postgres | You can change this by editing pg_hba.conf or using the option -A, or
nakama-crash-postgres | --auth-local and --auth-host, the next time you run initdb.
nakama-crash-postgres | ok
nakama-crash-postgres |
nakama-crash-postgres | Success. You can now start the database server using:
nakama-crash-postgres |
nakama-crash-postgres |     pg_ctl -D /var/lib/postgresql/data -l logfile start
nakama-crash-postgres |
nakama-crash-postgres | waiting for server to start....2019-12-10 23:04:02.029 UTC [41] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
nakama-crash-postgres | 2019-12-10 23:04:02.069 UTC [42] LOG:  database system was shut down at 2019-12-10 23:04:01 UTC
nakama-crash-postgres | 2019-12-10 23:04:02.103 UTC [41] LOG:  database system is ready to accept connections
nakama-crash-postgres |  done
nakama-crash-postgres | server started
nakama-crash-postgres |
nakama-crash-postgres | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
nakama-crash-postgres |
nakama-crash-postgres | waiting for server to shut down....2019-12-10 23:04:02.155 UTC [41] LOG:  received fast shutdown request
nakama-crash-postgres | 2019-12-10 23:04:02.158 UTC [41] LOG:  aborting any active transactions
nakama-crash-postgres | 2019-12-10 23:04:02.161 UTC [41] LOG:  background worker "logical replication launcher" (PID 48) exited with exit code 1
nakama-crash-postgres | 2019-12-10 23:04:02.161 UTC [43] LOG:  shutting down
nakama-crash-postgres | 2019-12-10 23:04:02.183 UTC [41] LOG:  database system is shut down
nakama-crash-postgres |  done
nakama-crash-postgres | server stopped
nakama-crash-postgres |
nakama-crash-postgres | PostgreSQL init process complete; ready for start up.
nakama-crash-postgres |
nakama-crash-postgres | 2019-12-10 23:04:02.238 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
nakama-crash-postgres | 2019-12-10 23:04:02.238 UTC [1] LOG:  listening on IPv6 address "::", port 5432
nakama-crash-postgres | 2019-12-10 23:04:02.243 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
nakama-crash-postgres | 2019-12-10 23:04:02.257 UTC [50] LOG:  database system was shut down at 2019-12-10 23:04:02 UTC
nakama-crash-postgres | 2019-12-10 23:04:02.261 UTC [1] LOG:  database system is ready to accept connections
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.388Z","msg":"Database information","version":"PostgreSQL 11.3 (Debian 11.3-1.pgdg90+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516, 64-bit"}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.698Z","msg":"Creating new database","name":"nakama"}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.910Z","msg":"Successfully applied migration","count":2}
nakama-crash | + exec /nakama/nakama --database.address postgres:password@postgres:5432/nakama
nakama-crash | {"level":"warn","ts":"2019-12-10T23:04:04.945Z","msg":"WARNING: insecure default parameter value, change this for production!","param":"console.username"}
nakama-crash | {"level":"warn","ts":"2019-12-10T23:04:04.945Z","msg":"WARNING: insecure default parameter value, change this for production!","param":"console.password"}
nakama-crash | {"level":"warn","ts":"2019-12-10T23:04:04.945Z","msg":"WARNING: insecure default parameter value, change this for production!","param":"console.signing_key"}
nakama-crash | {"level":"warn","ts":"2019-12-10T23:04:04.945Z","msg":"WARNING: insecure default parameter value, change this for production!","param":"socket.server_key"}
nakama-crash | {"level":"warn","ts":"2019-12-10T23:04:04.945Z","msg":"WARNING: insecure default parameter value, change this for production!","param":"session.encryption_key"}
nakama-crash | {"level":"warn","ts":"2019-12-10T23:04:04.945Z","msg":"WARNING: insecure default parameter value, change this for production!","param":"runtime.http_key"}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.945Z","msg":"Nakama starting"}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.945Z","msg":"Node","name":"nakama","version":"2.7.0+d5e0b5bb","runtime":"go1.13","cpu":2,"proc":2}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.945Z","msg":"Data directory","path":"/nakama/data"}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.945Z","msg":"Database connections","dsns":["postgres:password@postgres:5432/nakama"]}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.952Z","msg":"Database information","version":"PostgreSQL 11.3 (Debian 11.3-1.pgdg90+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516, 64-bit"}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.956Z","msg":"Initializing leaderboard rank cache"}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.956Z","msg":"Leaderboard rank cache initialization completed successfully","cached":[],"skipped":[]}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.956Z","msg":"Initialising runtime","path":"/nakama/data/modules"}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.956Z","msg":"Initialising runtime event queue processor"}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.956Z","msg":"Runtime event queue processor started","size":8192,"workers":8}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.956Z","msg":"Initialising Go runtime provider","path":"/nakama/data/modules"}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.965Z","msg":"Go runtime modules loaded"}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.965Z","msg":"Initialising Lua runtime provider","path":"/nakama/data/modules"}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.965Z","msg":"Lua runtime modules loaded"}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.965Z","msg":"Allocating minimum runtime pool","count":16}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.965Z","msg":"Allocated minimum runtime pool"}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.965Z","msg":"Found runtime modules","count":1,"modules":["main.so"]}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.965Z","msg":"Leaderboard scheduler start"}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.965Z","msg":"Leaderboard scheduler update","end_active":"-1ns","end_active_count":0,"expiry":"-1ns","expiry_count":0}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.966Z","msg":"Starting Console server for gRPC requests","port":7348}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.966Z","msg":"Starting Console server gateway for HTTP requests","port":7351}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.966Z","msg":"Starting API server for gRPC requests","port":7349}
nakama-crash | {"level":"info","ts":"2019-12-10T23:04:04.966Z","msg":"Starting API server gateway for HTTP requests","port":7350}
nakama-crash | runtime: unexpected return pc for os/signal.loop called from 0x0
nakama-crash | stack: frame={sp:0xc0000567c0, fp:0xc0000567e0} stack=[0xc000056000,0xc000056800)
nakama-crash | 000000c0000566c0:  0000000000000000  0000000000000000
nakama-crash | 000000c0000566d0:  0000000000000000  0000000000a04046 <runtime.futexsleep+70>
nakama-crash | 000000c0000566e0:  00000000029a7c00  0000000000000080
nakama-crash | 000000c0000566f0:  0000000000000000  0000000000000000
nakama-crash | 000000c000056700:  0000000000000000  000000c000056768
nakama-crash | 000000c000056710:  000000c000094480  0000000000000000
nakama-crash | 000000c000056720:  000000c000056758  00000000009e3286 <runtime.notetsleep_internal+134>
nakama-crash | 000000c000056730:  00000000029a7c00  0000000000000000
nakama-crash | 000000c000056740:  0000000000000000  0000000000000000
nakama-crash | 000000c000056750:  0000000000000000  0000000000000000
nakama-crash | 000000c000056760:  0000000000000000  0000000000000000
nakama-crash | 000000c000056770:  0000000000000000  0000000000000000
nakama-crash | 000000c000056780:  0000000000000000  0000000000000000
nakama-crash | 000000c000056790:  0000000000a1d57c <os/signal.signal_recv+156>  00000000029a7c00
nakama-crash | 000000c0000567a0:  ffffffffffffffff  0000000000000000
nakama-crash | 000000c0000567b0:  000000c0000567d0  0000000000d0f0e2 <os/signal.loop+34>
nakama-crash | 000000c0000567c0: <0000000000000000  0000000000000000
nakama-crash | 000000c0000567d0:  0000000000000000 !0000000000000000
nakama-crash | 000000c0000567e0: >0000000000000000  0000000000000000
nakama-crash | 000000c0000567f0:  0000000000000000  0000000000000000
nakama-crash | fatal error: unknown caller pc
nakama-crash |
nakama-crash | runtime stack:
nakama-crash | runtime.throw(0x1ac7719, 0x11)
nakama-crash |  runtime/panic.go:774 +0x72
nakama-crash | runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc000094480, 0x0, 0x0, 0x7fffffff, 0x7fffb5f3d390, 0x0, 0x0, ...)
nakama-crash |  runtime/traceback.go:273 +0x19e5
nakama-crash | runtime.scanstack(0xc000094480, 0xc000049270)
nakama-crash |  runtime/mgcmark.go:711 +0x15d
nakama-crash | runtime.scang(0xc000094480, 0xc000049270)
nakama-crash |  runtime/proc.go:886 +0x1e3
nakama-crash | runtime.markroot.func1()
nakama-crash |  runtime/mgcmark.go:221 +0x6d
nakama-crash | runtime.markroot(0xc000049270, 0x7fff0000000c)
nakama-crash |  runtime/mgcmark.go:202 +0x2f3
nakama-crash | runtime.gcDrain(0xc000049270, 0x7)
nakama-crash |  runtime/mgcmark.go:915 +0x110
nakama-crash | runtime.gcBgMarkWorker.func2()
nakama-crash |  runtime/mgc.go:1927 +0x16b
nakama-crash | runtime.systemstack(0xa332a4)
nakama-crash |  runtime/asm_amd64.s:370 +0x66
nakama-crash | runtime.mstart()
nakama-crash |  runtime/proc.go:1146
nakama-crash |
nakama-crash | goroutine 22 [GC worker (idle)]:
nakama-crash | runtime.systemstack_switch()
nakama-crash |  runtime/asm_amd64.s:330 fp=0xc000057f60 sp=0xc000057f58 pc=0xa333a0
nakama-crash | runtime.gcBgMarkWorker(0xc000048000)
nakama-crash |  runtime/mgc.go:1891 +0x1bd fp=0xc000057fd8 sp=0xc000057f60 pc=0x9f472d
nakama-crash | runtime.goexit()
nakama-crash |  runtime/asm_amd64.s:1357 +0x1 fp=0xc000057fe0 sp=0xc000057fd8 pc=0xa35471
nakama-crash | created by runtime.gcBgMarkStartWorkers
nakama-crash |  runtime/mgc.go:1785 +0x77
nakama-crash |
nakama-crash | goroutine 1 [select]:
nakama-crash | net/http.(*Transport).getConn(0x28c8460, 0xc000404d80, 0x0, 0x1aeaafa, 0x5, 0xc0000a9c60, 0x1c, 0x0, 0x0, 0x0, ...)
nakama-crash |  net/http/transport.go:1226 +0x54c
nakama-crash | net/http.(*Transport).roundTrip(0x28c8460, 0xc00043a000, 0xc000404cc0, 0xc000644a6c, 0xc000644a60)
nakama-crash |  net/http/transport.go:522 +0x603
nakama-crash | net/http.(*Transport).RoundTrip(0x28c8460, 0xc00043a000, 0x28c8460, 0xbf7426f99bd66828, 0x5c74e1fd)
nakama-crash |  net/http/roundtrip.go:17 +0x35
nakama-crash | net/http.send(0xc000133f00, 0x1fbf180, 0x28c8460, 0xbf7426f99bd66828, 0x5c74e1fd, 0x2986660, 0xc0002583b0, 0xbf7426f99bd66828, 0x1, 0x0)
nakama-crash |  net/http/client.go:250 +0x43a
nakama-crash | net/http.(*Client).send(0xc000404bd0, 0xc000133f00, 0xbf7426f99bd66828, 0x5c74e1fd, 0x2986660, 0xc0002583b0, 0x0, 0x1, 0x1928400)
nakama-crash |  net/http/client.go:174 +0xfa
nakama-crash | net/http.(*Client).do(0xc000404bd0, 0xc000133f00, 0x0, 0x0, 0x0)
nakama-crash |  net/http/client.go:641 +0x3ce
nakama-crash | net/http.(*Client).Do(...)
nakama-crash |  net/http/client.go:509
nakama-crash | net/http.(*Client).Post(0xc000404bd0, 0x1aeaafa, 0x28, 0x1ade06b, 0x21, 0x1fbc5c0, 0xc000404c00, 0x1, 0x0, 0x0)
nakama-crash |  net/http/client.go:769 +0x195
nakama-crash | github.com/heroiclabs/nakama/v2/ga.SendValues(0xc000404bd0, 0x1ac2a36, 0xd, 0xc00064c240, 0x24, 0xc0005433d8, 0x0, 0x0)
nakama-crash |  github.com/heroiclabs/nakama/v2@/ga/ga.go:72 +0x2fd
nakama-crash | github.com/heroiclabs/nakama/v2/ga.SendSessionStart(0xc000404bd0, 0x1ac2a36, 0xd, 0xc00064c240, 0x24, 0xc000404bd0, 0x30)
nakama-crash |  github.com/heroiclabs/nakama/v2@/ga/ga.go:90 +0x1c1
nakama-crash | main.runTelemetry(0xc000404bd0, 0x1ac2a36, 0xd, 0xc00064c240, 0x24)
nakama-crash |  main.go:271 +0x74
nakama-crash | main.main()
nakama-crash |  main.go:153 +0x21fc
nakama-crash |
nakama-crash | goroutine 19 [syscall (scan)]:
nakama-crash | os/signal.signal_recv(0x0)
nakama-crash |  runtime/sigqueue.go:147 +0x9c
nakama-crash | runtime: unexpected return pc for os/signal.loop called from 0x0
nakama-crash | stack: frame={sp:0xc0000567c0, fp:0xc0000567e0} stack=[0xc000056000,0xc000056800)
nakama-crash | 000000c0000566c0:  0000000000000000  0000000000000000
nakama-crash | 000000c0000566d0:  0000000000000000  0000000000a04046 <runtime.futexsleep+70>
nakama-crash | 000000c0000566e0:  00000000029a7c00  0000000000000080
nakama-crash | 000000c0000566f0:  0000000000000000  0000000000000000
nakama-crash | 000000c000056700:  0000000000000000  000000c000056768
nakama-crash | 000000c000056710:  000000c000094480  0000000000000000
nakama-crash | 000000c000056720:  000000c000056758  00000000009e3286 <runtime.notetsleep_internal+134>
nakama-crash | 000000c000056730:  00000000029a7c00  0000000000000000
nakama-crash | 000000c000056740:  0000000000000000  0000000000000000
nakama-crash | 000000c000056750:  0000000000000000  0000000000000000
nakama-crash | 000000c000056760:  0000000000000000  0000000000000000
nakama-crash | 000000c000056770:  0000000000000000  0000000000000000
nakama-crash | 000000c000056780:  0000000000000000  0000000000000000
nakama-crash | 000000c000056790:  0000000000a1d57c <os/signal.signal_recv+156>  00000000029a7c00
nakama-crash | 000000c0000567a0:  ffffffffffffffff  0000000000000000
nakama-crash | 000000c0000567b0:  000000c0000567d0  0000000000d0f0e2 <os/signal.loop+34>
nakama-crash | 000000c0000567c0: <0000000000000000  0000000000000000
nakama-crash | 000000c0000567d0:  0000000000000000 !0000000000000000
nakama-crash | 000000c0000567e0: >0000000000000000  0000000000000000
nakama-crash | 000000c0000567f0:  0000000000000000  0000000000000000
nakama-crash | os/signal.loop()
nakama-crash |  os/signal/signal_unix.go:23 +0x22
nakama-crash | created by os/signal.init.0
nakama-crash |  os/signal/signal_unix.go:29 +0x41
nakama-crash |
nakama-crash | goroutine 21 [select]:
nakama-crash | go.opencensus.io/stats/view.(*worker).start(0xc0000a2a50)
nakama-crash |  go.opencensus.io@v0.22.1/stats/view/worker.go:154 +0x100
nakama-crash | created by go.opencensus.io/stats/view.init.0
nakama-crash |  go.opencensus.io@v0.22.1/stats/view/worker.go:32 +0x57
nakama-crash |
nakama-crash | goroutine 40 [select]:
nakama-crash | github.com/heroiclabs/nakama/v2/server.NewRuntimeEventQueue.func1(0xc0003a99e0)
nakama-crash |  github.com/heroiclabs/nakama/v2@/server/runtime_event.go:43 +0xda
nakama-crash | created by github.com/heroiclabs/nakama/v2/server.NewRuntimeEventQueue
nakama-crash |  github.com/heroiclabs/nakama/v2@/server/runtime_event.go:41 +0x100
nakama-crash |
nakama-crash | goroutine 10 [select]:
nakama-crash | database/sql.(*DB).connectionCleaner(0xc00028a240, 0x34630b8a000)
nakama-crash |  database/sql/sql.go:950 +0x370
nakama-crash | created by database/sql.(*DB).startCleanerLocked
nakama-crash |  database/sql/sql.go:937 +0xa7
nakama-crash |
nakama-crash | goroutine 26 [select]:
nakama-crash | github.com/blevesearch/bleve/index.AnalysisWorker(0xc00064a960, 0xc00064a9c0)
nakama-crash |  github.com/blevesearch/bleve@v0.0.0-20190613153557-26d2099fa968/index/analysis.go:102 +0x102
nakama-crash | created by github.com/blevesearch/bleve/index.NewAnalysisQueue
nakama-crash |  github.com/blevesearch/bleve@v0.0.0-20190613153557-26d2099fa968/index/analysis.go:94 +0xc8
nakama-crash |
nakama-crash | goroutine 27 [select]:
nakama-crash | github.com/blevesearch/bleve/index.AnalysisWorker(0xc00064a960, 0xc00064a9c0)
nakama-crash |  github.com/blevesearch/bleve@v0.0.0-20190613153557-26d2099fa968/index/analysis.go:102 +0x102
nakama-crash | created by github.com/blevesearch/bleve/index.NewAnalysisQueue
nakama-crash |  github.com/blevesearch/bleve@v0.0.0-20190613153557-26d2099fa968/index/analysis.go:94 +0xc8
nakama-crash |
nakama-crash | goroutine 28 [select]:
nakama-crash | github.com/blevesearch/bleve/index.AnalysisWorker(0xc00064a960, 0xc00064a9c0)
nakama-crash |  github.com/blevesearch/bleve@v0.0.0-20190613153557-26d2099fa968/index/analysis.go:102 +0x102
nakama-crash | created by github.com/blevesearch/bleve/index.NewAnalysisQueue
nakama-crash |  github.com/blevesearch/bleve@v0.0.0-20190613153557-26d2099fa968/index/analysis.go:94 +0xc8
nakama-crash |
nakama-crash | goroutine 29 [select]:
nakama-crash | github.com/blevesearch/bleve/index.AnalysisWorker(0xc00064a960, 0xc00064a9c0)
nakama-crash |  github.com/blevesearch/bleve@v0.0.0-20190613153557-26d2099fa968/index/analysis.go:102 +0x102
nakama-crash | created by github.com/blevesearch/bleve/index.NewAnalysisQueue
nakama-crash |  github.com/blevesearch/bleve@v0.0.0-20190613153557-26d2099fa968/index/analysis.go:94 +0xc8
nakama-crash |
nakama-crash | goroutine 32 [select]:
nakama-crash | database/sql.(*DB).connectionOpener(0xc00028a240, 0x1feb840, 0xc0003a6580)
nakama-crash |  database/sql/sql.go:1052 +0xe8
nakama-crash | created by database/sql.OpenDB
nakama-crash |  database/sql/sql.go:722 +0x15d
nakama-crash |
nakama-crash | goroutine 33 [select]:
nakama-crash | database/sql.(*DB).connectionResetter(0xc00028a240, 0x1feb840, 0xc0003a6580)
nakama-crash |  database/sql/sql.go:1065 +0xfb
nakama-crash | created by database/sql.OpenDB
nakama-crash |  database/sql/sql.go:723 +0x193
nakama-crash |
nakama-crash | goroutine 39 [select]:
nakama-crash | github.com/heroiclabs/nakama/v2/server.StartLocalTracker.func1(0xc000371b00)
nakama-crash |  github.com/heroiclabs/nakama/v2@/server/tracker.go:191 +0xeb
nakama-crash | created by github.com/heroiclabs/nakama/v2/server.StartLocalTracker
nakama-crash |  github.com/heroiclabs/nakama/v2@/server/tracker.go:188 +0x1cb
nakama-crash |
nakama-crash | goroutine 41 [select]:
nakama-crash | fatal error: unexpected signal during runtime execution
nakama-crash | panic during panic
nakama-crash | [signal SIGSEGV: segmentation violation code=0x1 addr=0x118 pc=0xa29396]
nakama-crash |
nakama-crash | runtime stack:
nakama-crash | runtime.throw(0x1aee167, 0x2a)
nakama-crash |  runtime/panic.go:774 +0x72
nakama-crash | runtime.sigpanic()
nakama-crash |  runtime/signal_unix.go:378 +0x47c
nakama-crash | runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc000001e00, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0, ...)
nakama-crash |  runtime/traceback.go:261 +0x1306
nakama-crash | runtime.traceback1(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc000001e00, 0x0)
nakama-crash |  runtime/traceback.go:722 +0xf0
nakama-crash | runtime.traceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc000001e00)
nakama-crash |  runtime/traceback.go:676 +0x52
nakama-crash | runtime.tracebackothers(0x298a0e0)
nakama-crash |  runtime/traceback.go:929 +0x1a6
nakama-crash | runtime.dopanic_m(0x298a0e0, 0xa06b72, 0x7fffb5f3cff8, 0x1)
nakama-crash |  runtime/panic.go:974 +0x29e
nakama-crash | runtime.fatalthrow.func1()
nakama-crash |  runtime/panic.go:829 +0x5f
nakama-crash | runtime.fatalthrow()
nakama-crash |  runtime/panic.go:826 +0x57
nakama-crash | runtime.throw(0x1ac7719, 0x11)
nakama-crash |  runtime/panic.go:774 +0x72
nakama-crash | runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc000094480, 0x0, 0x0, 0x7fffffff, 0x7fffb5f3d390, 0x0, 0x0, ...)
nakama-crash |  runtime/traceback.go:273 +0x19e5
nakama-crash | runtime.scanstack(0xc000094480, 0xc000049270)
nakama-crash |  runtime/mgcmark.go:711 +0x15d
nakama-crash | runtime.scang(0xc000094480, 0xc000049270)
nakama-crash |  runtime/proc.go:886 +0x1e3
nakama-crash | runtime.markroot.func1()
nakama-crash |  runtime/mgcmark.go:221 +0x6d
nakama-crash | runtime.markroot(0xc000049270, 0x7fff0000000c)
nakama-crash |  runtime/mgcmark.go:202 +0x2f3
nakama-crash | runtime.gcDrain(0xc000049270, 0x7)
nakama-crash |  runtime/mgcmark.go:915 +0x110
nakama-crash | runtime.gcBgMarkWorker.func2()
nakama-crash |  runtime/mgc.go:1927 +0x16b
nakama-crash | runtime.systemstack(0xa332a4)
nakama-crash |  runtime/asm_amd64.s:370 +0x66
nakama-crash | runtime.mstart()
nakama-crash |  runtime/proc.go:1146
nakama-crash exited with code 2

[novabyte]

Ah yes I'd forgotten 🤦‍♂ Thanks @joshgarnett. @randall77 Does that cover what you need to reproduce?

[randall77]

Well, windows only makes it a problem. I only have access to windows on gomotes, and they don't have docker installed. I'm not sure how to get around that (rdp + install chrome + install chocolatey + install docker?).

I thought the whole point of Docker was that the host OS doesn't matter?

[DisposaBoy]

Not sure if it supports Docker, but if it helps: Microsoft distributes free Windows VMs for testing in (Internet Explorer) https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/.

[joshgarnett]

@randall77 That's what makes this bug super gnarly.