Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: fatal error: mcall called on m->g0 stack on Windows #67108

Open
RichieSams opened this issue Apr 29, 2024 · 66 comments
Open

runtime: fatal error: mcall called on m->g0 stack on Windows #67108

RichieSams opened this issue Apr 29, 2024 · 66 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows
Milestone

Comments

@RichieSams
Copy link

RichieSams commented Apr 29, 2024

Go version

go1.21.4

Output of go env in your module/workspace:

NOTE: We compile inside docker. The output below is from running go env within the docker container that we use. We cross-compile for linux, windows, and darwin. The workers having a problem in this case are running windows/amd64.

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.4'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2751320750=/tmp/go-build -gno-record-gcc-switches'

What did you do?

We have a cli program that runs on many thousands of VMs / bare metal workers as part of our larger CI setup. Recently, we've been seeing a non-trivial number of panics on a specific set of workers. These workers are bare metal AMD Threadripper machines running Windows 10. Below is some info from DXDiag. (I can get additional information if needed).

      Time of this report: 4/23/2024, 09:36:37
             Machine name: XXXXXXXX
               Machine Id: {F429D65A-427C-4F90-A587-8AB68B33FB57}
         Operating System: Windows 10 Enterprise 64-bit (10.0, Build 19045) (19041.vb_release.191206-1406)
                 Language: English (Regional Setting: English)
      System Manufacturer: Equus Computer Systems
             System Model: Nobilis
                     BIOS: F5c (type: UEFI)
                Processor: AMD Ryzen Threadripper 3990X 64-Core Processor  (64 CPUs), ~2.9GHz
                   Memory: 262144MB RAM
      Available OS Memory: 262032MB RAM
                Page File: 38200MB used, 668832MB available
              Windows Dir: C:\Windows
          DirectX Version: DirectX 12
      DX Setup Parameters: Not found
         User DPI Setting: 96 DPI (100 percent)
       System DPI Setting: 96 DPI (100 percent)
          DWM DPI Scaling: Disabled
                 Miracast: Available, no HDCP
Microsoft Graphics Hybrid: Not Supported
 DirectX Database Version: 1.0.8
           DxDiag Version: 10.00.19041.3636 64bit Unicode

We're seeing two types of panics:

  1. fatal error: runtime: mcall called on m->g0 stack
  2. init order seemingly not following the spec: https://go.dev/ref/spec#Package_initialization
    • init() from a package is being called before the init() of a package it depends on
    • Meaning a global variable is not yet populated
    • Leading to a nil pointer dereference panic

I have included multiple call stacks of these panics below. NOTE: these callstacks all happen on different machines. But they only happen on the same "series" of machine, which is documented above.

All the panics seem to happen during the init() phase of the runtime start-up. The panics are not super reproduce-able. I'm only really seeing it due to our scale.

What did you see happen?

Below are the callstacks from a number of panics across multiple machines.

fatal error: runtime: mcall called on m->g0 stack
runtime stack:
runtime.throw({0x1801948?, 0x2f?})
	runtime/panic.go:1077 +0x65 fp=0xc00025bc20 sp=0xc00025bbf0 pc=0x60c7a5
runtime.badmcall(0xd0?)
	runtime/proc.go:503 +0x1f fp=0xc00025bc40 sp=0xc00025bc20 pc=0x60fa9f
runtime.badmcall(0x60db19)
	<autogenerated>:1 +0x25 fp=0xc00025bc58 sp=0xc00025bc40 pc=0x641ea5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc00025bc70
stack: frame={sp:0xc00025bc40, fp:0xc00025bc58} stack=[0x6fc804000,0x6fc9ff7b0)
goroutine 1 [running, locked to thread]:
google.golang.org/genproto/googleapis/type/expr.init()
	google.golang.org/genproto@v0.0.0-20240227224415-6ceb2ff114de/googleapis/type/expr/expr.pb.go:149 +0x69 fp=0xc00025be10 sp=0xc00025be08 pc=0xc11629
runtime.doInit1(0x243c990)
	runtime/proc.go:6740 +0xdd fp=0xc00025bf40 sp=0xc00025be10 pc=0x61bf1d
runtime.doInit(...)
	runtime/proc.go:6707
runtime.main()
	runtime/proc.go:249 +0x356 fp=0xc00025bfe0 sp=0xc00025bf40 pc=0x60f096
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc00025bfe8 sp=0xc00025bfe0 pc=0x63d241
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f5fa8 sp=0xc0000f5f88 pc=0x60f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.forcegchelper()
	runtime/proc.go:322 +0xb8 fp=0xc0000f5fe0 sp=0xc0000f5fa8 pc=0x60f298
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f5fe8 sp=0xc0000f5fe0 pc=0x63d241
created by runtime.init.6 in goroutine 1
	runtime/proc.go:310 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f7f78 sp=0xc0000f7f58 pc=0x60f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.bgsweep(0x0?)
	runtime/mgcsweep.go:280 +0x94 fp=0xc0000f7fc8 sp=0xc0000f7f78 pc=0x5f8ab4
runtime.gcenable.func1()
	runtime/mgc.go:200 +0x25 fp=0xc0000f7fe0 sp=0xc0000f7fc8 pc=0x5edc25
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0x63d241
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:200 +0x66
goroutine 18 [GC scavenge wait]:
runtime.gopark(0xc0000440e0?, 0x1a65f48?, 0x1?, 0x0?, 0xc0002041a0?)
	runtime/proc.go:398 +0xce fp=0xc00020df70 sp=0xc00020df50 pc=0x60f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.(*scavengerState).park(0x2768440)
	runtime/mgcscavenge.go:425 +0x49 fp=0xc00020dfa0 sp=0xc00020df70 pc=0x5f6369
runtime.bgscavenge(0x0?)
	runtime/mgcscavenge.go:653 +0x3c fp=0xc00020dfc8 sp=0xc00020dfa0 pc=0x5f68fc
runtime.gcenable.func2()
	runtime/mgc.go:201 +0x25 fp=0xc00020dfe0 sp=0xc00020dfc8 pc=0x5edbc5
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc00020dfe8 sp=0xc00020dfe0 pc=0x63d241
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:201 +0xa5
goroutine 19 [finalizer wait]:
runtime.gopark(0x400000?, 0x1000f9e70?, 0x0?, 0x0?, 0x27cba60?)
	runtime/proc.go:398 +0xce fp=0xc0000f9e28 sp=0xc0000f9e08 pc=0x60f40e
runtime.runfinq()
	runtime/mfinal.go:193 +0x107 fp=0xc0000f9fe0 sp=0xc0000f9e28 pc=0x5ecce7
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f9fe8 sp=0xc0000f9fe0 pc=0x63d241
created by runtime.createfing in goroutine 1
	runtime/mfinal.go:163 +0x3d
fatal error: runtime: mcall called on m->g0 stack
runtime stack:
runtime.throw({0x1522440?, 0x20?})
	/usr/local/go/src/runtime/panic.go:1047 +0x65 fp=0xc00083f748 sp=0xc00083f718 pc=0x2bc045
runtime.badmcall(0xa?)
	/usr/local/go/src/runtime/proc.go:468 +0x25 fp=0xc00083f768 sp=0xc00083f748 pc=0x2bf2c5
runtime.badmcall(0x0)
	<autogenerated>:1 +0x2c fp=0xc00083f780 sp=0xc00083f768 pc=0x2ec86c
runtime.asyncPreempt2()
	/usr/local/go/src/runtime/preempt.go:308 +0x3f fp=0xc00083f7a0 sp=0xc00083f780 pc=0x2bd27f
runtime.asyncPreempt()
	/usr/local/go/src/runtime/preempt_amd64.s:50 +0xdb fp=0xc00083f928 sp=0xc00083f7a0 pc=0x2ebbbb
github.com/dustin/go-humanize.revfmap(0xc00024f680)
	/builds/my-project/vendor/github.com/dustin/go-humanize/si.go:33 fp=0xc00083f930 sp=0xc00083f928 pc=0xe36ac0
github.com/dustin/go-humanize.init()
	/builds/my-project/vendor/github.com/dustin/go-humanize/si.go:30 +0x142b fp=0xc00083f990 sp=0xc00083f930 pc=0xe3872b
runtime.doInit(0x2156de0)
	/usr/local/go/src/runtime/proc.go:6329 +0x12d fp=0xc00083fac0 sp=0xc00083f990 pc=0x2cb2ed
runtime.doInit(0x2157c80)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fbf0 sp=0xc00083fac0 pc=0x2cb231
runtime.doInit(0x215a020)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fd20 sp=0xc00083fbf0 pc=0x2cb231
runtime.doInit(0x2164660)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fe50 sp=0xc00083fd20 pc=0x2cb231
runtime.doInit(0x2151b00)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083ff80 sp=0xc00083fe50 pc=0x2cb231
runtime.main()
	/usr/local/go/src/runtime/proc.go:233 +0x1bf fp=0xc00083ffe0 sp=0xc00083ff80 pc=0x2be73f
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00083ffe8 sp=0xc00083ffe0 pc=0x2ea561
goroutine 1 [runnable, locked to thread]:
github.com/dustin/go-humanize.revfmap(0xc00024f680?)
	/builds/my-project/vendor/github.com/dustin/go-humanize/si.go:33 +0xf0 fp=0xc00083f930 sp=0xc00083f928 pc=0xe36bb0
github.com/dustin/go-humanize.init()
	/builds/my-project/vendor/github.com/dustin/go-humanize/si.go:30 +0x142b fp=0xc00083f990 sp=0xc00083f930 pc=0xe3872b
runtime.doInit(0x2156de0)
	/usr/local/go/src/runtime/proc.go:6329 +0x12d fp=0xc00083fac0 sp=0xc00083f990 pc=0x2cb2ed
runtime.doInit(0x2157c80)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fbf0 sp=0xc00083fac0 pc=0x2cb231
runtime.doInit(0x215a020)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fd20 sp=0xc00083fbf0 pc=0x2cb231
runtime.doInit(0x2164660)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fe50 sp=0xc00083fd20 pc=0x2cb231
runtime.doInit(0x2151b00)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083ff80 sp=0xc00083fe50 pc=0x2cb231
runtime.main()
	/usr/local/go/src/runtime/proc.go:233 +0x1bf fp=0xc00083ffe0 sp=0xc00083ff80 pc=0x2be73f
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00083ffe8 sp=0xc00083ffe0 pc=0x2ea561
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000f3fb0 sp=0xc0000f3f90 pc=0x2beb16
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:369
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:302 +0xb1 fp=0xc0000f3fe0 sp=0xc0000f3fb0 pc=0x2be9b1
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000f3fe8 sp=0xc0000f3fe0 pc=0x2ea561
created by runtime.init.6
	/usr/local/go/src/runtime/proc.go:290 +0x25
goroutine 18 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00010ff90 sp=0xc00010ff70 pc=0x2beb16
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
	/usr/local/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc00010ffc8 sp=0xc00010ff90 pc=0x2a7c37
runtime.gcenable.func1()
	/usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc00010ffe0 sp=0xc00010ffc8 pc=0x29c786
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00010ffe8 sp=0xc00010ffe0 pc=0x2ea561
created by runtime.gcenable
	/usr/local/go/src/runtime/mgc.go:178 +0x6b
goroutine 19 [GC scavenge wait]:
runtime.gopark(0xc000106000?, 0x179b290?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000111f70 sp=0xc000111f50 pc=0x2beb16
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:369
runtime.(*scavengerState).park(0x2432600)
	/usr/local/go/src/runtime/mgcscavenge.go:389 +0x53 fp=0xc000111fa0 sp=0xc000111f70 pc=0x2a5c73
runtime.bgscavenge(0x0?)
	/usr/local/go/src/runtime/mgcscavenge.go:622 +0x65 fp=0xc000111fc8 sp=0xc000111fa0 pc=0x2a6285
runtime.gcenable.func2()
	/usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000111fe0 sp=0xc000111fc8 pc=0x29c726
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000111fe8 sp=0xc000111fe0 pc=0x2ea561
created by runtime.gcenable
	/usr/local/go/src/runtime/mgc.go:179 +0xaa
goroutine 34 [finalizer wait]:
runtime.gopark(0x0?, 0xc0000f7e70?, 0x6b?, 0x7d?, 0xc0000f7f70?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000f7e28 sp=0xc0000f7e08 pc=0x2beb16
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:369
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:180 +0x10f fp=0xc0000f7fe0 sp=0xc0000f7e28 pc=0x29b88f
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0x2ea561
created by runtime.createfing
	/usr/local/go/src/runtime/mfinal.go:157 +0x45
goroutine 35 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00010bf50 sp=0xc00010bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00010bfe0 sp=0xc00010bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00010bfe8 sp=0xc00010bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 3 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000f5f50 sp=0xc0000f5f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0000f5fe0 sp=0xc0000f5f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000f5fe8 sp=0xc0000f5fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 20 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00011df50 sp=0xc00011df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00011dfe0 sp=0xc00011df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011dfe8 sp=0xc00011dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 21 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00011ff50 sp=0xc00011ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00011ffe0 sp=0xc00011ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011ffe8 sp=0xc00011ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 36 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00010df50 sp=0xc00010df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00010dfe0 sp=0xc00010df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00010dfe8 sp=0xc00010dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 37 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000119f50 sp=0xc000119f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000119fe0 sp=0xc000119f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000119fe8 sp=0xc000119fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 4 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000505f50 sp=0xc000505f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000505fe0 sp=0xc000505f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000505fe8 sp=0xc000505fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 5 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000507f50 sp=0xc000507f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000507fe0 sp=0xc000507f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000507fe8 sp=0xc000507fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 6 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000501f50 sp=0xc000501f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000501fe0 sp=0xc000501f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000501fe8 sp=0xc000501fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 22 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000127f50 sp=0xc000127f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000127fe0 sp=0xc000127f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000127fe8 sp=0xc000127fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 23 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000129f50 sp=0xc000129f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000129fe0 sp=0xc000129f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000129fe8 sp=0xc000129fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 38 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00011bf50 sp=0xc00011bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00011bfe0 sp=0xc00011bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011bfe8 sp=0xc00011bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 7 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000503f50 sp=0xc000503f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000503fe0 sp=0xc000503f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000503fe8 sp=0xc000503fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 24 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000123f50 sp=0xc000123f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000123fe0 sp=0xc000123f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000123fe8 sp=0xc000123fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 25 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000125f50 sp=0xc000125f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000125fe0 sp=0xc000125f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000125fe8 sp=0xc000125fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 8 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00050df50 sp=0xc00050df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00050dfe0 sp=0xc00050df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00050dfe8 sp=0xc00050dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 26 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000509f50 sp=0xc000509f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000509fe0 sp=0xc000509f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000509fe8 sp=0xc000509fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 9 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00050ff50 sp=0xc00050ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00050ffe0 sp=0xc00050ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00050ffe8 sp=0xc00050ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 27 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00050bf50 sp=0xc00050bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00050bfe0 sp=0xc00050bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00050bfe8 sp=0xc00050bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 10 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000515f50 sp=0xc000515f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000515fe0 sp=0xc000515f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000515fe8 sp=0xc000515fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 28 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000511f50 sp=0xc000511f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000511fe0 sp=0xc000511f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000511fe8 sp=0xc000511fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 11 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000517f50 sp=0xc000517f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000517fe0 sp=0xc000517f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000517fe8 sp=0xc000517fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 29 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000513f50 sp=0xc000513f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000513fe0 sp=0xc000513f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000513fe8 sp=0xc000513fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 12 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00051df50 sp=0xc00051df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00051dfe0 sp=0xc00051df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00051dfe8 sp=0xc00051dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 30 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000519f50 sp=0xc000519f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000519fe0 sp=0xc000519f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000519fe8 sp=0xc000519fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 13 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00051ff50 sp=0xc00051ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00051ffe0 sp=0xc00051ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00051ffe8 sp=0xc00051ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 31 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00051bf50 sp=0xc00051bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00051bfe0 sp=0xc00051bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00051bfe8 sp=0xc00051bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 14 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000525f50 sp=0xc000525f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000525fe0 sp=0xc000525f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000525fe8 sp=0xc000525fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 39 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000521f50 sp=0xc000521f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000521fe0 sp=0xc000521f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000521fe8 sp=0xc000521fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 15 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000527f50 sp=0xc000527f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000527fe0 sp=0xc000527f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000527fe8 sp=0xc000527fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 40 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000523f50 sp=0xc000523f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000523fe0 sp=0xc000523f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000523fe8 sp=0xc000523fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 16 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052ff50 sp=0xc00052ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00052ffe0 sp=0xc00052ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052ffe8 sp=0xc00052ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 41 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052bf50 sp=0xc00052bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00052bfe0 sp=0xc00052bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052bfe8 sp=0xc00052bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 50 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000531f50 sp=0xc000531f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000531fe0 sp=0xc000531f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000531fe8 sp=0xc000531fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 42 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052df50 sp=0xc00052df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00052dfe0 sp=0xc00052df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052dfe8 sp=0xc00052dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 51 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000537f50 sp=0xc000537f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000537fe0 sp=0xc000537f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000537fe8 sp=0xc000537fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 43 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000533f50 sp=0xc000533f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000533fe0 sp=0xc000533f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000533fe8 sp=0xc000533fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 52 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000539f50 sp=0xc000539f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000539fe0 sp=0xc000539f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000539fe8 sp=0xc000539fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 44 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000535f50 sp=0xc000535f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000535fe0 sp=0xc000535f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000535fe8 sp=0xc000535fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 53 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00053ff50 sp=0xc00053ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00053ffe0 sp=0xc00053ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00053ffe8 sp=0xc00053ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 45 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00053bf50 sp=0xc00053bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00053bfe0 sp=0xc00053bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00053bfe8 sp=0xc00053bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 54 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000541f50 sp=0xc000541f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000541fe0 sp=0xc000541f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000541fe8 sp=0xc000541fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 46 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00053df50 sp=0xc00053df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00053dfe0 sp=0xc00053df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00053dfe8 sp=0xc00053dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 55 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000547f50 sp=0xc000547f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000547fe0 sp=0xc000547f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000547fe8 sp=0xc000547fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 47 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000543f50 sp=0xc000543f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000543fe0 sp=0xc000543f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000543fe8 sp=0xc000543fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 56 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000549f50 sp=0xc000549f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000549fe0 sp=0xc000549f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000549fe8 sp=0xc000549fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 48 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000545f50 sp=0xc000545f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000545fe0 sp=0xc000545f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000545fe8 sp=0xc000545fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 57 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00054ff50 sp=0xc00054ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00054ffe0 sp=0xc00054ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00054ffe8 sp=0xc00054ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 49 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00054bf50 sp=0xc00054bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00054bfe0 sp=0xc00054bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00054bfe8 sp=0xc00054bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 58 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000551f50 sp=0xc000551f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000551fe0 sp=0xc000551f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000551fe8 sp=0xc000551fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 66 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00054df50 sp=0xc00054df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00054dfe0 sp=0xc00054df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00054dfe8 sp=0xc00054dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 59 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000557f50 sp=0xc000557f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000557fe0 sp=0xc000557f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000557fe8 sp=0xc000557fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 67 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000553f50 sp=0xc000553f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000553fe0 sp=0xc000553f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000553fe8 sp=0xc000553fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 60 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000559f50 sp=0xc000559f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000559fe0 sp=0xc000559f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000559fe8 sp=0xc000559fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 68 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000555f50 sp=0xc000555f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000555fe0 sp=0xc000555f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000555fe8 sp=0xc000555fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 61 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00055ff50 sp=0xc00055ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00055ffe0 sp=0xc00055ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00055ffe8 sp=0xc00055ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 69 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00055bf50 sp=0xc00055bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00055bfe0 sp=0xc00055bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00055bfe8 sp=0xc00055bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 62 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000561f50 sp=0xc000561f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000561fe0 sp=0xc000561f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000561fe8 sp=0xc000561fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 70 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00055df50 sp=0xc00055df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00055dfe0 sp=0xc00055df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00055dfe8 sp=0xc00055dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 63 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000567f50 sp=0xc000567f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000567fe0 sp=0xc000567f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000567fe8 sp=0xc000567fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 71 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000563f50 sp=0xc000563f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000563fe0 sp=0xc000563f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000563fe8 sp=0xc000563fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 64 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000569f50 sp=0xc000569f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000569fe0 sp=0xc000569f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000569fe8 sp=0xc000569fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 72 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000565f50 sp=0xc000565f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000565fe0 sp=0xc000565f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000565fe8 sp=0xc000565fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 65 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00056ff50 sp=0xc00056ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00056ffe0 sp=0xc00056ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00056ffe8 sp=0xc00056ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 83 [select]:
runtime.gopark(0xc000571f88?, 0x3?, 0xc8?, 0xc?, 0xc000571f72?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000571df8 sp=0xc000571dd8 pc=0x2beb16
runtime.selectgo(0xc000571f88, 0xc000571f6c, 0xc0001d2500?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000571f38 sp=0xc000571df8 pc=0x2ccebc
go.opencensus.io/stats/view.(*worker).start(0xc0001d2500)
	/builds/my-project/vendor/go.opencensus.io/stats/view/worker.go:292 +0xad fp=0xc000571fc8 sp=0xc000571f38 pc=0x977c2d
go.opencensus.io/stats/view.init.0.func1()
	/builds/my-project/vendor/go.opencensus.io/stats/view/worker.go:34 +0x26 fp=0xc000571fe0 sp=0xc000571fc8 pc=0x976e66
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000571fe8 sp=0xc000571fe0 pc=0x2ea561
created by go.opencensus.io/stats/view.init.0
	/builds/my-project/vendor/go.opencensus.io/stats/view/worker.go:34 +0x8d

Then here are some callstacks from init's that are seemingly not following the right init order:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x5 pc=0xb84adc]
goroutine 1 [running]:
golang.org/x/text/internal/language.addTags({0x139, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0}})
	golang.org/x/text@v0.14.0/internal/language/match.go:99 +0x27c
golang.org/x/text/internal/language.Tag.Maximize(...)
	golang.org/x/text@v0.14.0/internal/language/match.go:68
golang.org/x/text/language.init.0()
	golang.org/x/text@v0.14.0/language/match.go:727 +0x26d
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x9 pc=0x132a0c0]
goroutine 1 [running]:
google.golang.org/protobuf/internal/filedesc.(*Enums).Len(0x3023620?)
	/builds/my-project/vendor/google.golang.org/protobuf/internal/filedesc/desc_list_gen.go:25
google.golang.org/protobuf/reflect/protoregistry.rangeTopLevelDescriptors({0x23b2710, 0xc0003fc000}, 0xc00035f288)
	/builds/my-project/vendor/google.golang.org/protobuf/reflect/protoregistry/registry.go:408 +0x5d
google.golang.org/protobuf/reflect/protoregistry.(*Files).RegisterFile(0xc0000081f8, {0x23b2710?, 0xc0003fc000?})
	/builds/my-project/vendor/google.golang.org/protobuf/reflect/protoregistry/registry.go:149 +0x739
google.golang.org/protobuf/internal/filedesc.Builder.Build({{0x1dafbd0, 0x2e}, {0x2d53d20, 0x1bf, 0x1bf}, 0x1, 0x0, 0x0, 0x0, {0x239d908, ...}, ...})
	/builds/my-project/vendor/google.golang.org/protobuf/internal/filedesc/build.go:112 +0x1d6
google.golang.org/protobuf/internal/filetype.Builder.Build({{{0x1dafbd0, 0x2e}, {0x2d53d20, 0x1bf, 0x1bf}, 0x1, 0x0, 0x0, 0x0, {0x0, ...}, ...}, ...})
	/builds/my-project/vendor/google.golang.org/protobuf/internal/filetype/build.go:138 +0x1b8
google.golang.org/genproto/googleapis/rpc/code.file_google_rpc_code_proto_init()
	/builds/my-project/vendor/google.golang.org/genproto/googleapis/rpc/code/code.pb.go:331 +0x198
google.golang.org/genproto/googleapis/rpc/code.init.0()
	/builds/my-project/vendor/google.golang.org/genproto/googleapis/rpc/code/code.pb.go:313 +0x17

What did you expect to see?

The program finishes the init() phase without panics

@gabyhelp's overview of this issue: #67108 (comment)

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Apr 29, 2024
@RichieSams
Copy link
Author

Below is my go.mod file (with a few redactions)

module my-project/v3

require (
	github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2
	github.com/antlr4-go/antlr/v4 v4.13.0
	github.com/bmatcuk/doublestar v1.3.4
	github.com/bsipos/thist v1.0.0
	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc
	github.com/dustin/go-humanize v1.0.1
	github.com/getsentry/sentry-go v0.27.0
	github.com/go-ole/go-ole v1.3.0
	github.com/gobwas/glob v0.2.3
	github.com/gofrs/flock v0.8.1
	github.com/gonum/stat v0.0.0-20181125101827-41a0da705a5b
	github.com/gorilla/handlers v1.5.2
	github.com/gorilla/mux v1.8.1
	github.com/hashicorp/go-multierror v1.1.1
	github.com/jotfs/fastcdc-go v0.2.0
	github.com/json-iterator/go v1.1.12
	github.com/juju/ratelimit v1.0.2
	github.com/karrick/godirwalk v1.16.2
	github.com/olekukonko/tablewriter v0.0.5
	github.com/pkg/errors v0.9.1
	github.com/restic/chunker v0.4.0
	github.com/satori/go.uuid v1.2.0
	github.com/schollz/progressbar/v3 v3.14.2
	github.com/sirupsen/logrus v1.9.3
	github.com/skratchdot/open-golang v0.0.0-20190104022628-a2dfa6d0dab6
	github.com/spf13/cobra v1.8.0
	github.com/spf13/viper v1.18.2
	github.com/stretchr/testify v1.9.0
	github.com/thediveo/enumflag v0.10.1
	github.com/tinylib/msgp v1.1.9
	github.com/xhit/go-str2duration/v2 v2.1.0
	internal-a v1.7.0
	internal-b v1.2.0
	internal-c v1.7.0
	internal-d/v2 v2.70.0
	go.opentelemetry.io/otel v1.25.0
	go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.25.0
	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.25.0
	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.25.0
	go.opentelemetry.io/otel/metric v1.25.0
	go.opentelemetry.io/otel/sdk v1.25.0
	go.opentelemetry.io/otel/sdk/metric v1.25.0
	go.opentelemetry.io/otel/trace v1.25.0
	golang.org/x/sync v0.7.0
	golang.org/x/sys v0.19.0
	golang.org/x/term v0.19.0
	golang.org/x/text v0.14.0
	google.golang.org/grpc v1.63.2
	gopkg.in/yaml.v2 v2.4.0
)

require (
	cloud.google.com/go v0.112.0 // indirect
	cloud.google.com/go/compute v1.24.0 // indirect
	cloud.google.com/go/compute/metadata v0.2.3 // indirect
	cloud.google.com/go/iam v1.1.6 // indirect
	cloud.google.com/go/storage v1.36.0 // indirect
	github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 // indirect
	github.com/Microsoft/go-winio v0.6.0 // indirect
	github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5 // indirect
	github.com/Shopify/toxiproxy/v2 v2.4.0 // indirect
	github.com/ajstarks/svgo v0.0.0-20180226025133-644b8db467af // indirect
	github.com/aws/aws-sdk-go v1.28.9 // indirect
	github.com/beorn7/perks v1.0.1 // indirect
	github.com/cenkalti/backoff/v4 v4.3.0 // indirect
	github.com/cespare/xxhash/v2 v2.2.0 // indirect
	github.com/containerd/continuity v0.3.0 // indirect
	github.com/cpuguy83/go-md2man/v2 v2.0.3 // indirect
	github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
	github.com/docker/cli v20.10.17+incompatible // indirect
	github.com/docker/docker v20.10.7+incompatible // indirect
	github.com/docker/go-connections v0.4.0 // indirect
	github.com/docker/go-units v0.4.0 // indirect
	github.com/fatih/color v1.14.1 // indirect
	github.com/felixge/httpsnoop v1.0.4 // indirect
	github.com/fogleman/gg v1.2.1-0.20190220221249-0403632d5b90 // indirect
	github.com/fsnotify/fsnotify v1.7.0 // indirect
	github.com/go-logr/logr v1.4.1 // indirect
	github.com/go-logr/stdr v1.2.2 // indirect
	github.com/gogo/protobuf v1.3.2 // indirect
	github.com/golang-jwt/jwt/v5 v5.0.0 // indirect
	github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0 // indirect
	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
	github.com/golang/protobuf v1.5.4 // indirect
	github.com/gonum/blas v0.0.0-20181208220705-f22b278b28ac // indirect
	github.com/gonum/floats v0.0.0-20181209220543-c233463c7e82 // indirect
	github.com/gonum/integrate v0.0.0-20181209220457-a422b5c0fdf2 // indirect
	github.com/gonum/internal v0.0.0-20181124074243-f884aa714029 // indirect
	github.com/gonum/lapack v0.0.0-20181123203213-e4cdc5a0bff9 // indirect
	github.com/gonum/matrix v0.0.0-20181209220409-c518dec07be9 // indirect
	github.com/google/s2a-go v0.1.7 // indirect
	github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
	github.com/google/uuid v1.6.0 // indirect
	github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
	github.com/googleapis/gax-go/v2 v2.12.0 // indirect
	github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.0 // indirect
	github.com/hashicorp/errwrap v1.0.0 // indirect
	github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
	github.com/hashicorp/go-hclog v1.5.0 // indirect
	github.com/hashicorp/go-retryablehttp v0.7.0 // indirect
	github.com/hashicorp/hcl v1.0.0 // indirect
	github.com/imdario/mergo v0.3.12 // indirect
	github.com/inconshreveable/mousetrap v1.1.0 // indirect
	github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af // indirect
	github.com/jung-kurt/gofpdf v1.0.3-0.20190309125859-24315acbbda5 // indirect
	github.com/kylelemons/godebug v1.1.0 // indirect
	github.com/magiconair/properties v1.8.7 // indirect
	github.com/mattn/go-colorable v0.1.13 // indirect
	github.com/mattn/go-isatty v0.0.20 // indirect
	github.com/mattn/go-runewidth v0.0.14 // indirect
	github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
	github.com/minio/minio-go/v6 v6.0.46 // indirect
	github.com/minio/sha256-simd v0.1.1 // indirect
	github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect
	github.com/mitchellh/go-homedir v1.1.0 // indirect
	github.com/mitchellh/mapstructure v1.5.0 // indirect
	github.com/moby/term v0.0.0-20201216013528-df9cb8a40635 // indirect
	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
	github.com/modern-go/reflect2 v1.0.2 // indirect
	github.com/opencontainers/go-digest v1.0.0 // indirect
	github.com/opencontainers/image-spec v1.0.2 // indirect
	github.com/opencontainers/runc v1.1.5 // indirect
	github.com/ory/dockertest/v3 v3.10.0 // indirect
	github.com/pelletier/go-toml/v2 v2.1.0 // indirect
	github.com/philhofer/fwd v1.1.2 // indirect
	github.com/pierrec/lz4 v2.0.5+incompatible // indirect
	github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
	github.com/prometheus/client_golang v1.12.1 // indirect
	github.com/prometheus/client_model v0.2.0 // indirect
	github.com/prometheus/common v0.32.1 // indirect
	github.com/prometheus/procfs v0.7.3 // indirect
	github.com/rivo/uniseg v0.4.7 // indirect
	github.com/russross/blackfriday/v2 v2.1.0 // indirect
	github.com/sagikazarmark/locafero v0.4.0 // indirect
	github.com/sagikazarmark/slog-shim v0.1.0 // indirect
	github.com/sourcegraph/conc v0.3.0 // indirect
	github.com/spf13/afero v1.11.0 // indirect
	github.com/spf13/cast v1.6.0 // indirect
	github.com/spf13/pflag v1.0.5 // indirect
	github.com/subosito/gotenv v1.6.0 // indirect
	github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f // indirect
	github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
	github.com/xeipuuv/gojsonschema v1.2.0 // indirect
	go.opencensus.io v0.24.0 // indirect
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.47.0 // indirect
	go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace v0.30.0 // indirect
	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.47.0 // indirect
	go.opentelemetry.io/proto/otlp v1.1.0 // indirect
	go.uber.org/atomic v1.9.0 // indirect
	go.uber.org/multierr v1.9.0 // indirect
	golang.org/x/crypto v0.21.0 // indirect
	golang.org/x/exp v0.0.0-20230905200255-921286631fa9 // indirect
	golang.org/x/image v0.0.0-20190802002840-cff245a6509b // indirect
	golang.org/x/mod v0.13.0 // indirect
	golang.org/x/net v0.23.0 // indirect
	golang.org/x/oauth2 v0.17.0 // indirect
	golang.org/x/time v0.5.0 // indirect
	golang.org/x/tools v0.14.0 // indirect
	gonum.org/v1/plot v0.0.0-20190515093506-e2840ee46a6b // indirect
	google.golang.org/api v0.162.0 // indirect
	google.golang.org/appengine v1.6.8 // indirect
	google.golang.org/genproto v0.0.0-20240227224415-6ceb2ff114de // indirect
	google.golang.org/genproto/googleapis/api v0.0.0-20240227224415-6ceb2ff114de // indirect
	google.golang.org/genproto/googleapis/rpc v0.0.0-20240401170217-c3f982113cda // indirect
	google.golang.org/protobuf v1.33.0 // indirect
	gopkg.in/ini.v1 v1.67.0 // indirect
	gopkg.in/yaml.v3 v3.0.1 // indirect
	nhooyr.io/websocket v1.8.10 // indirect
)

replace github.com/antlr/antlr4 v0.0.0-20180905200841-432022fc1ca0 => github.com/michilu/antlr4 v0.0.0-20180803091604-411960b1878f

replace github.com/uber/jaeger-lib v2.0.0+incompatible => github.com/uber/jaeger-lib v1.5.0

replace github.com/skratchdot/open-golang => github.com/seanhoughton/open-golang v0.0.0-20190322203133-7972e18a724d

replace git.apache.org/thrift.git => github.com/apache/thrift v0.12.0

go 1.21

@RichieSams
Copy link
Author

RichieSams commented Apr 29, 2024

This seems to be related to: #56774

But in that issue, the poster said updating to 1.21.3 fixed it. However, we're using 1.21.4, and still seeing the issue.

@RichieSams
Copy link
Author

RichieSams commented Apr 30, 2024

Slightly different panic:

unexpected fault address 0x411a89e
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0x411a89e pc=0xe003e1]
goroutine 1 [running, locked to thread]:
runtime.throw({0x1f23e6f?, 0x0?})
	runtime/panic.go:1077 +0x65 fp=0xc0000fbd58 sp=0xc0000fbd28 pc=0xd9c7a5
runtime.sigpanic()
	runtime/signal_windows.go:377 +0xd0 fp=0xc0000fbda0 sp=0xc0000fbd58 pc=0xdb06b0
time.map.init.1()
	time/zoneinfo_abbrs_windows.go:154 +0x81 fp=0xc0000fbe00 sp=0xc0000fbda0 pc=0xe003e1
time.init()
	time/zoneinfo_abbrs_windows.go:15 +0x25 fp=0xc0000fbe10 sp=0xc0000fbe00 pc=0xe001a5
runtime.doInit1(0x2b81dc0)
	runtime/proc.go:6740 +0xdd fp=0xc0000fbf40 sp=0xc0000fbe10 pc=0xdabf1d
runtime.doInit(...)
	runtime/proc.go:6707
runtime.main()
	runtime/proc.go:249 +0x356 fp=0xc0000fbfe0 sp=0xc0000fbf40 pc=0xd9f096
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000fbfe8 sp=0xc0000fbfe0 pc=0xdcd141
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f7fa8 sp=0xc0000f7f88 pc=0xd9f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.forcegchelper()
	runtime/proc.go:322 +0xb8 fp=0xc0000f7fe0 sp=0xc0000f7fa8 pc=0xd9f298
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0xdcd141
created by runtime.init.6 in goroutine 1
	runtime/proc.go:310 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f9f78 sp=0xc0000f9f58 pc=0xd9f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.bgsweep(0x0?)
	runtime/mgcsweep.go:280 +0x94 fp=0xc0000f9fc8 sp=0xc0000f9f78 pc=0xd88ab4
runtime.gcenable.func1()
	runtime/mgc.go:200 +0x25 fp=0xc0000f9fe0 sp=0xc0000f9fc8 pc=0xd7dc25
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f9fe8 sp=0xc0000f9fe0 pc=0xdcd141
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:200 +0x66
goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc0000480e0?, 0x21b8428?, 0x1?, 0x0?, 0xc0000f4b60?)
	runtime/proc.go:398 +0xce fp=0xc000109f70 sp=0xc000109f50 pc=0xd9f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.(*scavengerState).park(0x2e63e20)
	runtime/mgcscavenge.go:425 +0x49 fp=0xc000109fa0 sp=0xc000109f70 pc=0xd86369
runtime.bgscavenge(0x0?)
	runtime/mgcscavenge.go:653 +0x3c fp=0xc000109fc8 sp=0xc000109fa0 pc=0xd868fc
runtime.gcenable.func2()
	runtime/mgc.go:201 +0x25 fp=0xc000109fe0 sp=0xc000109fc8 pc=0xd7dbc5
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc000109fe8 sp=0xc000109fe0 pc=0xdcd141
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:201 +0xa5

@RichieSams RichieSams changed the title runtime: mcall called on m->g0 stack panic: runtime: mcall called on m->g0 stack May 1, 2024
@RichieSams RichieSams changed the title panic: runtime: mcall called on m->g0 stack fatal error: runtime: mcall called on m->g0 stack May 1, 2024
@RichieSams
Copy link
Author

A new type of panic:

fatal error: runtime: name offset out of range
goroutine 1 [running, locked to thread]:
runtime.throw({0x187d65c?, 0x1850bb5?})
	runtime/panic.go:1077 +0x65 fp=0xc0000fbcf8 sp=0xc0000fbcc8 pc=0x68c7a5
runtime.resolveNameOff(0x284d14a?, 0x6c5160)
	runtime/type.go:119 +0x245 fp=0xc0000fbd50 sp=0xc0000fbcf8 pc=0x6b1765
internal/reflectlite.resolveNameOff(0x0?, 0x0?)
	runtime/runtime1.go:626 +0x13 fp=0xc0000fbd70 sp=0xc0000fbd50 pc=0x6b82d3
internal/reflectlite.rtype.nameOff(...)
	internal/reflectlite/type.go:244
internal/reflectlite.rtype.String({0xc000218030?})
	internal/reflectlite/type.go:256 +0x1b fp=0xc0000fbd98 sp=0xc0000fbd70 pc=0x6c4d7b
internal/reflectlite.elem(0xc0000fbdf0?)
	internal/reflectlite/type.go:320 +0x73 fp=0xc0000fbdd0 sp=0xc0000fbd98 pc=0x6c5113
internal/reflectlite.rtype.Elem({0xc000218000?})
	internal/reflectlite/type.go:324 +0x13 fp=0xc0000fbde8 sp=0xc0000fbdd0 pc=0x6c5173
errors.init()
	errors/wrap.go:136 +0x43 fp=0xc0000fbe10 sp=0xc0000fbde8 pc=0x6c90c3
runtime.doInit1(0x24bccc0)
	runtime/proc.go:6740 +0xdd fp=0xc0000fbf40 sp=0xc0000fbe10 pc=0x69bf1d
runtime.doInit(...)
	runtime/proc.go:6707
runtime.main()
	runtime/proc.go:249 +0x356 fp=0xc0000fbfe0 sp=0xc0000fbf40 pc=0x68f096
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000fbfe8 sp=0xc0000fbfe0 pc=0x6bd241
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f7fa8 sp=0xc0000f7f88 pc=0x68f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.forcegchelper()
	runtime/proc.go:322 +0xb8 fp=0xc0000f7fe0 sp=0xc0000f7fa8 pc=0x68f298
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0x6bd241
created by runtime.init.6 in goroutine 1
	runtime/proc.go:310 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f9f78 sp=0xc0000f9f58 pc=0x68f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.bgsweep(0x0?)
	runtime/mgcsweep.go:280 +0x94 fp=0xc0000f9fc8 sp=0xc0000f9f78 pc=0x678ab4
runtime.gcenable.func1()
	runtime/mgc.go:200 +0x25 fp=0xc0000f9fe0 sp=0xc0000f9fc8 pc=0x66dc25
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f9fe8 sp=0xc0000f9fe0 pc=0x6bd241
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:200 +0x66
goroutine 18 [GC scavenge wait]:
runtime.gopark(0xc0000480e0?, 0x1ae7600?, 0x1?, 0x0?, 0xc0002041a0?)
	runtime/proc.go:398 +0xce fp=0xc00020df70 sp=0xc00020df50 pc=0x68f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.(*scavengerState).park(0x27ea440)
	runtime/mgcscavenge.go:425 +0x49 fp=0xc00020dfa0 sp=0xc00020df70 pc=0x676369
runtime.bgscavenge(0x0?)
	runtime/mgcscavenge.go:653 +0x3c fp=0xc00020dfc8 sp=0xc00020dfa0 pc=0x6768fc
runtime.gcenable.func2()
	runtime/mgc.go:201 +0x25 fp=0xc00020dfe0 sp=0xc00020dfc8 pc=0x66dbc5
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc00020dfe8 sp=0xc00020dfe0 pc=0x6bd241
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:201 +0xa5

@cherrymui
Copy link
Member

NOTE: We compile inside docker. The output below is from running go env within the docker container that we use. We cross-compile for linux, windows, and darwin. The workers having a problem in this case are running windows/amd64.

These workers are bare metal AMD Threadripper machines running Windows 10.

Just to be clear, you're cross compiling inside docker, but running the program on real Windows machines? Do you see it only on a specific type of Windows machines? Thanks.

@cherrymui cherrymui added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label May 6, 2024
@RichieSams
Copy link
Author

RichieSams commented May 6, 2024

Just to be clear, you're cross compiling inside docker, but running the program on real Windows machines?

Correct. We cross compile in docker. But we're running in native Windows.

Do you see it only on a specific type of Windows machines? Thanks.

Yes. We only see it on our AMD servers. Example specs of the machines is listed above in the original post. None of our Intel-based servers are seeing these panics

Let me know if you have any other questions. :)

@cherrymui cherrymui changed the title fatal error: runtime: mcall called on m->g0 stack runtime: fatal error: mcall called on m->g0 stack on Windows May 6, 2024
@cherrymui
Copy link
Member

Thanks. Have you tried running the program under the race detector? Is the program using cgo? Is there a way we can reproduce the issue ourselves?

@cherrymui cherrymui added this to the Backlog milestone May 6, 2024
@cherrymui
Copy link
Member

cc @golang/windows

@RichieSams
Copy link
Author

Have you tried running the program under the race detector?

No, but that's a good idea. I'll give that a shot today

Is the program using cgo?

No, we disable it with CGO_ENABLED=0 when compiling

Is there a way we can reproduce the issue ourselves?

Unfortunately, it's a proprietary program. But given it's panic'ing on init() and in standard libraries, I'm going to see if I can create a small "hello world" test program and just run it continuously on the machines to see if I can reproduce it.

@RichieSams
Copy link
Author

I'm wondering if there is some small incorrect assumption with AMD hardware in the golang runtime code? I see this other issue, which isn't the same stack trace, but is also limited to AMD hardware: #62440

@RichieSams
Copy link
Author

Some more callstacks from our servers:

fatal: morestack on g0
Exception 0x80000003 0x0 0x0 0xccfa1
PC=0xccfa1
runtime.abort()
	runtime/asm_amd64.s:1113 +0x1 fp=0xc0000f9bc0 sp=0xc0000f9bb8 pc=0xccfa1
traceback: unexpected SPWRITE function runtime.morestack
runtime.morestack()
	runtime/asm_amd64.s:560 +0x24 fp=0xc0000f9bc8 sp=0xc0000f9bc0 pc=0xcb204
goroutine 1 [running, locked to thread]:
runtime.doInit1(0x1e82a00?)
	runtime/proc.go:6711 +0x4c5 fp=0xc0000f9f40 sp=0xc0000f9f38 pc=0xac305
runtime.doInit(...)
	runtime/proc.go:6707
runtime.main()
	runtime/proc.go:249 +0x356 fp=0xc0000f9fe0 sp=0xc0000f9f40 pc=0x9f096
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f9fe8 sp=0xc0000f9fe0 pc=0xcd141
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f5fa8 sp=0xc0000f5f88 pc=0x9f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.forcegchelper()
	runtime/proc.go:322 +0xb8 fp=0xc0000f5fe0 sp=0xc0000f5fa8 pc=0x9f298
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f5fe8 sp=0xc0000f5fe0 pc=0xcd141
created by runtime.init.6 in goroutine 1
	runtime/proc.go:310 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f7f78 sp=0xc0000f7f58 pc=0x9f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.bgsweep(0x0?)
	runtime/mgcsweep.go:280 +0x94 fp=0xc0000f7fc8 sp=0xc0000f7f78 pc=0x88ab4
runtime.gcenable.func1()
	runtime/mgc.go:200 +0x25 fp=0xc0000f7fe0 sp=0xc0000f7fc8 pc=0x7dc25
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0xcd141
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:200 +0x66
goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc0000440e0?, 0x14b8428?, 0x1?, 0x0?, 0xc0000f2b60?)
	runtime/proc.go:398 +0xce fp=0xc000107f70 sp=0xc000107f50 pc=0x9f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.(*scavengerState).park(0x2163e20)
	runtime/mgcscavenge.go:425 +0x49 fp=0xc000107fa0 sp=0xc000107f70 pc=0x86369
runtime.bgscavenge(0x0?)
	runtime/mgcscavenge.go:653 +0x3c fp=0xc000107fc8 sp=0xc000107fa0 pc=0x868fc
runtime.gcenable.func2()
	runtime/mgc.go:201 +0x25 fp=0xc000107fe0 sp=0xc000107fc8 pc=0x7dbc5
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc000107fe8 sp=0xc000107fe0 pc=0xcd141
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:201 +0xa5
rax     0x17
rbx     0x2164e80
rcx     0x2164c00
rdi     0xcc34a2e000
rsi     0xc0000f9ab8
rbp     0xc0000f9bd8
rsp     0xc0000f9bb8
r8      0xc0000f9888
r9      0xc0000f9a60
r10     0x0
r11     0x246
r12     0xcc34dff6d0
r13     0x1000
r14     0x21645e0
r15     0xb2c
rip     0xccfa1
rflags  0x212
cs      0x33
fs      0x53
gs      0x2b
panic: godebug: Value of name not listed in godebugs.All: 
goroutine 1 [running]:
internal/godebug.(*Setting).Value.func1()
	internal/godebug/godebug.go:141 +0xb9
sync.(*Once).doSlow(0x28ba410?, 0xc0000f9dd8?)
	sync/once.go:74 +0xbf
sync.(*Once).Do(...)
	sync/once.go:65
internal/godebug.(*Setting).Value(0x28ba410)
	internal/godebug/godebug.go:138 +0x48
internal/intern.safeMap(...)
	internal/intern/intern.go:74
internal/intern.init()
	internal/intern/intern.go:66 +0x48

@RichieSams
Copy link
Author

Another callstack:

panic: runtime error: makeslice: cap out of range
goroutine 1 [running]:
runtime/metrics.init.0()
	runtime/metrics/description.go:457 +0xa6

@RichieSams
Copy link
Author

fatal error: fault
[signal 0xc0000005 code=0x0 addr=0x1f800608 pc=0x142df7e]
goroutine 1 gp=0xc0000fc000 m=0 mp=0x3240880 [running, locked to thread]:
runtime.throw({0x22296ae?, 0xfe15f40ffe15f40f?})
	runtime/panic.go:1023 +0x65 fp=0xc00019d798 sp=0xc00019d768 pc=0x102ebc5
runtime.sigpanic()
	runtime/signal_windows.go:414 +0xd0 fp=0xc00019d7e0 sp=0xc00019d798 pc=0x1044670
golang.org/x/text/internal/language/compact.getCoreIndex.func1(0x249?)
	golang.org/x/text@v0.14.0/internal/language/compact/compact.go:32 +0x1e fp=0xc00019d800 sp=0xc00019d7e0 pc=0x142df7e
sort.Search(0xc0001ee060?, 0xc00019d848)
	sort/search.go:65 +0x46 fp=0xc00019d830 sp=0xc00019d800 pc=0x109db26
golang.org/x/text/internal/language/compact.getCoreIndex({0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0}})
	golang.org/x/text@v0.14.0/internal/language/compact/compact.go:31 +0x9f fp=0xc00019d888 sp=0xc00019d830 pc=0x142debf
golang.org/x/text/internal/language/compact.FromTag({0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0}})
	golang.org/x/text@v0.14.0/internal/language/compact/language.go:238 +0x5a5 fp=0xc00019db90 sp=0xc00019d888 pc=0x142f145
golang.org/x/text/internal/language/compact.Make({0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0}})
	golang.org/x/text@v0.14.0/internal/language/compact/language.go:53 +0x285 fp=0xc00019dc30 sp=0xc00019db90 pc=0x142e505
golang.org/x/text/language.makeTag(...)
	golang.org/x/text@v0.14.0/language/language.go:25
golang.org/x/text/language.CanonType.Parse(0x17, {0x2241259?, 0xc0001bc070?})
	golang.org/x/text@v0.14.0/language/parse.go:62 +0x1f7 fp=0xc00019dd58 sp=0xc00019dc30 pc=0x14343d7
golang.org/x/text/language.Parse(...)
	golang.org/x/text@v0.14.0/language/parse.go:35
golang.org/x/text/language.MustParse({0x2241259?, 0x15?})
	golang.org/x/text@v0.14.0/language/tags.go:14 +0x25 fp=0xc00019dd80 sp=0xc00019dd58 pc=0x1434645
golang.org/x/text/cases.init.0()
	golang.org/x/text@v0.14.0/cases/map.go:43 +0xa8 fp=0xc00019de20 sp=0xc00019dd80 pc=0x143aaa8
runtime.doInit1(0x2f02670)
	runtime/proc.go:7176 +0xed fp=0xc00019df50 sp=0xc00019de20 pc=0x103f82d
runtime.doInit(...)
	runtime/proc.go:7143
runtime.main()
	runtime/proc.go:253 +0x327 fp=0xc00019dfe0 sp=0xc00019df50 pc=0x1031587
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc00019dfe8 sp=0xc00019dfe0 pc=0x1062801
goroutine 18 gp=0xc0001841c0 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xce fp=0xc000101fa8 sp=0xc000101f88 pc=0x10318ee
runtime.goparkunlock(...)
	runtime/proc.go:408
runtime.forcegchelper()
	runtime/proc.go:326 +0xb8 fp=0xc000101fe0 sp=0xc000101fa8 pc=0x1031778
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc000101fe8 sp=0xc000101fe0 pc=0x1062801
created by runtime.init.6 in goroutine 1
	runtime/proc.go:314 +0x1a
goroutine 2 gp=0xc0000fca80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xce fp=0xc00010ff80 sp=0xc00010ff60 pc=0x10318ee
runtime.goparkunlock(...)
	runtime/proc.go:408
runtime.bgsweep(0xc0000480e0)
	runtime/mgcsweep.go:278 +0x94 fp=0xc00010ffc8 sp=0xc00010ff80 pc=0x101ab14
runtime.gcenable.gowrap1()
	runtime/mgc.go:203 +0x25 fp=0xc00010ffe0 sp=0xc00010ffc8 pc=0x100f405
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc00010ffe8 sp=0xc00010ffe0 pc=0x1062801
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:203 +0x66
goroutine 3 gp=0xc0000fcc40 m=nil [GC scavenge wait]:
runtime.gopark(0xc0000480e0?, 0x2508298?, 0x1?, 0x0?, 0xc0000fcc40?)
	runtime/proc.go:402 +0xce fp=0xc000111f78 sp=0xc000111f58 pc=0x10318ee
runtime.goparkunlock(...)
	runtime/proc.go:408
runtime.(*scavengerState).park(0x323ec40)
	runtime/mgcscavenge.go:425 +0x49 fp=0xc000111fa8 sp=0xc000111f78 pc=0x10184e9
runtime.bgscavenge(0xc0000480e0)
	runtime/mgcscavenge.go:653 +0x3c fp=0xc000111fc8 sp=0xc000111fa8 pc=0x1018a7c
runtime.gcenable.gowrap2()
	runtime/mgc.go:204 +0x25 fp=0xc000111fe0 sp=0xc000111fc8 pc=0x100f3a5
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc000111fe8 sp=0xc000111fe0 pc=0x1062801
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:204 +0xa5
goroutine 4 gp=0xc0000fce00 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xce fp=0xc00010be20 sp=0xc00010be00 pc=0x10318ee
runtime.runfinq()
	runtime/mfinal.go:194 +0x107 fp=0xc00010bfe0 sp=0xc00010be20 pc=0x100e487
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc00010bfe8 sp=0xc00010bfe0 pc=0x1062801
created by runtime.createfing in goroutine 1
	runtime/mfinal.go:164 +0x3d

@cherrymui
Copy link
Member

I'm wondering if there is some small incorrect assumption with AMD hardware in the golang runtime code? I see this other issue, which isn't the same stack trace, but is also limited to AMD hardware: #62440

It could be. It could also be a kernel issue. A reproducer would be very helpful. Thanks.

@cherrymui cherrymui added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels May 10, 2024
@dmitshur dmitshur added help wanted WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels May 15, 2024
@RichieSams
Copy link
Author

I'm attempting to get a minimal repro. I created something and I'm running it on all our workers periodically to try to get a failure. I'll report back here if I can get it to trigger.

@prattmic
Copy link
Member

prattmic commented Sep 6, 2024

Oh, note that the restic binary is built with go1.22.5. Perhaps try that version as well as the latest release (go1.23.1)? You can install any version of Go and then just switch with GOTOOLCHAIN :)

$ GOTOOLCHAIN=go1.22.5 go install github.com/restic/restic/cmd/restic@v0.17.0
go: downloading go1.22.5 (linux/amd64)

@distancesprinter
Copy link

Someone on https://smartos.topicbox.com/groups/smartos-discuss/Tf8f450fd133908fb mentioned that they would get similar crashes just running go version, not just their larger, more complex programs. Can either of you reproduce that?

@prattmic Yes. I installed go1.23.1.windows-amd64.msi and ran go version. Output below:

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\Windows\system32> go version
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x1628213?, 0xc00002be1e?})
        runtime/panic.go:1067 +0x4d fp=0xc000077c30 sp=0xc000077c00 pc=0xf70a4d
runtime.badmcall(0xc000077c78?)
        runtime/proc.go:539 +0x1f fp=0xc000077c50 sp=0xc000077c30 pc=0xf3e01f
runtime.badmcall(0xf3c1b9)
        <autogenerated>:1 +0x25 fp=0xc000077c68 sp=0xc000077c50 pc=0xf7dda5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc000077c80
stack: frame={sp:0xc000077c50, fp:0xc000077c68} stack=[0x544d204000,0x544d3ffb20)


goroutine 1 gp=0xc00004a000 m=nil [runnable, locked to thread]:
cmd/go/internal/modcmd.init.6()
        cmd/go/internal/modcmd/verify.go:41 +0x66 fp=0xc000077e20 sp=0xc000077e18 pc=0x14c01e6
runtime.doInit1(0x1aa8840)
        runtime/proc.go:7290 +0xed fp=0xc000077f50 sp=0xc000077e20 pc=0xf4b84d
runtime.doInit(...)
        runtime/proc.go:7257
runtime.main()
        runtime/proc.go:254 +0x325 fp=0xc000077fe0 sp=0xc000077f50 pc=0xf3d6c5
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0xf79061

goroutine 2 gp=0xc00004a700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00004dfa8 sp=0xc00004df88 pc=0xf70b6e
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.forcegchelper()
        runtime/proc.go:337 +0xb8 fp=0xc00004dfe0 sp=0xc00004dfa8 pc=0xf3d938
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00004dfe8 sp=0xc00004dfe0 pc=0xf79061
created by runtime.init.7 in goroutine 1
        runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc00004aa80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00004ff80 sp=0xc00004ff60 pc=0xf70b6e
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.bgsweep(0xc00005a000)
        runtime/mgcsweep.go:277 +0x94 fp=0xc00004ffc8 sp=0xc00004ff80 pc=0xf26934
runtime.gcenable.gowrap1()
        runtime/mgc.go:203 +0x25 fp=0xc00004ffe0 sp=0xc00004ffc8 pc=0xf1b005
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00004ffe8 sp=0xc00004ffe0 pc=0xf79061
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:203 +0x66

goroutine 18 gp=0xc0000861c0 m=nil [GC scavenge wait]:
runtime.gopark(0xc00005a000?, 0x172b0c8?, 0x1?, 0x0?, 0xc0000861c0?)
        runtime/proc.go:424 +0xce fp=0xc00008ff78 sp=0xc00008ff58 pc=0xf70b6e
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.(*scavengerState).park(0x1b01400)
        runtime/mgcscavenge.go:425 +0x49 fp=0xc00008ffa8 sp=0xc00008ff78 pc=0xf24369
runtime.bgscavenge(0xc00005a000)
        runtime/mgcscavenge.go:653 +0x3c fp=0xc00008ffc8 sp=0xc00008ffa8 pc=0xf248dc
runtime.gcenable.gowrap2()
        runtime/mgc.go:204 +0x25 fp=0xc00008ffe0 sp=0xc00008ffc8 pc=0xf1afa5
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008ffe8 sp=0xc00008ffe0 pc=0xf79061
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:204 +0xa5

goroutine 19 gp=0xc000086380 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000091e20 sp=0xc000091e00 pc=0xf70b6e
runtime.runfinq()
        runtime/mfinal.go:193 +0x107 fp=0xc000091fe0 sp=0xc000091e20 pc=0xf1a0c7
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000091fe8 sp=0xc000091fe0 pc=0xf79061
created by runtime.createfing in goroutine 1
        runtime/mfinal.go:163 +0x3d

goroutine 4 gp=0xc00004ae00 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008bf18 sp=0xc00008bef8 pc=0xf70b6e
runtime.chanrecv(0xc0000645b0, 0x0, 0x1)
        runtime/chan.go:639 +0x41e fp=0xc00008bf90 sp=0xc00008bf18 pc=0xf0a11e
runtime.chanrecv1(0x0?, 0x0?)
        runtime/chan.go:489 +0x12 fp=0xc00008bfb8 sp=0xc00008bf90 pc=0xf09cd2
runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
        runtime/mgc.go:1732
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
        runtime/mgc.go:1735 +0x2f fp=0xc00008bfe0 sp=0xc00008bfb8 pc=0xf1e0af
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008bfe8 sp=0xc00008bfe0 pc=0xf79061
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
        runtime/mgc.go:1730 +0x96
PS C:\Windows\system32>

I can try rebuilding restic, but since I'm seeing this with a simple go version I suspect that's not going to tell us much more. Please confirm if you feel it is worthwhile.

@prattmic
Copy link
Member

I can try rebuilding restic, but since I'm seeing this with a simple go version I suspect that's not going to tell us much more. Please confirm if you feel it is worthwhile.

Agreed. If this reproduces with go version, no need to test restic.

@distancesprinter
Copy link

Here's another:

PS C:\Windows\system32> go version
go version go1.23.1 windows/amd64
PS C:\Windows\system32> go version
fatal: morestack on g0
Exception 0x80000003 0x0 0x0 0x1358ec1
PC=0x1358ec1

runtime.abort()
        runtime/asm_amd64.s:1162 +0x1 fp=0xc000153c40 sp=0xc000153c38 pc=0x1358ec1
runtime.morestack()
        runtime/asm_amd64.s:598 +0x3d fp=0xc000153c48 sp=0xc000153c40 pc=0x135713d

goroutine 1 gp=0xc00004a000 m=nil [runnable, locked to thread]:
cmd/go/internal/test.init()
        <autogenerated>:1 +0x1b fp=0xc000153e20 sp=0xc000153e18 pc=0x18b5e3b
runtime.doInit1(0x1e86140)
        runtime/proc.go:7290 +0xed fp=0xc000153f50 sp=0xc000153e20 pc=0x132b84d
runtime.doInit(...)
        runtime/proc.go:7257
runtime.main()
        runtime/proc.go:254 +0x325 fp=0xc000153fe0 sp=0xc000153f50 pc=0x131d6c5
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000153fe8 sp=0xc000153fe0 pc=0x1359061

goroutine 2 gp=0xc00004a700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00004dfa8 sp=0xc00004df88 pc=0x1350b6e
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.forcegchelper()
        runtime/proc.go:337 +0xb8 fp=0xc00004dfe0 sp=0xc00004dfa8 pc=0x131d938
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00004dfe8 sp=0xc00004dfe0 pc=0x1359061
created by runtime.init.7 in goroutine 1
        runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc00004aa80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00004ff80 sp=0xc00004ff60 pc=0x1350b6e
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.bgsweep(0xc00005a000)
        runtime/mgcsweep.go:277 +0x94 fp=0xc00004ffc8 sp=0xc00004ff80 pc=0x1306934
runtime.gcenable.gowrap1()
        runtime/mgc.go:203 +0x25 fp=0xc00004ffe0 sp=0xc00004ffc8 pc=0x12fb005
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00004ffe8 sp=0xc00004ffe0 pc=0x1359061
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc00004ac40 m=nil [GC scavenge wait]:
runtime.gopark(0xc00005a000?, 0x1b0b0c8?, 0x1?, 0x0?, 0xc00004ac40?)
        runtime/proc.go:424 +0xce fp=0xc000061f78 sp=0xc000061f58 pc=0x1350b6e
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.(*scavengerState).park(0x1ee1400)
        runtime/mgcscavenge.go:425 +0x49 fp=0xc000061fa8 sp=0xc000061f78 pc=0x1304369
runtime.bgscavenge(0xc00005a000)
        runtime/mgcscavenge.go:653 +0x3c fp=0xc000061fc8 sp=0xc000061fa8 pc=0x13048dc
runtime.gcenable.gowrap2()
        runtime/mgc.go:204 +0x25 fp=0xc000061fe0 sp=0xc000061fc8 pc=0x12fafa5
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0x1359061
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc00004afc0 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000063e20 sp=0xc000063e00 pc=0x1350b6e
runtime.runfinq()
        runtime/mfinal.go:193 +0x107 fp=0xc000063fe0 sp=0xc000063e20 pc=0x12fa0c7
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x1359061
created by runtime.createfing in goroutine 1
        runtime/mfinal.go:163 +0x3d

goroutine 6 gp=0xc00004b180 m=nil [chan receive]:
runtime.gopark(0x0?, 0x12faf01?, 0x28?, 0x61?, 0x1e86128?)
        runtime/proc.go:424 +0xce fp=0xc000051f18 sp=0xc000051ef8 pc=0x1350b6e
runtime.chanrecv(0xc00006a690, 0x0, 0x1)
        runtime/chan.go:639 +0x41e fp=0xc000051f90 sp=0xc000051f18 pc=0x12ea11e
runtime.chanrecv1(0x131d780?, 0xc000051f76?)
        runtime/chan.go:489 +0x12 fp=0xc000051fb8 sp=0xc000051f90 pc=0x12e9cd2
runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
        runtime/mgc.go:1732
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
        runtime/mgc.go:1735 +0x2f fp=0xc000051fe0 sp=0xc000051fb8 pc=0x12fe0af
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000051fe8 sp=0xc000051fe0 pc=0x1359061
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
        runtime/mgc.go:1730 +0x96
rax     0x17
rbx     0x1ee3a80
rcx     0x1ee37a0
rdx     0x0
rdi     0xda1de72000
rsi     0x1ee37a0
rbp     0xc000153c58
rsp     0xc000153c38
r8      0xc0001538e8
r9      0xc000153ac0
r10     0x0
r11     0x246
r12     0xda1e1ffa80
r13     0xc
r14     0x1ee1a80
r15     0x3ffffff
rip     0x1358ec1
rflags  0x206
cs      0x33
fs      0x53
gs      0x2b
PS C:\Windows\system32> go version
go version go1.23.1 windows/amd64
PS C:\Windows\system32>

@prattmic
Copy link
Member

prattmic commented Sep 10, 2024

The common theme behind these crashes is they are upset about values ultimately loaded from the G, which is ultimately stored in TLS.

I could imagine this being an OS or VMM bug corrupting the TLS, but there are some other possibilities we can investigate:

  1. Asynchronous preemption ends up loading the G from TLS. Could you try disabling async preemption to see if this makes the problem go away, or become like frequent? GODEBUG=asyncpreemptoff=1 go version.

  2. We changed the way we manage the TLS on Windows in Go 1.20 (https://go.dev/cl/431775) and Go 1.21 (https://go.dev/cl/486816). Could you test Go 1.19, 1.20, and 1.21 to see which, if any, of these fail?

cc @qmuntal since I mention your CLs and you probably know more about Windows TLS anyway. :)

@qmuntal
Copy link
Member

qmuntal commented Sep 10, 2024

cc @qmuntal since I mention your CLs and you probably know more about Windows TLS anyway.

Before CL 431775 loading the TLS was only one instruction. That CL added a second instruction to resolve a newly added indirection:

MOVQ TLS, BX

// is tranformed into

MOVQ runtime.tls_g(SB), BX
MOVQ 0(BX)(GS*1), BX

It is probably unsafe to add a preemption point in between those instructions. If that's the case, both instructions should be marked as unsafe somehow. I'm not sure if that's already done, @prattmic could you check? I don't fully understand how is async preemption implemented 😅

@prattmic
Copy link
Member

@cherrymui
Copy link
Member

Good point, @qmuntal . https://cs.opensource.google/go/go/+/master:src/cmd/internal/obj/x86/asm6.go;l=2249-2262 marks two-instruction TLS access nonpreemptible, and CanUse1InsnTLS is false on Windows so this code does run. However, it seems after the rewrite it doesn't use the TLS pseudo-register, so useTLS doesn't match it?

@cherrymui
Copy link
Member

cherrymui commented Sep 10, 2024

Before register ABI, we load G from TLS in almost all function prologues. With register ABI, usually we load G from the register (R14), so TLS accesses are mostly from assembly functions. But there are still a small amount of ABI0 Go functions, which will load G from TLS. Maybe they are all in the runtime package therefore not async preemptible anyway?

And at call sites in normal Go functions to ABI0 functions, we generate code sequence to reload G from TLS to R14. These could be preemptible, if the marking didn't work as expected.

@cherrymui
Copy link
Member

cherrymui commented Sep 10, 2024

Yeah, for a call from a normal Go function to an ABI0 function, we generate a load from TLS https://cs.opensource.google/go/go/+/master:src/cmd/compile/internal/amd64/ssa.go;l=1080 . After the expansion in CL 431775, and the unsafe point marking, the code looks like

	0x000a 00010 (/tmp/pp/x.go:6)	CALL	main.f(SB)
	0x000f 00015 (/tmp/pp/x.go:6)	XORPS	X15, X15
	0x0013 00019 (/tmp/pp/x.go:6)	MOVQ	runtime.tls_g(SB), R14
	0x001a 00026 (/tmp/pp/x.go:6)	MOVQ	(R14)(GS*1), R14
	0x001e 00030 (/tmp/pp/x.go:6)	PCDATA	$0, $-2
	0x001e 00030 (/tmp/pp/x.go:6)	MOVQ	(R14)(TLS*2), R14
	0x0021 00033 (/tmp/pp/x.go:6)	PCDATA	$0, $-1

The "second", actually third after expansion, instruction, MOVQ (R14)(TLS*2), R14 is marked nonpreemptible (PCDATA -2). But not the MOVQ (R14)(GS*1), R14. If a preemption happens at this instruction, and a thread switch occurs, and GS is thread dependent, it will go wrong...

@distancesprinter
Copy link

  1. Asynchronous preemption ends up loading the G from TLS. Could you try disabling async preemption to see if this makes the problem go away, or become like frequent? GODEBUG=asyncpreemptoff=1 go version.

Can't seem to trigger the error with go1.23.1 with GODEBUG=asyncpreemptoff=1:

Microsoft Windows [Version 10.0.17763.6189]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\Windows\system32>set GODEBUG=asyncpreemptoff=1

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>set
ALLUSERSPROFILE=C:\ProgramData
APPDATA=C:\Users\[redacted]\AppData\Roaming
CommonProgramFiles=C:\Program Files\Common Files
CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files
CommonProgramW6432=C:\Program Files\Common Files
COMPUTERNAME=[redacted]
ComSpec=C:\Windows\system32\cmd.exe
DriverData=C:\Windows\System32\Drivers\DriverData
GODEBUG=asyncpreemptoff=1
GOPATH=C:\Users\[redacted]\go
HOMEDRIVE=C:
[redacted]
NUMBER_OF_PROCESSORS=4
OS=Windows_NT
Path=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Go\bin;[redacted]
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE=AMD64
PROCESSOR_IDENTIFIER=AMD64 Family 25 Model 1 Stepping 1, AuthenticAMD
PROCESSOR_LEVEL=25
PROCESSOR_REVISION=0101
ProgramData=C:\ProgramData
ProgramFiles=C:\Program Files
ProgramFiles(x86)=C:\Program Files (x86)
ProgramW6432=C:\Program Files
PROMPT=$P$G
PSModulePath=C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
PUBLIC=C:\Users\Public
SystemDrive=C:
SystemRoot=C:\Windows
[redacted]
windir=C:\Windows

C:\Windows\system32>
  1. We changed the way we manage the TLS on Windows in Go 1.20 (https://go.dev/cl/431775) and Go 1.21 (https://go.dev/cl/486816). Could you test Go 1.19, 1.20, and 1.21 to see which, if any, of these fail?

Also have not been able to trigger the error with go1.19.13:

PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32>

@prattmic
Copy link
Member

Thanks all, sounds like we found the bug (or at least a bug!). @qmuntal would you like to send a CL.

I remain confused why this fails so readily on these machines but very rarely otherwise. My best guess would be that the load from GS is particularly slow for some reason, making signals more likely to land there, but that is a bit of a stretch. Inside the Bhyve VM, perhaps the load from GS is causing a VM exit, which would be very slow? But that doesn't explain the bare metal cases.

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/612535 mentions this issue: cmd/internal/obj/x86: mark Windows TLS access sequence nonpreemptible

@cherrymui
Copy link
Member

cherrymui commented Sep 11, 2024

Could you try if CL 612535 makes a difference? Thanks.

@prattmic
Copy link
Member

@distancesprinter The easiest way to test a change like @cherrymui's is probably to use gotip:

# Install gotip
$ go install golang.org/dl/gotip@latest
# Installs to $HOME/go/bin
# Download and build a toolchain with CL 612535
$ gotip download 612535
# Now use gotip the same as the go command. e.g., gotip version, gotip build, etc.

Except this gets weird since your current version of Go crashes. So I think you'll need GODEBUG=asyncpreemptoff=1 for go install and gotip download. Once that is done gotip version would normally find the go toolchain it built and run go version, but since gotip is built with the bad Go, it might crash before running the "good" Go...

I think gotip installs to $HOME/sdk/gotip, so perhaps find the go binary there and run that go version.

(Thanks for all the help testing despite not being a Go developer!)

@distancesprinter
Copy link

@prattmic @cherrymui So far, so good.

Microsoft Windows [Version 10.0.17763.6293]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\Users\[username]>set GODEBUG=asyncpreemptoff=1

C:\Users\[username]>go install golang.org/dl/gotip@latest
go: downloading golang.org/dl v0.0.0-20240905150950-a427666708fc

C:\Users\[username]>gotip download 612535
gotip: failed to clone git repository: exec: "git": executable file not found in %PATH%

C:\Users\[username]>

Installed Git for Windows, then retried gotip download:

Microsoft Windows [Version 10.0.17763.6293]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\Users\[username]>set GODEBUG=asyncpreemptoff=1

C:\Users\[username]>gotip download 612535
Cloning into 'C:\Users\[username]\sdk\gotip'...
remote: Counting objects: 14605, done
remote: Finding sources: 100% (14605/14605)
remote: Total 14605 (delta 1912), reused 9534 (delta 1912)
Receiving objects: 100% (14605/14605), 29.69 MiB | 10.62 MiB/s, done.
Resolving deltas: 100% (1912/1912), done.
Updating files: 100% (13414/13414), done.
This will download and execute code from golang.org/cl/612535, continue? [y/n] y
Fetching CL 612535, Patch Set 1...
remote: Counting objects: 597575, done
remote: Finding sources: 100% (597575/597575)
remote: Total 597575 (delta 422213), reused 589375 (delta 422213)
Receiving objects: 100% (597575/597575), 564.51 MiB | 17.78 MiB/s, done.
Resolving deltas: 100% (422213/422213), done.
From https://go.googlesource.com/go
 * branch                  refs/changes/35/612535/1 -> FETCH_HEAD
HEAD is now at 68306ba416 cmd/internal/obj/x86: mark Windows TLS access sequence nonpreemptible
Building Go cmd/dist using C:\Program Files\Go. (go1.23.1 windows/amd64)
Building Go toolchain1 using C:\Program Files\Go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for windows/amd64.
---
Installed Go for windows/amd64 in C:\Users\[username]\sdk\gotip
Installed commands in C:\Users\[username]\sdk\gotip\bin
Success. You may now run 'gotip'!

C:\Users\[username]>gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64

Rebooted and tried in PowerShell and Command Prompt:

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\Windows\system32> gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64
PS C:\Windows\system32> gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64
PS C:\Windows\system32> gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64
PS C:\Windows\system32> gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64
PS C:\Windows\system32> gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64
PS C:\Windows\system32> gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64
PS C:\Windows\system32> gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64
PS C:\Windows\system32> gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64
PS C:\Windows\system32> gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64
PS C:\Windows\system32> gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64
PS C:\Windows\system32> gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64
PS C:\Windows\system32> gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64
PS C:\Windows\system32> gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64
PS C:\Windows\system32> gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64
PS C:\Windows\system32> gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64
PS C:\Windows\system32> gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64
PS C:\Windows\system32> gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64
PS C:\Windows\system32> gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64
PS C:\Windows\system32>

Microsoft Windows [Version 10.0.17763.6293]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\Windows\system32>gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64

C:\Windows\system32>gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64

C:\Windows\system32>gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64

C:\Windows\system32>gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64

C:\Windows\system32>gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64

C:\Windows\system32>gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64

C:\Windows\system32>gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64

C:\Windows\system32>gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64

C:\Windows\system32>gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64

C:\Windows\system32>gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64

C:\Windows\system32>gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64

C:\Windows\system32>gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64

C:\Windows\system32>gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64

C:\Windows\system32>

@prattmic
Copy link
Member

@distancesprinter Great, thanks! I suppose for good measure you should double-check that tip is affected without the patch. gotip download (without a CL number) should build a tip toolchain without Cherry's patch.

@distancesprinter
Copy link

@prattmic @cherrymui I have some new information to report. Based on the discussion about the TLS changes introduced in Go 1.20, and because I didn't see the crash after rebooting and running go version with Go 1.19.13, I got the wise idea of trying to compile restic 0.17.1 with Go 1.19.13 in Docker and give that a whirl (what could go wrong? /s).

Turns out everything worked fine last night, but when I checked this morning, several crashes (output below).

I will try again this evening to do a gotip version before and after a reboot to see if I can trigger the crash. I will then do a gotip download and see if I observe the crash with a gotip version on tip toolchain without Cherry's patch.

However, since I see the problem this morning, running restic with Go 1.19.13, I am concerned, as my understanding was that the bug we're dealing with here wasn't introduced until Go 1.20. Also, it remains suspicious that I'm only seeing this on my AMD boxes and there doesn't seem to be a good explanation for why that would be the case.

@prattmic Is it possible to/can you provide instructions to build restic with Cherry's patch? That would give me a chance to monitor a lot more executions with a scheduled backup job than just running go version a few times after a reboot (which until this point was a reliable way for me to trigger the crash). Cherry's patch likely fixes a bona fide bug, but it's possible that something else is causing the problem I am experiencing. It is also possible that some other bug fixed between Go 1.19.13 and dev go1.24-68306ba416 changes how this issue manifests itself.

Here's the output (the .cmd file simply invokes the restic binary with all the flags necessary to specify the proper repository location, options, and secret) and this is running restic 0.17.1 compiled with go1.19.13 on windows/amd64:

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\Windows\system32> cd C:\restic\
PS C:\restic> .\backup_prod.cmd
open repository
repository 5304de1b opened (version 2, compression level auto)
using parent snapshot 380401e3
load index files
[0:00] 100.00%  36 / 36 index files loaded
start scan on [K:\]
start backup on [K:\]
creating VSS snapshot for [k:\]
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x202c3df?, 0xc002deaeb0?})
        /usr/local/go/src/runtime/panic.go:1047 +0x65 fp=0xc002deae50 sp=0xc002deae20 pc=0x1168c65
runtime.badmcall(0xc000205380?)
        /usr/local/go/src/runtime/proc.go:468 +0x25 fp=0xc002deae70 sp=0xc002deae50 pc=0x116bee5
runtime.badmcall(0x11732c0)
        <autogenerated>:1 +0x2c fp=0xc002deae88 sp=0xc002deae70 pc=0x11993cc
runtime.asyncPreempt2()
        /usr/local/go/src/runtime/preempt.go:308 +0x3f fp=0xc002deaea8 sp=0xc002deae88 pc=0x1169e9f
runtime.asyncPreempt()
        /usr/local/go/src/runtime/preempt_amd64.s:50 +0xdb fp=0xc002deb030 sp=0xc002deaea8 pc=0x119871b
github.com/restic/restic/internal/fs.(*IVssBackupComponents).SetBackupState(0x28f29849750, 0x0, 0x0, 0x5, 0x0)
        /restic/internal/fs/vss_windows.go:459 fp=0xc002deb038 sp=0xc002deb030 pc=0x1503520
github.com/restic/restic/internal/fs.NewVssSnapshot({0x0, 0x0}, {0xc0017ee5f2, 0x3}, 0x11f3567?, 0xc002deb6a8, 0xc000284740)
        /restic/internal/fs/vss_windows.go:924 +0x1e5 fp=0xc002deb4a8 sp=0xc002deb038 pc=0x15047a5
github.com/restic/restic/internal/fs.(*LocalVss).snapshotPath(0xc000112000, {0xc0017ee560, 0x3})
        /restic/internal/fs/fs_local_vss.go:213 +0x545 fp=0xc002debae0 sp=0xc002deb4a8 pc=0x14fd9a5
github.com/restic/restic/internal/fs.(*LocalVss).Lstat(0xc0017ee560?, {0xc0017ee560?, 0xc0017ee560?})
        /restic/internal/fs/fs_local_vss.go:145 +0x1e fp=0xc002debb08 sp=0xc002debae0 pc=0x14fd25e
github.com/restic/restic/internal/archiver.(*Scanner).scan(0xc0001a26f0, {0x2218b88, 0xc00028ad00}, {0x1dcb340?, 0xc0000e7b01?, 0xc000241ce0?, 0x159006c?}, {0xc0017ee560, 0x3})
        /restic/internal/archiver/scanner.go:112 +0xc2 fp=0xc002debbc8 sp=0xc002debb08 pc=0x158e362
github.com/restic/restic/internal/archiver.(*Scanner).scanTree(0xc0001a26f0, {0x2218b88, 0xc00028ad00}, {0xc00028adc0?, 0x1?, 0x683cee9b00000000?, 0xc000241d78?}, {0x0, {0xc00003c790, 0x3}, ...})
        /restic/internal/archiver/scanner.go:49 +0x118 fp=0xc002debcf0 sp=0xc002debbc8 pc=0x158dbb8
github.com/restic/restic/internal/archiver.(*Scanner).scanTree(0xc0001a26f0, {0x2218b88, 0xc00028ad00}, {0xc000241ea8?, 0x1136a25?, 0x1b0dc6fcddd81d4?, 0x121fc5a4?}, {0xc0001a2780, {0x0, 0x0}, ...})
        /restic/internal/archiver/scanner.go:60 +0x331 fp=0xc002debe18 sp=0xc002debcf0 pc=0x158ddd1
github.com/restic/restic/internal/archiver.(*Scanner).Scan(0xc0001a26f0, {0x2218b88, 0xc00028ad00}, {0xc00052a8d0, 0x1, 0x1})
        /restic/internal/archiver/scanner.go:91 +0x1f5 fp=0xc002debf38 sp=0xc002debe18 pc=0x158e0f5
main.runBackup.func5()
        /restic/cmd/restic/cmd_backup.go:615 +0x31 fp=0xc002debf78 sp=0xc002debf38 pc=0x1c940d1
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc002debfe0 sp=0xc002debf78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc002debfe8 sp=0xc002debfe0 pc=0x11970a1

goroutine 69 [syscall, locked to thread]:
runtime: g 69: unexpected return pc for runtime.cgocall called from 0x0
stack: frame={sp:0xc002deaec8, fp:0xc002deaf00} stack=[0xc002de8000,0xc002dec000)
0x000000c002deadc8:  0x000000c000299d40  0x000000c002deae10
0x000000c002deadd8:  0x0000000001168fac <runtime.fatalthrow+0x000000000000006c>  0x000000c002deadf0
0x000000c002deade8:  0x000000c000299d40  0x0000000001168fe0 <runtime.fatalthrow.func1+0x0000000000000000>
0x000000c002deadf8:  0x000000c000299d40  0x0000000001168c65 <runtime.throw+0x0000000000000065>
0x000000c002deae08:  0x000000c002deae20  0x000000c002deae40
0x000000c002deae18:  0x0000000001168c65 <runtime.throw+0x0000000000000065>  0x000000c002deae28
0x000000c002deae28:  0x0000000001168c80 <runtime.throw.func1+0x0000000000000000>  0x000000000202c3df
0x000000c002deae38:  0x0000000000000024  0x000000c002deae60
0x000000c002deae48:  0x000000000116bee5 <runtime.badmcall+0x0000000000000025>  0x000000000202c3df
0x000000c002deae58:  0x000000c002deaeb0  0x000000c002deae78
0x000000c002deae68:  0x00000000011993cc <runtime.badmcall+0x000000000000002c>  0x000000c000205380
0x000000c002deae78:  0x000000c002deae98  0x0000000001169e9f <runtime.asyncPreempt2+0x000000000000003f>
0x000000c002deae88:  0x00000000011732c0 <runtime.exitsyscallfast.func1+0x0000000000000000>  0x000000c000299d40
0x000000c002deae98:  0x000000c002deb020  0x000000000119871b <runtime.asyncPreempt+0x00000000000000db>
0x000000c002deaea8:  0x0000000040000024  0x00007fff9faf0894
0x000000c002deaeb8:  0x0000000000000000  0x000000a9d75ffa30
0x000000c002deaec8: <0x0000000000000000  0x0000000001199360 <runtime.switchtothread+0x0000000000000000>
0x000000c002deaed8:  0x000000a9d75ff9b8  0x000000c002deaef0
0x000000c002deaee8:  0x0000000000000000  0x0000000000000246
0x000000c002deaef8: !0x0000000000000000 >0x0000000000000000
0x000000c002deaf08:  0x000000c000299d40  0xffffffffffffffff
0x000000c002deaf18:  0x0000000000000000  0x0000000000000000
0x000000c002deaf28:  0x0000000000000000  0x0000000000000000
0x000000c002deaf38:  0x0000000000000000  0x0000000000000000
0x000000c002deaf48:  0x0000000000000000  0x0000000000000000
0x000000c002deaf58:  0x0000000000000000  0x0000000000000000
0x000000c002deaf68:  0x0000000000000000  0x0000000000000000
0x000000c002deaf78:  0x000000c0000ce380  0x00000000014d00c0 <golang.org/x/sync/errgroup.(*Group).Go.func1.2+0x0000000000000000>
0x000000c002deaf88:  0x000000c0000ce380  0x0000000000000483
0x000000c002deaf98:  0x00008e550080ab6e  0x00000000000167f6
0x000000c002deafa8:  0x465516f8758a5b25  0x000000c0005a9f98
0x000000c002deafb8:  0x0000000000000000  0x00000000011970a1 <runtime.goexit+0x0000000000000001>
0x000000c002deafc8:  0x0000000000000000  0x0000000000000483
0x000000c002deafd8:  0x00081185008139c3  0x00000000000b21fb
0x000000c002deafe8:  0x000000c0001da6e0  0x000000c0002086f0
0x000000c002deaff8:  0x000000c0001a30e0
runtime.cgocall(0x0, 0xc000299d40)
        /usr/local/go/src/runtime/cgocall.go:158 +0x4a fp=0xc002deaf00 sp=0xc002deaec8 pc=0x11348ea
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 1 [semacquire]:
runtime.gopark(0x28f295a6d02?, 0x28f295a6d02?, 0xc0?, 0x26?, 0x113db7f?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc002deecd0 sp=0xc002deecb0 pc=0x116b736
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.semacquire1(0xc0000ce290, 0xd8?, 0x1, 0x0)
        /usr/local/go/src/runtime/sema.go:150 +0x20f fp=0xc002deed38 sp=0xc002deecd0 pc=0x117abcf
sync.runtime_Semacquire(0xc0005f20d8?)
        /usr/local/go/src/runtime/sema.go:62 +0x25 fp=0xc002deed68 sp=0xc002deed38 pc=0x1192205
sync.(*WaitGroup).Wait(0x1f6e5e0?)
        /usr/local/go/src/sync/waitgroup.go:139 +0x52 fp=0xc002deed90 sp=0xc002deed68 pc=0x11a5252
golang.org/x/sync/errgroup.(*Group).Wait(0xc0000ce280)
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:56 +0x27 fp=0xc002deedb0 sp=0xc002deed90 pc=0x14cfe07
github.com/restic/restic/internal/archiver.(*Archiver).Snapshot(0xc0002bc140, {0x2218b88?, 0xc000491180}, {0xc00052a8d0, 0x1, 0x1}, {{0x2a66db0, 0x0, 0x0}, {0xc00003c740, ...}, ...})
        /restic/internal/archiver/archiver.go:865 +0x3b8 fp=0xc002def010 sp=0xc002deedb0 pc=0x158a498
main.runBackup({_, _}, {{{0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, {0xc00052a850, 0x1, ...}, ...}, ...}, ...)
        /restic/cmd/restic/cmd_backup.go:660 +0x1a48 fp=0xc002def888 sp=0xc002def010 pc=0x1c93c28
main.glob..func2(0x29f01a0, {0xc00047fb80, 0x0, 0xb})
        /restic/cmd/restic/cmd_backup.go:63 +0xf8 fp=0xc002defbb8 sp=0xc002def888 pc=0x1c8f6b8
github.com/spf13/cobra.(*Command).execute(0x29f01a0, {0xc00047fad0, 0xb, 0xb})
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985 +0xab4 fp=0xc002defd58 sp=0xc002defbb8 pc=0x14c1914
github.com/spf13/cobra.(*Command).ExecuteC(0x29efbe0)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x425 fp=0xc002defe30 sp=0xc002defd58 pc=0x14c2245
github.com/spf13/cobra.(*Command).Execute(...)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1034
main.main()
        /restic/cmd/restic/main.go:142 +0x258 fp=0xc002deff80 sp=0xc002defe30 pc=0x1cceab8
runtime.main()
        /usr/local/go/src/runtime/proc.go:250 +0x1fe fp=0xc002deffe0 sp=0xc002deff80 pc=0x116b39e
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc002deffe8 sp=0xc002deffe0 pc=0x11970a1

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00005ffb0 sp=0xc00005ff90 pc=0x116b736
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:302 +0xb1 fp=0xc00005ffe0 sp=0xc00005ffb0 pc=0x116b5d1
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00005ffe8 sp=0xc00005ffe0 pc=0x11970a1
created by runtime.init.6
        /usr/local/go/src/runtime/proc.go:290 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000061f90 sp=0xc000061f70 pc=0x116b736
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
        /usr/local/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc000061fc8 sp=0xc000061f90 pc=0x11557b7
runtime.gcenable.func1()
        /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc000061fe0 sp=0xc000061fc8 pc=0x114a266
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0x11970a1
created by runtime.gcenable
        /usr/local/go/src/runtime/mgc.go:178 +0x6b

goroutine 4 [sleep]:
runtime.gopark(0xc000078000?, 0x3237a42af3b0?, 0x0?, 0x0?, 0x209d090?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000071f30 sp=0xc000071f10 pc=0x116b736
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.(*scavengerState).sleep(0x2a0f740, 0x4131170000000000)
        /usr/local/go/src/runtime/mgcscavenge.go:468 +0x12a fp=0xc000071fa0 sp=0xc000071f30 pc=0x1153a0a
runtime.bgscavenge(0x0?)
        /usr/local/go/src/runtime/mgcscavenge.go:626 +0x7f fp=0xc000071fc8 sp=0xc000071fa0 pc=0x1153e1f
runtime.gcenable.func2()
        /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000071fe0 sp=0xc000071fc8 pc=0x114a206
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000071fe8 sp=0xc000071fe0 pc=0x11970a1
created by runtime.gcenable
        /usr/local/go/src/runtime/mgc.go:179 +0xaa

goroutine 18 [finalizer wait]:
runtime.gopark(0x0?, 0xc000063e70?, 0x8b?, 0x76?, 0xc000063f70?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000063e28 sp=0xc000063e08 pc=0x116b736
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:180 +0x10f fp=0xc000063fe0 sp=0xc000063e28 pc=0x114936f
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x11970a1
created by runtime.createfing
        /usr/local/go/src/runtime/mfinal.go:157 +0x45

goroutine 19 [chan receive]:
runtime.gopark(0xc00006de90?, 0xc00006dee8?, 0x51?, 0x61?, 0xc00006dee8?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00006deb8 sp=0xc00006de98 pc=0x116b736
runtime.chanrecv(0xc000076060, 0xc00006dfa0, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc00006df48 sp=0xc00006deb8 pc=0x1136fbb
runtime.chanrecv2(0xc000076060?, 0xc00006dfc0?)
        /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc00006df70 sp=0xc00006df48 pc=0x1136af8
github.com/restic/restic/internal/restic.init.0.func1.1()
        /restic/internal/restic/lock.go:429 +0x9d fp=0xc00006dfe0 sp=0xc00006df70 pc=0x156c25d
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0x11970a1
created by github.com/restic/restic/internal/restic.init.0.func1
        /restic/internal/restic/lock.go:426 +0x25

goroutine 5 [syscall]:
runtime.notetsleepg(0x0?, 0x0?)
        /usr/local/go/src/runtime/lock_sema.go:294 +0x3c fp=0xc000073fa0 sp=0xc000073f58 pc=0x113c5dc
os/signal.signal_recv()
        /usr/local/go/src/runtime/sigqueue.go:152 +0x2f fp=0xc000073fc0 sp=0xc000073fa0 pc=0x119292f
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x19 fp=0xc000073fe0 sp=0xc000073fc0 pc=0x11f6579
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000073fe8 sp=0xc000073fe0 pc=0x11970a1
created by os/signal.Notify.func1.1
        /usr/local/go/src/os/signal/signal.go:151 +0x2a

goroutine 6 [GC worker (idle)]:
runtime.gopark(0x32376045ede4?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00054bf50 sp=0xc00054bf30 pc=0x116b736
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00054bfe0 sp=0xc00054bf50 pc=0x114c471
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00054bfe8 sp=0xc00054bfe0 pc=0x11970a1
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 50 [GC worker (idle)]:
runtime.gopark(0x323760ccb248?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000547f50 sp=0xc000547f30 pc=0x116b736
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000547fe0 sp=0xc000547f50 pc=0x114c471
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000547fe8 sp=0xc000547fe0 pc=0x11970a1
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 51 [GC worker (idle)]:
runtime.gopark(0x323760ccb248?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000549f50 sp=0xc000549f30 pc=0x116b736
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000549fe0 sp=0xc000549f50 pc=0x114c471
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000549fe8 sp=0xc000549fe0 pc=0x11970a1
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 21 [GC worker (idle)]:
runtime.gopark(0x323760ccb248?, 0x3?, 0x9c?, 0x33?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0001f7f50 sp=0xc0001f7f30 pc=0x116b736
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001f7fe0 sp=0xc0001f7f50 pc=0x114c471
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001f7fe8 sp=0xc0001f7fe0 pc=0x11970a1
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 52 [select]:
runtime.gopark(0xc0001f3f88?, 0x3?, 0x0?, 0x2a?, 0xc0001f3f72?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0001f3df8 sp=0xc0001f3dd8 pc=0x116b736
runtime.selectgo(0xc0001f3f88, 0xc0001f3f6c, 0xc0002c0000?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc0001f3f38 sp=0xc0001f3df8 pc=0x1179adc
go.opencensus.io/stats/view.(*worker).start(0xc0002c0000)
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0xad fp=0xc0001f3fc8 sp=0xc0001f3f38 pc=0x1b6abed
go.opencensus.io/stats/view.init.0.func1()
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x26 fp=0xc0001f3fe0 sp=0xc0001f3fc8 pc=0x1b69e26
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001f3fe8 sp=0xc0001f3fe0 pc=0x11970a1
created by go.opencensus.io/stats/view.init.0
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 22 [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0001f5e80 sp=0xc0001f5e60 pc=0x116b736
runtime.chanrecv(0xc000076900, 0xc0001f5f80, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc0001f5f10 sp=0xc0001f5e80 pc=0x1136fbb
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc0001f5f38 sp=0xc0001f5f10 pc=0x1136ab8
main.cleanupHandler(0x0?, 0xc00052a7a0)
        /restic/cmd/restic/cleanup.go:24 +0x30 fp=0xc0001f5fc0 sp=0xc0001f5f38 pc=0x1c8f310
main.createGlobalContext.func1()
        /restic/cmd/restic/cleanup.go:16 +0x2a fp=0xc0001f5fe0 sp=0xc0001f5fc0 pc=0x1c8f2aa
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001f5fe8 sp=0xc0001f5fe0 pc=0x11970a1
created by main.createGlobalContext
        /restic/cmd/restic/cleanup.go:16 +0xa5

goroutine 23 [select]:
runtime.gopark(0xc0001f9f10?, 0x3?, 0x20?, 0x0?, 0xc0001f9eaa?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0001f9d00 sp=0xc0001f9ce0 pc=0x116b736
runtime.selectgo(0xc0001f9f10, 0xc0001f9ea4, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc0001f9e40 sp=0xc0001f9d00 pc=0x1179adc
github.com/restic/restic/internal/ui/termstatus.(*Terminal).run(0xc000076a20, {0x2218b88, 0xc000074740})
        /restic/internal/ui/termstatus/status.go:105 +0x125 fp=0xc0001f9f50 sp=0xc0001f9e40 pc=0x174adc5
github.com/restic/restic/internal/ui/termstatus.(*Terminal).Run(0x0?, {0x2218b88?, 0xc000074740?})
        /restic/internal/ui/termstatus/status.go:94 +0x68 fp=0xc0001f9f98 sp=0xc0001f9f50 pc=0x174abc8
main.setupTermstatus.func1()
        /restic/cmd/restic/termstatus.go:28 +0x65 fp=0xc0001f9fe0 sp=0xc0001f9f98 pc=0x1cd0f45
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001f9fe8 sp=0xc0001f9fe0 pc=0x11970a1
created by main.setupTermstatus
        /restic/cmd/restic/termstatus.go:26 +0x138

goroutine 85 [select]:
runtime.gopark(0xc00054df08?, 0x3?, 0x30?, 0xdd?, 0xc00054de8a?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00054dcc8 sp=0xc00054dca8 pc=0x116b736
runtime.selectgo(0xc00054df08, 0xc00054de84, 0x0?, 0x0, 0x521697906c058c9e?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc00054de08 sp=0xc00054dcc8 pc=0x1179adc
github.com/restic/restic/internal/repository.(*locker).refreshLocks(0x2972ba0, {0x2218b88, 0xc000491180}, {0x221f6e8, 0xc0001a3ec0}, 0xc000096700, 0xc00020ac60, 0xc00020acc0, 0x209cb38)
        /restic/internal/repository/lock.go:144 +0x22a fp=0xc00054df88 sp=0xc00054de08 pc=0x17304aa
github.com/restic/restic/internal/repository.(*locker).Lock.func1()
        /restic/internal/repository/lock.go:105 +0x46 fp=0xc00054dfe0 sp=0xc00054df88 pc=0x1730246
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00054dfe8 sp=0xc00054dfe0 pc=0x11970a1
created by github.com/restic/restic/internal/repository.(*locker).Lock
        /restic/internal/repository/lock.go:105 +0x705

goroutine 8 [IO wait]:
runtime.gopark(0x0?, 0xc000004c98?, 0x48?, 0x4d?, 0xc000004cc8?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0004995d0 sp=0xc0004995b0 pc=0x116b736
runtime.netpollblock(0xc000002658?, 0x499688?, 0xc0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc000499608 sp=0xc0004995d0 pc=0x1161df7
internal/poll.runtime_pollWait(0x28f4f2d0f08, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc000499628 sp=0xc000499608 pc=0x1190c09
internal/poll.(*pollDesc).wait(0xc0000440b0?, 0x2c?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000499650 sp=0xc000499628 pc=0x11de392
internal/poll.execIO(0xc000004c98, 0x209ca48)
        /usr/local/go/src/internal/poll/fd_windows.go:175 +0xe5 fp=0xc0004996a8 sp=0xc000499650 pc=0x11df9c5
internal/poll.(*FD).Read(0xc000004c80, {0xc0003f9800, 0x1800, 0x1800})
        /usr/local/go/src/internal/poll/fd_windows.go:441 +0x26b fp=0xc000499738 sp=0xc0004996a8 pc=0x11e084b
net.(*netFD).Read(0xc000004c80, {0xc0003f9800?, 0xc00004c280?, 0xc0003f9805?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000499780 sp=0xc000499738 pc=0x131cc69
net.(*conn).Read(0xc00000a1c0, {0xc0003f9800?, 0x28f2954b328?, 0xa0?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0004997c8 sp=0xc000499780 pc=0x13329a5
crypto/tls.(*atLeastReader).Read(0xc000390330, {0xc0003f9800?, 0x0?, 0x8?})
        /usr/local/go/src/crypto/tls/conn.go:787 +0x3d fp=0xc000499810 sp=0xc0004997c8 pc=0x136dafd
bytes.(*Buffer).ReadFrom(0xc0000fe978, {0x220b300, 0xc000390330})
        /usr/local/go/src/bytes/buffer.go:202 +0x98 fp=0xc000499868 sp=0xc000499810 pc=0x1246118
crypto/tls.(*Conn).readFromUntil(0xc0000fe700, {0x28f4f2d0ff8?, 0xc00000a1c0}, 0x1800?)
        /usr/local/go/src/crypto/tls/conn.go:809 +0xe5 fp=0xc0004998a8 sp=0xc000499868 pc=0x136dce5
crypto/tls.(*Conn).readRecordOrCCS(0xc0000fe700, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:616 +0x116 fp=0xc000499c30 sp=0xc0004998a8 pc=0x136b136
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0xc0000fe700, {0xc000576000, 0x1000, 0x0?})
        /usr/local/go/src/crypto/tls/conn.go:1315 +0x16f fp=0xc000499ca0 sp=0xc000499c30 pc=0x13710af
bufio.(*Reader).Read(0xc000496de0, {0xc0003c8e40, 0x9, 0xc00022a688?})
        /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc000499cd8 sp=0xc000499ca0 pc=0x13a67bb
io.ReadAtLeast({0x220a1a0, 0xc000496de0}, {0xc0003c8e40, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:332 +0x9a fp=0xc000499d20 sp=0xc000499cd8 pc=0x11d71da
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc0003c8e40?, 0x9?, 0x0?}, {0x220a1a0?, 0xc000496de0?})
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/frame.go:237 +0x6e fp=0xc000499d70 sp=0xc000499d20 pc=0x15413ae
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0003c8e00)
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/frame.go:501 +0x95 fp=0xc000499e20 sp=0xc000499d70 pc=0x1541c15
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000499f98)
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:2358 +0xd8 fp=0xc000499f60 sp=0xc000499e20 pc=0x15554f8
golang.org/x/net/http2.(*ClientConn).readLoop(0xc00047c780)
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:2254 +0x99 fp=0xc000499fc8 sp=0xc000499f60 pc=0x1554a99
golang.org/x/net/http2.(*Transport).newClientConn.func1()
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:869 +0x26 fp=0xc000499fe0 sp=0xc000499fc8 pc=0x154d366
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000499fe8 sp=0xc000499fe0 pc=0x11970a1
created by golang.org/x/net/http2.(*Transport).newClientConn
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:869 +0xccd

goroutine 86 [select]:
runtime.gopark(0xc00049df40?, 0x4?, 0x0?, 0x0?, 0xc00049de80?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00049dce8 sp=0xc00049dcc8 pc=0x116b736
runtime.selectgo(0xc00049df40, 0xc00049de78, 0x66e2fa3f?, 0x0, 0x11a1c17?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc00049de28 sp=0xc00049dce8 pc=0x1179adc
github.com/restic/restic/internal/repository.(*locker).monitorLockRefresh(0x2972ba0, {0x2218b88, 0xc000491180}, 0xc000096700, 0xc00020ac60, 0xc00020acc0, 0x209cb38)
        /restic/internal/repository/lock.go:208 +0x23e fp=0xc00049df98 sp=0xc00049de28 pc=0x1730c7e
github.com/restic/restic/internal/repository.(*locker).Lock.func2()
        /restic/internal/repository/lock.go:106 +0x3e fp=0xc00049dfe0 sp=0xc00049df98 pc=0x17301be
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00049dfe8 sp=0xc00049dfe0 pc=0x11970a1
created by github.com/restic/restic/internal/repository.(*locker).Lock
        /restic/internal/repository/lock.go:106 +0x7ec

goroutine 87 [select]:
runtime.gopark(0xc0001ebf78?, 0x3?, 0x28?, 0x82?, 0xc0001ebf0a?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0001ebd90 sp=0xc0001ebd70 pc=0x116b736
runtime.selectgo(0xc0001ebf78, 0xc0001ebf04, 0x2a0f2e0?, 0x0, 0x4371323d?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc0001ebed0 sp=0xc0001ebd90 pc=0x1179adc
github.com/restic/restic/internal/ui/progress.(*Updater).run(0xc000491200)
        /restic/internal/ui/progress/updater.go:73 +0x186 fp=0xc0001ebfc8 sp=0xc0001ebed0 pc=0x1529686
github.com/restic/restic/internal/ui/progress.NewUpdater.func1()
        /restic/internal/ui/progress/updater.go:40 +0x26 fp=0xc0001ebfe0 sp=0xc0001ebfc8 pc=0x1529406
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001ebfe8 sp=0xc0001ebfe0 pc=0x11970a1
created by github.com/restic/restic/internal/ui/progress.NewUpdater
        /restic/internal/ui/progress/updater.go:40 +0x16d

goroutine 99 [select]:
runtime.gopark(0xc0004fdf48?, 0x2?, 0x11?, 0x34?, 0xc0004fdf14?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0004fdd90 sp=0xc0004fdd70 pc=0x116b736
runtime.selectgo(0xc0004fdf48, 0xc0004fdf10, 0x1e2ac9a9000c3411?, 0x0, 0x8fb68bdd6a026055?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc0004fded0 sp=0xc0004fdd90 pc=0x1179adc
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xce fp=0xc0004fdf78 sp=0xc0004fded0 pc=0x173292e
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc0004fdfe0 sp=0xc0004fdf78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0004fdfe8 sp=0xc0004fdfe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 101 [select]:
runtime.gopark(0xc00051df48?, 0x2?, 0x3e?, 0xe2?, 0xc00051df14?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00051dd90 sp=0xc00051dd70 pc=0x116b736
runtime.selectgo(0xc00051df48, 0xc00051df10, 0x62113f37b0cfe23e?, 0x0, 0xae6c37b178831633?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc00051ded0 sp=0xc00051dd90 pc=0x1179adc
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xce fp=0xc00051df78 sp=0xc00051ded0 pc=0x173292e
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc00051dfe0 sp=0xc00051df78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00051dfe8 sp=0xc00051dfe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 100 [select]:
runtime.gopark(0xc0001edf48?, 0x2?, 0xb9?, 0x40?, 0xc0001edf14?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0001edd90 sp=0xc0001edd70 pc=0x116b736
runtime.selectgo(0xc0001edf48, 0xc0001edf10, 0x69c5aa62fd7040b9?, 0x0, 0xc7b33495cf6cb88c?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc0001eded0 sp=0xc0001edd90 pc=0x1179adc
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xce fp=0xc0001edf78 sp=0xc0001eded0 pc=0x173292e
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc0001edfe0 sp=0xc0001edf78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001edfe8 sp=0xc0001edfe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 98 [select]:
runtime.gopark(0xc00023df48?, 0x2?, 0x9d?, 0x83?, 0xc00023df14?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00023dd90 sp=0xc00023dd70 pc=0x116b736
runtime.selectgo(0xc00023df48, 0xc00023df10, 0x9b041c7378f8839d?, 0x0, 0xc1b1140a437cb732?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc00023ded0 sp=0xc00023dd90 pc=0x1179adc
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xce fp=0xc00023df78 sp=0xc00023ded0 pc=0x173292e
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc00023dfe0 sp=0xc00023df78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00023dfe8 sp=0xc00023dfe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 102 [select]:
runtime.gopark(0xc00023ff48?, 0x2?, 0x9a?, 0x12?, 0xc00023ff14?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00023fd90 sp=0xc00023fd70 pc=0x116b736
runtime.selectgo(0xc00023ff48, 0xc00023ff10, 0xd066fb78067e129a?, 0x0, 0x55803a065c14ede8?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc00023fed0 sp=0xc00023fd90 pc=0x1179adc
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xce fp=0xc00023ff78 sp=0xc00023fed0 pc=0x173292e
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc00023ffe0 sp=0xc00023ff78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00023ffe8 sp=0xc00023ffe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 103 [semacquire]:
runtime.gopark(0x0?, 0x0?, 0x40?, 0x71?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0005a3e80 sp=0xc0005a3e60 pc=0x116b736
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.semacquire1(0xc0000ce310, 0x0?, 0x1, 0x0)
        /usr/local/go/src/runtime/sema.go:150 +0x20f fp=0xc0005a3ee8 sp=0xc0005a3e80 pc=0x117abcf
sync.runtime_Semacquire(0x0?)
        /usr/local/go/src/runtime/sema.go:62 +0x25 fp=0xc0005a3f18 sp=0xc0005a3ee8 pc=0x1192205
sync.(*WaitGroup).Wait(0x0?)
        /usr/local/go/src/sync/waitgroup.go:139 +0x52 fp=0xc0005a3f40 sp=0xc0005a3f18 pc=0x11a5252
golang.org/x/sync/errgroup.(*Group).Wait(0xc0000ce300)
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:56 +0x27 fp=0xc0005a3f60 sp=0xc0005a3f40 pc=0x14cfe07
github.com/restic/restic/internal/repository.(*Repository).StartPackUploader.func1()
        /restic/internal/repository/repository.go:536 +0x1d fp=0xc0005a3f78 sp=0xc0005a3f60 pc=0x173dfbd
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc0005a3fe0 sp=0xc0005a3f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005a3fe8 sp=0xc0005a3fe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 104 [semacquire]:
runtime.gopark(0x28f295a8642?, 0x28f295a8642?, 0x40?, 0x25?, 0x113db7f?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00051fde0 sp=0xc00051fdc0 pc=0x116b736
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.semacquire1(0xc0000ce390, 0xf0?, 0x1, 0x0)
        /usr/local/go/src/runtime/sema.go:150 +0x20f fp=0xc00051fe48 sp=0xc00051fde0 pc=0x117abcf
sync.runtime_Semacquire(0xc0005f20f0?)
        /usr/local/go/src/runtime/sema.go:62 +0x25 fp=0xc00051fe78 sp=0xc00051fe48 pc=0x1192205
sync.(*WaitGroup).Wait(0x1f6e4c0?)
        /usr/local/go/src/sync/waitgroup.go:139 +0x52 fp=0xc00051fea0 sp=0xc00051fe78 pc=0x11a5252
golang.org/x/sync/errgroup.(*Group).Wait(0xc0000ce380)
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:56 +0x27 fp=0xc00051fec0 sp=0xc00051fea0 pc=0x14cfe07
github.com/restic/restic/internal/archiver.(*Archiver).Snapshot.func1()
        /restic/internal/archiver/archiver.go:855 +0x1ef fp=0xc00051ff78 sp=0xc00051fec0 pc=0x158ad4f
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc00051ffe0 sp=0xc00051ff78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00051ffe8 sp=0xc00051ffe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 105 [semacquire]:
runtime.gopark(0x0?, 0xc000629210?, 0x40?, 0x65?, 0x2?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0006291f0 sp=0xc0006291d0 pc=0x116b736
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.semacquire1(0xc00011202c, 0xa0?, 0x3, 0x0)
        /usr/local/go/src/runtime/sema.go:150 +0x20f fp=0xc000629258 sp=0xc0006291f0 pc=0x117abcf
sync.runtime_SemacquireMutex(0xc00015eb9a?, 0x2?, 0x1180d37?)
        /usr/local/go/src/runtime/sema.go:77 +0x25 fp=0xc000629288 sp=0xc000629258 pc=0x1192325
sync.(*RWMutex).RLock(...)
        /usr/local/go/src/sync/rwmutex.go:71
github.com/restic/restic/internal/fs.(*LocalVss).snapshotPath(0xc000112000, {0xc00003c790, 0x3})
        /restic/internal/fs/fs_local_vss.go:184 +0x1fa fp=0xc0006298c0 sp=0xc000629288 pc=0x14fd65a
github.com/restic/restic/internal/fs.(*LocalVss).Lstat(0xc00015eb40?, {0xc00003c790?, 0xc000629a68?})
        /restic/internal/fs/fs_local_vss.go:145 +0x1e fp=0xc0006298e8 sp=0xc0006298c0 pc=0x14fd25e
github.com/restic/restic/internal/archiver.(*Archiver).save(0xc0002bc140, {0x2218b88, 0xc0000ce340}, {0xc00015eb26, 0x2}, {0xc00003c790, 0x3}, 0xc000486420)
        /restic/internal/archiver/archiver.go:434 +0x1c2 fp=0xc000629b78 sp=0xc0006298e8 pc=0x1586ac2
github.com/restic/restic/internal/archiver.(*Archiver).saveTree(0xc0002bc140, {0x2218b88, 0xc0000ce340}, {0x1ff7131, 0x1}, 0xc0000ce200, 0xc0005f22b8, 0xe48834e3f773097a?)
        /restic/internal/archiver/archiver.go:664 +0x6ab fp=0xc000629e18 sp=0xc000629b78 pc=0x1588d4b
github.com/restic/restic/internal/archiver.(*Archiver).Snapshot.func1.1()
        /restic/internal/archiver/archiver.go:830 +0x1a5 fp=0xc000629f78 sp=0xc000629e18 pc=0x158b025
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000629fe0 sp=0xc000629f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000629fe8 sp=0xc000629fe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 106 [select]:
runtime.gopark(0xc0005a1ea8?, 0x2?, 0x96?, 0x6f?, 0xc0005a1e0c?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0005a1c50 sp=0xc0005a1c30 pc=0x116b736
runtime.selectgo(0xc0005a1ea8, 0xc0005a1e08, 0x0?, 0x0, 0x52f61006b239a?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc0005a1d90 sp=0xc0005a1c50 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0x9e096c6e797c9395?, {0x2218b88, 0xc0000ce340}, 0xc000086240)
        /restic/internal/archiver/blob_saver.go:88 +0xf5 fp=0xc0005a1f48 sp=0xc0005a1d90 pc=0x158b995
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:34 +0x29 fp=0xc0005a1f78 sp=0xc0005a1f48 pc=0x158b549
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc0005a1fe0 sp=0xc0005a1f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005a1fe8 sp=0xc0005a1fe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 107 [select]:
runtime.gopark(0xc0005abea8?, 0x2?, 0x71?, 0x69?, 0xc0005abe0c?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0005abc50 sp=0xc0005abc30 pc=0x116b736
runtime.selectgo(0xc0005abea8, 0xc0005abe08, 0x4a2a?, 0x0, 0xe7b1a00a20f6b?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc0005abd90 sp=0xc0005abc50 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0x148d9f92da2460d7?, {0x2218b88, 0xc0000ce340}, 0xc000086240)
        /restic/internal/archiver/blob_saver.go:88 +0xf5 fp=0xc0005abf48 sp=0xc0005abd90 pc=0x158b995
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:34 +0x29 fp=0xc0005abf78 sp=0xc0005abf48 pc=0x158b549
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc0005abfe0 sp=0xc0005abf78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005abfe8 sp=0xc0005abfe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 108 [select]:
runtime.gopark(0xc00059dea8?, 0x2?, 0x71?, 0x0?, 0xc00059de0c?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00059dc50 sp=0xc00059dc30 pc=0x116b736
runtime.selectgo(0xc00059dea8, 0xc00059de08, 0x0?, 0x0, 0x235b00249200?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc00059dd90 sp=0xc00059dc50 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0x1117d110cfe4692f?, {0x2218b88, 0xc0000ce340}, 0xc000086240)
        /restic/internal/archiver/blob_saver.go:88 +0xf5 fp=0xc00059df48 sp=0xc00059dd90 pc=0x158b995
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:34 +0x29 fp=0xc00059df78 sp=0xc00059df48 pc=0x158b549
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc00059dfe0 sp=0xc00059df78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00059dfe8 sp=0xc00059dfe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 109 [select]:
runtime.gopark(0xc0005a7ea8?, 0x2?, 0xab?, 0x1?, 0xc0005a7e0c?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0005a7c50 sp=0xc0005a7c30 pc=0x116b736
runtime.selectgo(0xc0005a7ea8, 0xc0005a7e08, 0x168?, 0x0, 0x1fb003c87ac?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc0005a7d90 sp=0xc0005a7c50 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0x79bfeb4449bd860f?, {0x2218b88, 0xc0000ce340}, 0xc000086240)
        /restic/internal/archiver/blob_saver.go:88 +0xf5 fp=0xc0005a7f48 sp=0xc0005a7d90 pc=0x158b995
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:34 +0x29 fp=0xc0005a7f78 sp=0xc0005a7f48 pc=0x158b549
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc0005a7fe0 sp=0xc0005a7f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005a7fe8 sp=0xc0005a7fe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 110 [select]:
runtime.gopark(0xc00062ddf0?, 0x2?, 0x67?, 0x1f?, 0xc00062dddc?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00062dc20 sp=0xc00062dc00 pc=0x116b736
runtime.selectgo(0xc00062ddf0, 0xc00062ddd8, 0x37ee558338ca41?, 0x0, 0x800000?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc00062dd60 sp=0xc00062dc20 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*FileSaver).worker(0xc00031c300, {0x2218b88, 0xc0000ce340}, 0xc000086360)
        /restic/internal/archiver/file_saver.go:256 +0x1ce fp=0xc00062df48 sp=0xc00062dd60 pc=0x158d7ee
github.com/restic/restic/internal/archiver.NewFileSaver.func2()
        /restic/internal/archiver/file_saver.go:55 +0x29 fp=0xc00062df78 sp=0xc00062df48 pc=0x158bfc9
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc00062dfe0 sp=0xc00062df78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00062dfe8 sp=0xc00062dfe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 111 [select]:
runtime.gopark(0xc000683df0?, 0x2?, 0x67?, 0x1f?, 0xc000683ddc?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000683c20 sp=0xc000683c00 pc=0x116b736
runtime.selectgo(0xc000683df0, 0xc000683dd8, 0x37ee558338ca41?, 0x0, 0x800000?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000683d60 sp=0xc000683c20 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*FileSaver).worker(0xc00031c300, {0x2218b88, 0xc0000ce340}, 0xc000086360)
        /restic/internal/archiver/file_saver.go:256 +0x1ce fp=0xc000683f48 sp=0xc000683d60 pc=0x158d7ee
github.com/restic/restic/internal/archiver.NewFileSaver.func2()
        /restic/internal/archiver/file_saver.go:55 +0x29 fp=0xc000683f78 sp=0xc000683f48 pc=0x158bfc9
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000683fe0 sp=0xc000683f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000683fe8 sp=0xc000683fe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 112 [select]:
runtime.gopark(0xc0005a5e10?, 0x2?, 0xb2?, 0xeb?, 0xc0005a5d84?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0005a5c00 sp=0xc0005a5be0 pc=0x116b736
runtime.selectgo(0xc0005a5e10, 0xc0005a5d80, 0x2463c1ce92072fcd?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc0005a5d40 sp=0xc0005a5c00 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc611185ed82950b0?, {0x2218b88, 0xc0000ce340}, 0xc0000863c0)
        /restic/internal/archiver/tree_saver.go:156 +0x1b9 fp=0xc0005a5f48 sp=0xc0005a5d40 pc=0x1592319
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x29 fp=0xc0005a5f78 sp=0xc0005a5f48 pc=0x1591209
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc0005a5fe0 sp=0xc0005a5f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005a5fe8 sp=0xc0005a5fe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 113 [select]:
runtime.gopark(0xc0005b7e10?, 0x2?, 0xc4?, 0xa?, 0xc0005b7d84?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0005b7c00 sp=0xc0005b7be0 pc=0x116b736
runtime.selectgo(0xc0005b7e10, 0xc0005b7d80, 0x92a7a628b8e31d6e?, 0x0, 0x6635?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc0005b7d40 sp=0xc0005b7c00 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xa213fcb82d309ea0?, {0x2218b88, 0xc0000ce340}, 0xc0000863c0)
        /restic/internal/archiver/tree_saver.go:156 +0x1b9 fp=0xc0005b7f48 sp=0xc0005b7d40 pc=0x1592319
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x29 fp=0xc0005b7f78 sp=0xc0005b7f48 pc=0x1591209
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc0005b7fe0 sp=0xc0005b7f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005b7fe8 sp=0xc0005b7fe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 114 [select]:
runtime.gopark(0xc0005b9e10?, 0x2?, 0xd7?, 0x8b?, 0xc0005b9d84?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0005b9c00 sp=0xc0005b9be0 pc=0x116b736
runtime.selectgo(0xc0005b9e10, 0xc0005b9d80, 0x43b20e2891c5c9ce?, 0x0, 0x6d65?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc0005b9d40 sp=0xc0005b9c00 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xb317997e3868d498?, {0x2218b88, 0xc0000ce340}, 0xc0000863c0)
        /restic/internal/archiver/tree_saver.go:156 +0x1b9 fp=0xc0005b9f48 sp=0xc0005b9d40 pc=0x1592319
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x29 fp=0xc0005b9f78 sp=0xc0005b9f48 pc=0x1591209
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc0005b9fe0 sp=0xc0005b9f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005b9fe8 sp=0xc0005b9fe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 115 [select]:
runtime.gopark(0xc00063de10?, 0x2?, 0x53?, 0x95?, 0xc00063dd84?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00063dc00 sp=0xc00063dbe0 pc=0x116b736
runtime.selectgo(0xc00063de10, 0xc00063dd80, 0x1dfd91fe0b2539d9?, 0x0, 0xb5?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc00063dd40 sp=0xc00063dc00 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xf1389c3625fea17c?, {0x2218b88, 0xc0000ce340}, 0xc0000863c0)
        /restic/internal/archiver/tree_saver.go:156 +0x1b9 fp=0xc00063df48 sp=0xc00063dd40 pc=0x1592319
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x29 fp=0xc00063df78 sp=0xc00063df48 pc=0x1591209
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc00063dfe0 sp=0xc00063df78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00063dfe8 sp=0xc00063dfe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 116 [select]:
runtime.gopark(0xc0005b5e10?, 0x2?, 0xa8?, 0xf?, 0xc0005b5d84?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0005b5c00 sp=0xc0005b5be0 pc=0x116b736
runtime.selectgo(0xc0005b5e10, 0xc0005b5d80, 0x6c8afec310438dc6?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc0005b5d40 sp=0xc0005b5c00 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x418d70acd23db748?, {0x2218b88, 0xc0000ce340}, 0xc0000863c0)
        /restic/internal/archiver/tree_saver.go:156 +0x1b9 fp=0xc0005b5f48 sp=0xc0005b5d40 pc=0x1592319
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x29 fp=0xc0005b5f78 sp=0xc0005b5f48 pc=0x1591209
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc0005b5fe0 sp=0xc0005b5f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005b5fe8 sp=0xc0005b5fe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 117 [select]:
runtime.gopark(0xc000639e10?, 0x2?, 0x54?, 0x38?, 0xc000639d84?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000639c00 sp=0xc000639be0 pc=0x116b736
runtime.selectgo(0xc000639e10, 0xc000639d80, 0x3e309b87c4fa1aad?, 0x0, 0x581a?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000639d40 sp=0xc000639c00 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x91dda1a7389534a1?, {0x2218b88, 0xc0000ce340}, 0xc0000863c0)
        /restic/internal/archiver/tree_saver.go:156 +0x1b9 fp=0xc000639f48 sp=0xc000639d40 pc=0x1592319
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x29 fp=0xc000639f78 sp=0xc000639f48 pc=0x1591209
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000639fe0 sp=0xc000639f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000639fe8 sp=0xc000639fe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5
PS C:\restic>
PS C:\restic> .\r_prod.cmd version
restic 0.17.1 compiled with go1.19.13 on windows/amd64
PS C:\restic>
PS C:\restic> .\backup_prod.cmd
open repository
repository 5304de1b opened (version 2, compression level auto)
using parent snapshot 380401e3
load index files
[0:00] 100.00%  36 / 36 index files loaded
start scan on [K:\]
start backup on [K:\]
creating VSS snapshot for [k:\]
             runtime: mcall called on m->g0 stack
successfully created snapshot for [k:\]

runtime stack:
runtime.throw({0x202c3df?, 0x2205f80?})
        /usr/local/go/src/runtime/panic.go:1047 +0x65 fp=0xc000e092c8 sp=0xc000e09298 pc=0x1168c65
runtime.badmcall(0x310100c000e09310?)
        /usr/local/go/src/runtime/proc.go:468 +0x25 fp=0xc000e092e8 sp=0xc000e092c8 pc=0x116bee5
runtime.badmcall(0x20)
        <autogenerated>:1 +0x2c fp=0xc000e09300 sp=0xc000e092e8 pc=0x11993cc
runtime.asyncPreempt2()
        /usr/local/go/src/runtime/preempt.go:308 +0x3f fp=0xc000e09320 sp=0xc000e09300 pc=0x1169e9f
runtime.asyncPreempt()
        /usr/local/go/src/runtime/preempt_amd64.s:50 +0xdb fp=0xc000e094a8 sp=0xc000e09320 pc=0x119871b
github.com/restic/restic/internal/fs.(*LocalVss).snapshotPath(0xc00019bf10, {0xc0001a7410, 0x3})
        /restic/internal/fs/fs_local_vss.go:280 +0x101d fp=0xc000e09ae0 sp=0xc000e094a8 pc=0x14fe47d
github.com/restic/restic/internal/fs.(*LocalVss).Lstat(0xc0001a7410?, {0xc0001a7410?, 0xc0001a7410?})
        /restic/internal/fs/fs_local_vss.go:145 +0x1e fp=0xc000e09b08 sp=0xc000e09ae0 pc=0x14fd25e
github.com/restic/restic/internal/archiver.(*Scanner).scan(0xc0000e61b0, {0x2218b88, 0xc0000745c0}, {0x1dcb340?, 0xc0001abb01?, 0xc00065bce0?, 0x159006c?}, {0xc0001a7410, 0x3})
        /restic/internal/archiver/scanner.go:112 +0xc2 fp=0xc000e09bc8 sp=0xc000e09b08 pc=0x158e362
github.com/restic/restic/internal/archiver.(*Scanner).scanTree(0xc0000e61b0, {0x2218b88, 0xc0000745c0}, {0xc000074640?, 0x1?, 0xa37f09a500000000?, 0xc00065bd78?}, {0x0, {0xc00030ee80, 0x3}, ...})
        /restic/internal/archiver/scanner.go:49 +0x118 fp=0xc000e09cf0 sp=0xc000e09bc8 pc=0x158dbb8
github.com/restic/restic/internal/archiver.(*Scanner).scanTree(0xc0000e61b0, {0x2218b88, 0xc0000745c0}, {0xc00065bea8?, 0x1136a25?, 0x1b0dc962fa29160?, 0xfce81dc?}, {0xc0000e61e0, {0x0, 0x0}, ...})
        /restic/internal/archiver/scanner.go:60 +0x331 fp=0xc000e09e18 sp=0xc000e09cf0 pc=0x158ddd1
github.com/restic/restic/internal/archiver.(*Scanner).Scan(0xc0000e61b0, {0x2218b88, 0xc0000745c0}, {0xc0004a4800, 0x1, 0x1})
        /restic/internal/archiver/scanner.go:91 +0x1f5 fp=0xc000e09f38 sp=0xc000e09e18 pc=0x158e0f5
main.runBackup.func5()
        /restic/cmd/restic/cmd_backup.go:615 +0x31 fp=0xc000e09f78 sp=0xc000e09f38 pc=0x1c940d1
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000e09fe0 sp=0xc000e09f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000e09fe8 sp=0xc000e09fe0 pc=0x11970a1

goroutine 57 [running]:
runtime: g 57: unexpected return pc for runtime.systemstack_switch called from 0xc0002841a0
stack: frame={sp:0xc000e09380, fp:0xc000e09388} stack=[0xc000dea000,0xc000e0a000)
0x000000c000e09280:  0x000000c000e09298  0x000000c000e092b8
0x000000c000e09290:  0x0000000001168c65 <runtime.throw+0x0000000000000065>  0x000000c000e092a0
0x000000c000e092a0:  0x0000000001168c80 <runtime.throw.func1+0x0000000000000000>  0x000000000202c3df
0x000000c000e092b0:  0x0000000000000024  0x000000c000e092d8
0x000000c000e092c0:  0x000000000116bee5 <runtime.badmcall+0x0000000000000025>  0x000000000202c3df
0x000000c000e092d0:  0x0000000002205f80  0x000000c000e092f0
0x000000c000e092e0:  0x00000000011993cc <runtime.badmcall+0x000000000000002c>  0x310100c000e09310
0x000000c000e092f0:  0x000000c000e09310  0x0000000001169e9f <runtime.asyncPreempt2+0x000000000000003f>
0x000000c000e09300:  0x0000000000000020  0x000000c0002841a0
0x000000c000e09310:  0x000000c000e09498  0x000000000119871b <runtime.asyncPreempt+0x00000000000000db>
0x000000c000e09320:  0x0000000000000000  0x0000000001198640 <runtime.asyncPreempt+0x0000000000000000>
0x000000c000e09330:  0x0000000000000000  0x000000c000280678
0x000000c000e09340:  0x00000091d79ff988  0x00000091d659e000
0x000000c000e09350:  0x000000c000e094a0  0x00000091d79ff958
0x000000c000e09360:  0x0000000000000000  0x0000000000010244
0x000000c000e09370:  0x0000000000000000  0x0000000000000000
0x000000c000e09380: <0x000000c0002841a0 >0xffffffffffffffff
0x000000c000e09390:  0x0000000000000000  0x0000000000000000
0x000000c000e093a0:  0x0000000000000000  0x0000000000000000
0x000000c000e093b0:  0x0000000000000000  0x0000000000000000
0x000000c000e093c0:  0x0000000000000000  0x0000000000000000
0x000000c000e093d0:  0x0000000000000000  0x0000000000000000
0x000000c000e093e0:  0x0000000000000000  0x0000000000000000
0x000000c000e093f0:  0x000000c000074580  0x00000000014d00c0 <golang.org/x/sync/errgroup.(*Group).Go.func1.2+0x0000000000000000>
0x000000c000e09400:  0x000000c000074580  0x000000c00065bf30
0x000000c000e09410:  0x000000c00065bf40  0x000000c000685fd0
0x000000c000e09420:  0x0000000001529406 <github.com/restic/restic/internal/ui/progress.NewUpdater.func1+0x0000000000000026>  0x000000c000685f98
0x000000c000e09430:  0x0000000000000000  0x00000000011970a1 <runtime.goexit+0x0000000000000001>
0x000000c000e09440:  0x0000000000000000  0x0000000000000000
0x000000c000e09450:  0x0000000000000000  0x0000000000000000
0x000000c000e09460:  0x000000c000075200  0x000000c0001e3620
0x000000c000e09470:  0x000000c000302000  0x000000c000306000
0x000000c000e09480:  0x0000000000000000
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:459 fp=0xc000e09388 sp=0xc000e09380 pc=0x1194d40
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 1 [semacquire]:
runtime.gopark(0x254ea192498?, 0x6?, 0x0?, 0x66?, 0x113db7f?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000688cd0 sp=0xc000688cb0 pc=0x116b736
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.semacquire1(0xc000074750, 0xe8?, 0x1, 0x0)
        /usr/local/go/src/runtime/sema.go:150 +0x20f fp=0xc000688d38 sp=0xc000688cd0 pc=0x117abcf
sync.runtime_Semacquire(0xc00064a5e8?)
        /usr/local/go/src/runtime/sema.go:62 +0x25 fp=0xc000688d68 sp=0xc000688d38 pc=0x1192205
sync.(*WaitGroup).Wait(0x1f6e5e0?)
        /usr/local/go/src/sync/waitgroup.go:139 +0x52 fp=0xc000688d90 sp=0xc000688d68 pc=0x11a5252
golang.org/x/sync/errgroup.(*Group).Wait(0xc000074740)
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:56 +0x27 fp=0xc000688db0 sp=0xc000688d90 pc=0x14cfe07
github.com/restic/restic/internal/archiver.(*Archiver).Snapshot(0xc0000ae780, {0x2218b88?, 0xc000074440}, {0xc0004a4800, 0x1, 0x1}, {{0x2a66db0, 0x0, 0x0}, {0xc00030ece0, ...}, ...})
        /restic/internal/archiver/archiver.go:865 +0x3b8 fp=0xc000689010 sp=0xc000688db0 pc=0x158a498
main.runBackup({_, _}, {{{0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, {0xc0004a4710, 0x1, ...}, ...}, ...}, ...)
        /restic/cmd/restic/cmd_backup.go:660 +0x1a48 fp=0xc000689888 sp=0xc000689010 pc=0x1c93c28
main.glob..func2(0x29f01a0, {0xc0004ffc30, 0x0, 0xb})
        /restic/cmd/restic/cmd_backup.go:63 +0xf8 fp=0xc000689bb8 sp=0xc000689888 pc=0x1c8f6b8
github.com/spf13/cobra.(*Command).execute(0x29f01a0, {0xc0004ffb80, 0xb, 0xb})
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985 +0xab4 fp=0xc000689d58 sp=0xc000689bb8 pc=0x14c1914
github.com/spf13/cobra.(*Command).ExecuteC(0x29efbe0)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x425 fp=0xc000689e30 sp=0xc000689d58 pc=0x14c2245
github.com/spf13/cobra.(*Command).Execute(...)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1034
main.main()
        /restic/cmd/restic/main.go:142 +0x258 fp=0xc000689f80 sp=0xc000689e30 pc=0x1cceab8
runtime.main()
        /usr/local/go/src/runtime/proc.go:250 +0x1fe fp=0xc000689fe0 sp=0xc000689f80 pc=0x116b39e
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000689fe8 sp=0xc000689fe0 pc=0x11970a1

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00005ffb0 sp=0xc00005ff90 pc=0x116b736
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:302 +0xb1 fp=0xc00005ffe0 sp=0xc00005ffb0 pc=0x116b5d1
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00005ffe8 sp=0xc00005ffe0 pc=0x11970a1
created by runtime.init.6
        /usr/local/go/src/runtime/proc.go:290 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000061f90 sp=0xc000061f70 pc=0x116b736
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
        /usr/local/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc000061fc8 sp=0xc000061f90 pc=0x11557b7
runtime.gcenable.func1()
        /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc000061fe0 sp=0xc000061fc8 pc=0x114a266
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0x11970a1
created by runtime.gcenable
        /usr/local/go/src/runtime/mgc.go:178 +0x6b

goroutine 4 [sleep]:
runtime.gopark(0xc000078000?, 0x325ba0d1a974?, 0x0?, 0x0?, 0x209d090?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000071f30 sp=0xc000071f10 pc=0x116b736
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.(*scavengerState).sleep(0x2a0f740, 0x4131170000000000)
        /usr/local/go/src/runtime/mgcscavenge.go:468 +0x12a fp=0xc000071fa0 sp=0xc000071f30 pc=0x1153a0a
runtime.bgscavenge(0x0?)
        /usr/local/go/src/runtime/mgcscavenge.go:626 +0x7f fp=0xc000071fc8 sp=0xc000071fa0 pc=0x1153e1f
runtime.gcenable.func2()
        /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000071fe0 sp=0xc000071fc8 pc=0x114a206
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000071fe8 sp=0xc000071fe0 pc=0x11970a1
created by runtime.gcenable
        /usr/local/go/src/runtime/mgc.go:179 +0xaa

goroutine 5 [finalizer wait]:
runtime.gopark(0x0?, 0xc000063e70?, 0x8b?, 0x76?, 0xc000063f70?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000063e28 sp=0xc000063e08 pc=0x116b736
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:180 +0x10f fp=0xc000063fe0 sp=0xc000063e28 pc=0x114936f
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x11970a1
created by runtime.createfing
        /usr/local/go/src/runtime/mfinal.go:157 +0x45

goroutine 6 [chan receive]:
runtime.gopark(0xc000073e90?, 0xc000073ee8?, 0x51?, 0x61?, 0xc000073ee8?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000073eb8 sp=0xc000073e98 pc=0x116b736
runtime.chanrecv(0xc000086000, 0xc000073fa0, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc000073f48 sp=0xc000073eb8 pc=0x1136fbb
runtime.chanrecv2(0xc000086000?, 0xc000073fc0?)
        /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000073f70 sp=0xc000073f48 pc=0x1136af8
github.com/restic/restic/internal/restic.init.0.func1.1()
        /restic/internal/restic/lock.go:429 +0x9d fp=0xc000073fe0 sp=0xc000073f70 pc=0x156c25d
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000073fe8 sp=0xc000073fe0 pc=0x11970a1
created by github.com/restic/restic/internal/restic.init.0.func1
        /restic/internal/restic/lock.go:426 +0x25

goroutine 18 [syscall]:
runtime.notetsleepg(0x0?, 0x0?)
        /usr/local/go/src/runtime/lock_sema.go:294 +0x3c fp=0xc00006dfa0 sp=0xc00006df58 pc=0x113c5dc
os/signal.signal_recv()
        /usr/local/go/src/runtime/sigqueue.go:152 +0x2f fp=0xc00006dfc0 sp=0xc00006dfa0 pc=0x119292f
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x19 fp=0xc00006dfe0 sp=0xc00006dfc0 pc=0x11f6579
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0x11970a1
created by os/signal.Notify.func1.1
        /usr/local/go/src/os/signal/signal.go:151 +0x2a

goroutine 136 [select]:
runtime.gopark(0xc000243e10?, 0x2?, 0x80?, 0x11?, 0xc000243d84?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000243c00 sp=0xc000243be0 pc=0x116b736
runtime.selectgo(0xc000243e10, 0xc000243d80, 0x4?, 0x0, 0x3?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000243d40 sp=0xc000243c00 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x0?, {0x2218b88, 0xc000074800}, 0xc00007a300)
        /restic/internal/archiver/tree_saver.go:156 +0x1b9 fp=0xc000243f48 sp=0xc000243d40 pc=0x1592319
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x29 fp=0xc000243f78 sp=0xc000243f48 pc=0x1591209
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000243fe0 sp=0xc000243f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000243fe8 sp=0xc000243fe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 50 [GC worker (idle)]:
runtime.gopark(0x325b1fdc5a08?, 0x1?, 0x10?, 0x63?, 0x1136a09?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00023ff50 sp=0xc00023ff30 pc=0x116b736
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00023ffe0 sp=0xc00023ff50 pc=0x114c471
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00023ffe8 sp=0xc00023ffe0 pc=0x11970a1
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x325b1fdc5a08?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00006ff50 sp=0xc00006ff30 pc=0x116b736
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00006ffe0 sp=0xc00006ff50 pc=0x114c471
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00006ffe8 sp=0xc00006ffe0 pc=0x11970a1
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 35 [GC worker (idle)]:
runtime.gopark(0x325b1fd3ba38?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000241f50 sp=0xc000241f30 pc=0x116b736
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000241fe0 sp=0xc000241f50 pc=0x114c471
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000241fe8 sp=0xc000241fe0 pc=0x11970a1
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 8 [GC worker (idle)]:
runtime.gopark(0x325b12937cf0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000245f50 sp=0xc000245f30 pc=0x116b736
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000245fe0 sp=0xc000245f50 pc=0x114c471
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000245fe8 sp=0xc000245fe0 pc=0x11970a1
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 20 [select]:
runtime.gopark(0xc0005e3f88?, 0x3?, 0x0?, 0x2a?, 0xc0005e3f72?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0005e3df8 sp=0xc0005e3dd8 pc=0x116b736
runtime.selectgo(0xc0005e3f88, 0xc0005e3f6c, 0xc000094480?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc0005e3f38 sp=0xc0005e3df8 pc=0x1179adc
go.opencensus.io/stats/view.(*worker).start(0xc000094480)
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0xad fp=0xc0005e3fc8 sp=0xc0005e3f38 pc=0x1b6abed
go.opencensus.io/stats/view.init.0.func1()
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x26 fp=0xc0005e3fe0 sp=0xc0005e3fc8 pc=0x1b69e26
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005e3fe8 sp=0xc0005e3fe0 pc=0x11970a1
created by go.opencensus.io/stats/view.init.0
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 9 [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0005e1e80 sp=0xc0005e1e60 pc=0x116b736
runtime.chanrecv(0xc000310cc0, 0xc0005e1f80, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc0005e1f10 sp=0xc0005e1e80 pc=0x1136fbb
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc0005e1f38 sp=0xc0005e1f10 pc=0x1136ab8
main.cleanupHandler(0x0?, 0xc0004a4660)
        /restic/cmd/restic/cleanup.go:24 +0x30 fp=0xc0005e1fc0 sp=0xc0005e1f38 pc=0x1c8f310
main.createGlobalContext.func1()
        /restic/cmd/restic/cleanup.go:16 +0x2a fp=0xc0005e1fe0 sp=0xc0005e1fc0 pc=0x1c8f2aa
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005e1fe8 sp=0xc0005e1fe0 pc=0x11970a1
created by main.createGlobalContext
        /restic/cmd/restic/cleanup.go:16 +0xa5

goroutine 10 [select]:
runtime.gopark(0xc0005e5f10?, 0x3?, 0x28?, 0x0?, 0xc0005e5eaa?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0005e5d00 sp=0xc0005e5ce0 pc=0x116b736
runtime.selectgo(0xc0005e5f10, 0xc0005e5ea4, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc0005e5e40 sp=0xc0005e5d00 pc=0x1179adc
github.com/restic/restic/internal/ui/termstatus.(*Terminal).run(0xc000310de0, {0x2218b88, 0xc000511400})
        /restic/internal/ui/termstatus/status.go:105 +0x125 fp=0xc0005e5f50 sp=0xc0005e5e40 pc=0x174adc5
github.com/restic/restic/internal/ui/termstatus.(*Terminal).Run(0x0?, {0x2218b88?, 0xc000511400?})
        /restic/internal/ui/termstatus/status.go:94 +0x68 fp=0xc0005e5f98 sp=0xc0005e5f50 pc=0x174abc8
main.setupTermstatus.func1()
        /restic/cmd/restic/termstatus.go:28 +0x65 fp=0xc0005e5fe0 sp=0xc0005e5f98 pc=0x1cd0f45
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005e5fe8 sp=0xc0005e5fe0 pc=0x11970a1
created by main.setupTermstatus
        /restic/cmd/restic/termstatus.go:26 +0x138

goroutine 134 [select]:
runtime.gopark(0xc000d5fdf0?, 0x2?, 0x67?, 0x1f?, 0xc000d5fddc?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000d5fc20 sp=0xc000d5fc00 pc=0x116b736
runtime.selectgo(0xc000d5fdf0, 0xc000d5fdd8, 0x37ee558338ca41?, 0x0, 0x800000?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000d5fd60 sp=0xc000d5fc20 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*FileSaver).worker(0xc0000e6540, {0x2218b88, 0xc000074800}, 0xc00007a2a0)
        /restic/internal/archiver/file_saver.go:256 +0x1ce fp=0xc000d5ff48 sp=0xc000d5fd60 pc=0x158d7ee
github.com/restic/restic/internal/archiver.NewFileSaver.func2()
        /restic/internal/archiver/file_saver.go:55 +0x29 fp=0xc000d5ff78 sp=0xc000d5ff48 pc=0x158bfc9
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000d5ffe0 sp=0xc000d5ff78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000d5ffe8 sp=0xc000d5ffe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 135 [select]:
runtime.gopark(0xc000d2ddf0?, 0x2?, 0x67?, 0x1f?, 0xc000d2dddc?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000d2dc20 sp=0xc000d2dc00 pc=0x116b736
runtime.selectgo(0xc000d2ddf0, 0xc000d2ddd8, 0x37ee558338ca41?, 0x0, 0x800000?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000d2dd60 sp=0xc000d2dc20 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*FileSaver).worker(0xc0000e6540, {0x2218b88, 0xc000074800}, 0xc00007a2a0)
        /restic/internal/archiver/file_saver.go:256 +0x1ce fp=0xc000d2df48 sp=0xc000d2dd60 pc=0x158d7ee
github.com/restic/restic/internal/archiver.NewFileSaver.func2()
        /restic/internal/archiver/file_saver.go:55 +0x29 fp=0xc000d2df78 sp=0xc000d2df48 pc=0x158bfc9
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000d2dfe0 sp=0xc000d2df78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000d2dfe8 sp=0xc000d2dfe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 133 [select]:
runtime.gopark(0xc000275ea8?, 0x2?, 0xff?, 0xff?, 0xc000275e0c?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000275c50 sp=0xc000275c30 pc=0x116b736
runtime.selectgo(0xc000275ea8, 0xc000275e08, 0x0?, 0x0, 0x10000?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000275d90 sp=0xc000275c50 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0x0?, {0x2218b88, 0xc000074800}, 0xc00007a240)
        /restic/internal/archiver/blob_saver.go:88 +0xf5 fp=0xc000275f48 sp=0xc000275d90 pc=0x158b995
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:34 +0x29 fp=0xc000275f78 sp=0xc000275f48 pc=0x158b549
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000275fe0 sp=0xc000275f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000275fe8 sp=0xc000275fe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 68 [IO wait]:
runtime.gopark(0x0?, 0xc000348c98?, 0x48?, 0x8d?, 0xc000348cc8?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0005175d0 sp=0xc0005175b0 pc=0x116b736
runtime.netpollblock(0xc00031e658?, 0x517688?, 0xc0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc000517608 sp=0xc0005175d0 pc=0x1161df7
internal/poll.runtime_pollWait(0x254e4848918, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc000517628 sp=0xc000517608 pc=0x1190c09
internal/poll.(*pollDesc).wait(0xc0006220b0?, 0x2c?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000517650 sp=0xc000517628 pc=0x11de392
internal/poll.execIO(0xc000348c98, 0x209ca48)
        /usr/local/go/src/internal/poll/fd_windows.go:175 +0xe5 fp=0xc0005176a8 sp=0xc000517650 pc=0x11df9c5
internal/poll.(*FD).Read(0xc000348c80, {0xc000477800, 0x1800, 0x1800})
        /usr/local/go/src/internal/poll/fd_windows.go:441 +0x26b fp=0xc000517738 sp=0xc0005176a8 pc=0x11e084b
net.(*netFD).Read(0xc000348c80, {0xc000477800?, 0xc000092200?, 0xc000477805?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000517780 sp=0xc000517738 pc=0x131cc69
net.(*conn).Read(0xc00030a280, {0xc000477800?, 0x254e457b600?, 0xc000517828?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0005177c8 sp=0xc000517780 pc=0x13329a5
crypto/tls.(*atLeastReader).Read(0xc0002a4780, {0xc000477800?, 0x0?, 0x8?})
        /usr/local/go/src/crypto/tls/conn.go:787 +0x3d fp=0xc000517810 sp=0xc0005177c8 pc=0x136dafd
bytes.(*Buffer).ReadFrom(0xc000600cf8, {0x220b300, 0xc0002a4780})
        /usr/local/go/src/bytes/buffer.go:202 +0x98 fp=0xc000517868 sp=0xc000517810 pc=0x1246118
crypto/tls.(*Conn).readFromUntil(0xc000600a80, {0x254e484a258?, 0xc00030a280}, 0x1800?)
        /usr/local/go/src/crypto/tls/conn.go:809 +0xe5 fp=0xc0005178a8 sp=0xc000517868 pc=0x136dce5
crypto/tls.(*Conn).readRecordOrCCS(0xc000600a80, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:616 +0x116 fp=0xc000517c30 sp=0xc0005178a8 pc=0x136b136
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0xc000600a80, {0xc0000cc000, 0x1000, 0x0?})
        /usr/local/go/src/crypto/tls/conn.go:1315 +0x16f fp=0xc000517ca0 sp=0xc000517c30 pc=0x13710af
bufio.(*Reader).Read(0xc0000871a0, {0xc000444e40, 0x9, 0xc0004fd108?})
        /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc000517cd8 sp=0xc000517ca0 pc=0x13a67bb
io.ReadAtLeast({0x220a1a0, 0xc0000871a0}, {0xc000444e40, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:332 +0x9a fp=0xc000517d20 sp=0xc000517cd8 pc=0x11d71da
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc000444e40?, 0x9?, 0x0?}, {0x220a1a0?, 0xc0000871a0?})
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/frame.go:237 +0x6e fp=0xc000517d70 sp=0xc000517d20 pc=0x15413ae
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000444e00)
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/frame.go:501 +0x95 fp=0xc000517e20 sp=0xc000517d70 pc=0x1541c15
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000517f98)
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:2358 +0xd8 fp=0xc000517f60 sp=0xc000517e20 pc=0x15554f8
golang.org/x/net/http2.(*ClientConn).readLoop(0xc0004fc780)
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:2254 +0x99 fp=0xc000517fc8 sp=0xc000517f60 pc=0x1554a99
golang.org/x/net/http2.(*Transport).newClientConn.func1()
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:869 +0x26 fp=0xc000517fe0 sp=0xc000517fc8 pc=0x154d366
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000517fe8 sp=0xc000517fe0 pc=0x11970a1
created by golang.org/x/net/http2.(*Transport).newClientConn
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:869 +0xccd

goroutine 62 [select]:
runtime.gopark(0xc000671f48?, 0x2?, 0xd0?, 0xea?, 0xc000671f14?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000671d90 sp=0xc000671d70 pc=0x116b736
runtime.selectgo(0xc000671f48, 0xc000671f10, 0x3d87b8f3c25ac520?, 0x0, 0xd349e68ff34ad945?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000671ed0 sp=0xc000671d90 pc=0x1179adc
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xce fp=0xc000671f78 sp=0xc000671ed0 pc=0x173292e
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000671fe0 sp=0xc000671f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000671fe8 sp=0xc000671fe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 131 [select]:
runtime.gopark(0xc000277ea8?, 0x2?, 0x20?, 0x0?, 0xc000277e0c?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000277c50 sp=0xc000277c30 pc=0x116b736
runtime.selectgo(0xc000277ea8, 0xc000277e08, 0x0?, 0x0, 0xc000080400?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000277d90 sp=0xc000277c50 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0xc000277f78?, {0x2218b88, 0xc000074800}, 0xc00007a240)
        /restic/internal/archiver/blob_saver.go:88 +0xf5 fp=0xc000277f48 sp=0xc000277d90 pc=0x158b995
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:34 +0x29 fp=0xc000277f78 sp=0xc000277f48 pc=0x158b549
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000277fe0 sp=0xc000277f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000277fe8 sp=0xc000277fe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 60 [select]:
runtime.gopark(0xc0005dff48?, 0x2?, 0xf8?, 0x87?, 0xc0005dff14?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0005dfd90 sp=0xc0005dfd70 pc=0x116b736
runtime.selectgo(0xc0005dff48, 0xc0005dff10, 0x9357da99cf287f8?, 0x0, 0xca5164c245e41817?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc0005dfed0 sp=0xc0005dfd90 pc=0x1179adc
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xce fp=0xc0005dff78 sp=0xc0005dfed0 pc=0x173292e
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc0005dffe0 sp=0xc0005dff78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005dffe8 sp=0xc0005dffe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 25 [select]:
runtime.gopark(0xc000661f08?, 0x3?, 0x90?, 0xb0?, 0xc000661e8a?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000661cc8 sp=0xc000661ca8 pc=0x116b736
runtime.selectgo(0xc000661f08, 0xc000661e84, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000661e08 sp=0xc000661cc8 pc=0x1179adc
github.com/restic/restic/internal/repository.(*locker).refreshLocks(0x2972ba0, {0x2218b88, 0xc000074440}, {0x221f6e8, 0xc0001e3f80}, 0xc00004c7a0, 0xc0000fee40, 0xc0000feea0, 0x209cb38)
        /restic/internal/repository/lock.go:144 +0x22a fp=0xc000661f88 sp=0xc000661e08 pc=0x17304aa
github.com/restic/restic/internal/repository.(*locker).Lock.func1()
        /restic/internal/repository/lock.go:105 +0x46 fp=0xc000661fe0 sp=0xc000661f88 pc=0x1730246
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000661fe8 sp=0xc000661fe0 pc=0x11970a1
created by github.com/restic/restic/internal/repository.(*locker).Lock
        /restic/internal/repository/lock.go:105 +0x705

goroutine 130 [select]:
runtime.gopark(0xc000015ea8?, 0x2?, 0x20?, 0x0?, 0xc000015e0c?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000015c50 sp=0xc000015c30 pc=0x116b736
runtime.selectgo(0xc000015ea8, 0xc000015e08, 0xc0003be240?, 0x0, 0x1144711?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000015d90 sp=0xc000015c50 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0xc000015f78?, {0x2218b88, 0xc000074800}, 0xc00007a240)
        /restic/internal/archiver/blob_saver.go:88 +0xf5 fp=0xc000015f48 sp=0xc000015d90 pc=0x158b995
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:34 +0x29 fp=0xc000015f78 sp=0xc000015f48 pc=0x158b549
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000015fe0 sp=0xc000015f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000015fe8 sp=0xc000015fe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 132 [select]:
runtime.gopark(0xc000c9bea8?, 0x2?, 0x0?, 0x0?, 0xc000c9be0c?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000c9bc50 sp=0xc000c9bc30 pc=0x116b736
runtime.selectgo(0xc000c9bea8, 0xc000c9be08, 0x0?, 0x0, 0xcaae80006f7f6?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000c9bd90 sp=0xc000c9bc50 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0x17180d31fae48136?, {0x2218b88, 0xc000074800}, 0xc00007a240)
        /restic/internal/archiver/blob_saver.go:88 +0xf5 fp=0xc000c9bf48 sp=0xc000c9bd90 pc=0x158b995
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:34 +0x29 fp=0xc000c9bf78 sp=0xc000c9bf48 pc=0x158b549
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000c9bfe0 sp=0xc000c9bf78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000c9bfe8 sp=0xc000c9bfe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 26 [select]:
runtime.gopark(0xc000013f40?, 0x4?, 0x0?, 0x0?, 0xc000013e80?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000013ce8 sp=0xc000013cc8 pc=0x116b736
runtime.selectgo(0xc000013f40, 0xc000013e78, 0xc0000858e8?, 0x0, 0xc0000859a8?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000013e28 sp=0xc000013ce8 pc=0x1179adc
github.com/restic/restic/internal/repository.(*locker).monitorLockRefresh(0x2972ba0, {0x2218b88, 0xc000074440}, 0xc00004c7a0, 0xc0000fee40, 0xc0000feea0, 0x209cb38)
        /restic/internal/repository/lock.go:208 +0x23e fp=0xc000013f98 sp=0xc000013e28 pc=0x1730c7e
github.com/restic/restic/internal/repository.(*locker).Lock.func2()
        /restic/internal/repository/lock.go:106 +0x3e fp=0xc000013fe0 sp=0xc000013f98 pc=0x17301be
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000013fe8 sp=0xc000013fe0 pc=0x11970a1
created by github.com/restic/restic/internal/repository.(*locker).Lock
        /restic/internal/repository/lock.go:106 +0x7ec

goroutine 27 [select]:
runtime.gopark(0xc000519f78?, 0x3?, 0xb8?, 0xa2?, 0xc000519f0a?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000519d90 sp=0xc000519d70 pc=0x116b736
runtime.selectgo(0xc000519f78, 0xc000519f04, 0x2a0f2e0?, 0x0, 0x3963598d?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000519ed0 sp=0xc000519d90 pc=0x1179adc
github.com/restic/restic/internal/ui/progress.(*Updater).run(0xc0000744c0)
        /restic/internal/ui/progress/updater.go:73 +0x186 fp=0xc000519fc8 sp=0xc000519ed0 pc=0x1529686
github.com/restic/restic/internal/ui/progress.NewUpdater.func1()
        /restic/internal/ui/progress/updater.go:40 +0x26 fp=0xc000519fe0 sp=0xc000519fc8 pc=0x1529406
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000519fe8 sp=0xc000519fe0 pc=0x11970a1
created by github.com/restic/restic/internal/ui/progress.NewUpdater
        /restic/internal/ui/progress/updater.go:40 +0x16d

goroutine 65 [runnable]:
runtime.mapaccess2(0x1e07020, 0xc0000e65a0, 0xc000d31358?)
        /usr/local/go/src/runtime/map.go:456 +0x217 fp=0xc000d31308 sp=0xc000d31300 pc=0x113f2f7
sync.(*Map).Load(0x2a06640, {0x1fce500, 0x1e5bc60})
        /usr/local/go/src/sync/map.go:112 +0xb7 fp=0xc000d31378 sp=0xc000d31308 pc=0x11a1d57
encoding/json.cachedTypeFields({0x22283a0?, 0x1e5bc60})
        /usr/local/go/src/encoding/json/encode.go:1413 +0x4c fp=0xc000d313e0 sp=0xc000d31378 pc=0x151adac
encoding/json.(*decodeState).object(0xc000646000, {0x1ef9ca0?, 0xc000300990?, 0x0?})
        /usr/local/go/src/encoding/json/decode.go:655 +0x271 fp=0xc000d31640 sp=0xc000d313e0 pc=0x150dff1
encoding/json.(*decodeState).value(0xc000646000, {0x1ef9ca0?, 0xc000300990?, 0x90?})
        /usr/local/go/src/encoding/json/decode.go:374 +0x45 fp=0xc000d316b0 sp=0xc000d31640 pc=0x150cce5
encoding/json.(*decodeState).unmarshal(0xc000646000, {0x1ef9ca0?, 0xc000300990?})
        /usr/local/go/src/encoding/json/decode.go:181 +0x1de fp=0xc000d31728 sp=0xc000d316b0 pc=0x150c5fe
encoding/json.Unmarshal({0xc0001bc380, 0x356, 0x366}, {0x1ef9ca0, 0xc000300990})
        /usr/local/go/src/encoding/json/decode.go:108 +0x125 fp=0xc000d31760 sp=0xc000d31728 pc=0x150c105
github.com/restic/restic/internal/restic.LoadTree({0x2218b88, 0xc000074800}, {0x220cfa0, 0xc0005121a0}, {0x3a, 0x62, 0x3f, 0x4d, 0x8f, 0xe6, ...})
        /restic/internal/restic/tree.go:121 +0x159 fp=0xc000d31818 sp=0xc000d31760 pc=0x157da59
github.com/restic/restic/internal/archiver.(*Archiver).loadSubtree(0xc0000ae780, {0x2218b88, 0xc000074800}, 0xc000506580)
        /restic/internal/archiver/archiver.go:279 +0x10f fp=0xc000d318e8 sp=0xc000d31818 pc=0x1585aef
github.com/restic/restic/internal/archiver.(*Archiver).save(0xc0000ae780, {0x2218b88, 0xc000074800}, {0xc0001a7a76, 0x2}, {0xc00030ee80, 0x3}, 0xc000506580)
        /restic/internal/archiver/archiver.go:531 +0xefc fp=0xc000d31b78 sp=0xc000d318e8 pc=0x15877fc
github.com/restic/restic/internal/archiver.(*Archiver).saveTree(0xc0000ae780, {0x2218b88, 0xc000074800}, {0x1ff7131, 0x1}, 0xc000074680, 0xc00064a7c8, 0xfc667a67e02d258c?)
        /restic/internal/archiver/archiver.go:664 +0x6ab fp=0xc000d31e18 sp=0xc000d31b78 pc=0x1588d4b
github.com/restic/restic/internal/archiver.(*Archiver).Snapshot.func1.1()
        /restic/internal/archiver/archiver.go:830 +0x1a5 fp=0xc000d31f78 sp=0xc000d31e18 pc=0x158b025
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000d31fe0 sp=0xc000d31f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000d31fe8 sp=0xc000d31fe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 61 [select]:
runtime.gopark(0xc00066bf48?, 0x2?, 0x27?, 0x37?, 0xc00066bf14?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00066bd90 sp=0xc00066bd70 pc=0x116b736
runtime.selectgo(0xc00066bf48, 0xc00066bf10, 0x1ff8343a92733727?, 0x0, 0x12313d4303fd2baa?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc00066bed0 sp=0xc00066bd90 pc=0x1179adc
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xce fp=0xc00066bf78 sp=0xc00066bed0 pc=0x173292e
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc00066bfe0 sp=0xc00066bf78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00066bfe8 sp=0xc00066bfe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 64 [semacquire]:
runtime.gopark(0x254e4584342?, 0x254e4584342?, 0x60?, 0x63?, 0x113db7f?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00051dde0 sp=0xc00051ddc0 pc=0x116b736
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.semacquire1(0xc000074850, 0x0?, 0x1, 0x0)
        /usr/local/go/src/runtime/sema.go:150 +0x20f fp=0xc00051de48 sp=0xc00051dde0 pc=0x117abcf
sync.runtime_Semacquire(0xc00064a600?)
        /usr/local/go/src/runtime/sema.go:62 +0x25 fp=0xc00051de78 sp=0xc00051de48 pc=0x1192205
sync.(*WaitGroup).Wait(0x1f6e4c0?)
        /usr/local/go/src/sync/waitgroup.go:139 +0x52 fp=0xc00051dea0 sp=0xc00051de78 pc=0x11a5252
golang.org/x/sync/errgroup.(*Group).Wait(0xc000074840)
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:56 +0x27 fp=0xc00051dec0 sp=0xc00051dea0 pc=0x14cfe07
github.com/restic/restic/internal/archiver.(*Archiver).Snapshot.func1()
        /restic/internal/archiver/archiver.go:855 +0x1ef fp=0xc00051df78 sp=0xc00051dec0 pc=0x158ad4f
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc00051dfe0 sp=0xc00051df78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00051dfe8 sp=0xc00051dfe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 63 [semacquire]:
runtime.gopark(0x961718fb88de8ead?, 0xbd5f666e9a57d784?, 0x0?, 0x66?, 0xa931188e0397333e?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000c9de80 sp=0xc000c9de60 pc=0x116b736
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.semacquire1(0xc0000747d0, 0x31?, 0x1, 0x0)
        /usr/local/go/src/runtime/sema.go:150 +0x20f fp=0xc000c9dee8 sp=0xc000c9de80 pc=0x117abcf
sync.runtime_Semacquire(0xed3c48d437052f47?)
        /usr/local/go/src/runtime/sema.go:62 +0x25 fp=0xc000c9df18 sp=0xc000c9dee8 pc=0x1192205
sync.(*WaitGroup).Wait(0x24b5bd029adeb034?)
        /usr/local/go/src/sync/waitgroup.go:139 +0x52 fp=0xc000c9df40 sp=0xc000c9df18 pc=0x11a5252
golang.org/x/sync/errgroup.(*Group).Wait(0xc0000747c0)
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:56 +0x27 fp=0xc000c9df60 sp=0xc000c9df40 pc=0x14cfe07
github.com/restic/restic/internal/repository.(*Repository).StartPackUploader.func1()
        /restic/internal/repository/repository.go:536 +0x1d fp=0xc000c9df78 sp=0xc000c9df60 pc=0x173dfbd
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000c9dfe0 sp=0xc000c9df78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000c9dfe8 sp=0xc000c9dfe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 58 [select]:
runtime.gopark(0xc00065bf48?, 0x2?, 0x0?, 0x0?, 0xc00065bf14?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00065bd90 sp=0xc00065bd70 pc=0x116b736
runtime.selectgo(0xc00065bf48, 0xc00065bf10, 0x1d85860?, 0x0, 0xc0000e61e0?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc00065bed0 sp=0xc00065bd90 pc=0x1179adc
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xce fp=0xc00065bf78 sp=0xc00065bed0 pc=0x173292e
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc00065bfe0 sp=0xc00065bf78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00065bfe8 sp=0xc00065bfe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 59 [select]:
runtime.gopark(0xc00066ff48?, 0x2?, 0x80?, 0x79?, 0xc00066ff14?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00066fd90 sp=0xc00066fd70 pc=0x116b736
runtime.selectgo(0xc00066ff48, 0xc00066ff10, 0x808e09a860fc151e?, 0x0, 0xa75db6d96c335b6f?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc00066fed0 sp=0xc00066fd90 pc=0x1179adc
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xce fp=0xc00066ff78 sp=0xc00066fed0 pc=0x173292e
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc00066ffe0 sp=0xc00066ff78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00066ffe8 sp=0xc00066ffe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 137 [select]:
runtime.gopark(0xc000cdde10?, 0x2?, 0xa2?, 0xab?, 0xc000cddd84?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000cddc00 sp=0xc000cddbe0 pc=0x116b736
runtime.selectgo(0xc000cdde10, 0xc000cddd80, 0x5c6e5b16a5da5bd7?, 0x0, 0x1?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000cddd40 sp=0xc000cddc00 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xf3c10910cd5d38c9?, {0x2218b88, 0xc000074800}, 0xc00007a300)
        /restic/internal/archiver/tree_saver.go:156 +0x1b9 fp=0xc000cddf48 sp=0xc000cddd40 pc=0x1592319
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x29 fp=0xc000cddf78 sp=0xc000cddf48 pc=0x1591209
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000cddfe0 sp=0xc000cddf78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000cddfe8 sp=0xc000cddfe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 138 [select]:
runtime.gopark(0xc000c97e10?, 0x2?, 0x51?, 0xb9?, 0xc000c97d84?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000c97c00 sp=0xc000c97be0 pc=0x116b736
runtime.selectgo(0xc000c97e10, 0xc000c97d80, 0xed005ccbfa5041cb?, 0x0, 0x59?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000c97d40 sp=0xc000c97c00 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x5c48a9d3491b9420?, {0x2218b88, 0xc000074800}, 0xc00007a300)
        /restic/internal/archiver/tree_saver.go:156 +0x1b9 fp=0xc000c97f48 sp=0xc000c97d40 pc=0x1592319
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x29 fp=0xc000c97f78 sp=0xc000c97f48 pc=0x1591209
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000c97fe0 sp=0xc000c97f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000c97fe8 sp=0xc000c97fe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 139 [select]:
runtime.gopark(0xc000cd9e10?, 0x2?, 0x80?, 0x6d?, 0xc000cd9d84?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000cd9c00 sp=0xc000cd9be0 pc=0x116b736
runtime.selectgo(0xc000cd9e10, 0xc000cd9d80, 0x71ceff21ab7fa9a9?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000cd9d40 sp=0xc000cd9c00 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x97785b60c6fa1d08?, {0x2218b88, 0xc000074800}, 0xc00007a300)
        /restic/internal/archiver/tree_saver.go:156 +0x1b9 fp=0xc000cd9f48 sp=0xc000cd9d40 pc=0x1592319
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x29 fp=0xc000cd9f78 sp=0xc000cd9f48 pc=0x1591209
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000cd9fe0 sp=0xc000cd9f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000cd9fe8 sp=0xc000cd9fe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 140 [select]:
runtime.gopark(0xc000cdbe10?, 0x2?, 0xdd?, 0xd?, 0xc000cdbd84?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000cdbc00 sp=0xc000cdbbe0 pc=0x116b736
runtime.selectgo(0xc000cdbe10, 0xc000cdbd80, 0xb89d4e5cb89d3350?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000cdbd40 sp=0xc000cdbc00 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc4c6c6f5890aeb97?, {0x2218b88, 0xc000074800}, 0xc00007a300)
        /restic/internal/archiver/tree_saver.go:156 +0x1b9 fp=0xc000cdbf48 sp=0xc000cdbd40 pc=0x1592319
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x29 fp=0xc000cdbf78 sp=0xc000cdbf48 pc=0x1591209
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000cdbfe0 sp=0xc000cdbf78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000cdbfe8 sp=0xc000cdbfe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 141 [select]:
runtime.gopark(0xc000ce5e10?, 0x2?, 0x8b?, 0x8f?, 0xc000ce5d84?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000ce5c00 sp=0xc000ce5be0 pc=0x116b736
runtime.selectgo(0xc000ce5e10, 0xc000ce5d80, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000ce5d40 sp=0xc000ce5c00 pc=0x1179adc
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x0?, {0x2218b88, 0xc000074800}, 0xc00007a300)
        /restic/internal/archiver/tree_saver.go:156 +0x1b9 fp=0xc000ce5f48 sp=0xc000ce5d40 pc=0x1592319
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x29 fp=0xc000ce5f78 sp=0xc000ce5f48 pc=0x1591209
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000ce5fe0 sp=0xc000ce5f78 pc=0x14cffa4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000ce5fe8 sp=0xc000ce5fe0 pc=0x11970a1
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5
PS C:\restic>

@prattmic
Copy link
Member

@prattmic Is it possible to/can you provide instructions to build restic with Cherry's patch?

$ gotip download 612535
$ gotip install github.com/restic/restic/cmd/restic@v0.17.1

Or see the more complicated final command in #67108 (comment) if this simple one doesn't work for some reason (just replace go with gotip).

@distancesprinter
Copy link

@prattmic Sad news.

After reboot, gotip version with Cherry's patch threw the following:

Microsoft Windows [Version 10.0.17763.6293]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\Windows\system32>gotip version
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x9d63b9?, 0xc0000f0c38?})
        C:/Users/[username]/sdk/gotip/src/runtime/panic.go:1069 +0x4d fp=0xc0000c1c30 sp=0xc0000c1c00 pc=0x321b6d
runtime.badmcall(0x3df42c?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:550 +0x1f fp=0xc0000c1c50 sp=0xc0000c1c30 pc=0x2edd7f
runtime.badmcall(0x2ebf39)
        <autogenerated>:1 +0x25 fp=0xc0000c1c68 sp=0xc0000c1c50 pc=0x32dee5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc0000c1c80
stack: frame={sp:0xc0000c1c50, fp:0xc0000c1c68} stack=[0x34ea04000,0x34ebffda0)


goroutine 1 gp=0xc0000021c0 m=nil [runnable, locked to thread]:
cmd/go/internal/toolchain.init()
        <autogenerated>:1 +0x199 fp=0xc0000c1e20 sp=0xc0000c1e18 pc=0x85b899
runtime.doInit1(0xe69d70)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:7303 +0xec fp=0xc0000c1f50 sp=0xc0000c1e20 pc=0x2fb8cc
runtime.doInit(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:7270
runtime.main()
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:254 +0x325 fp=0xc0000c1fe0 sp=0xc0000c1f50 pc=0x2ed445
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000c1fe8 sp=0xc0000c1fe0 pc=0x329161

goroutine 2 gp=0xc0000028c0 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc00004dfa8 sp=0xc00004df88 pc=0x321c8e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.forcegchelper()
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:348 +0xb8 fp=0xc00004dfe0 sp=0xc00004dfa8 pc=0x2ed6b8
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00004dfe8 sp=0xc00004dfe0 pc=0x329161
created by runtime.init.7 in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:336 +0x1a

goroutine 3 gp=0xc000002c40 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc00004ff80 sp=0xc00004ff60 pc=0x321c8e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.bgsweep(0xc00005a000)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcsweep.go:277 +0x94 fp=0xc00004ffc8 sp=0xc00004ff80 pc=0x2d6754
runtime.gcenable.gowrap1()
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:203 +0x25 fp=0xc00004ffe0 sp=0xc00004ffc8 pc=0x2cade5
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00004ffe8 sp=0xc00004ffe0 pc=0x329161
created by runtime.gcenable in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000002e00 m=nil [GC scavenge wait]:
runtime.gopark(0xc00005a000?, 0xada050?, 0x1?, 0x0?, 0xc000002e00?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc000061f78 sp=0xc000061f58 pc=0x321c8e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.(*scavengerState).park(0xec5d20)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000061fa8 sp=0xc000061f78 pc=0x2d41e9
runtime.bgscavenge(0xc00005a000)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000061fc8 sp=0xc000061fa8 pc=0x2d475c
runtime.gcenable.gowrap2()
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:204 +0x25 fp=0xc000061fe0 sp=0xc000061fc8 pc=0x2cad85
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0x329161
created by runtime.gcenable in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000003180 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc000063e20 sp=0xc000063e00 pc=0x321c8e
runtime.runfinq()
        C:/Users/[username]/sdk/gotip/src/runtime/mfinal.go:193 +0x107 fp=0xc000063fe0 sp=0xc000063e20 pc=0x2c9ea7
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x329161
created by runtime.createfing in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mfinal.go:163 +0x3d

goroutine 18 gp=0xc0000861c0 m=nil [chan receive]:
runtime.gopark(0x0?, 0x2cad01?, 0x28?, 0xb0?, 0xe6b028?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc000051f18 sp=0xc000051ef8 pc=0x321c8e
runtime.chanrecv(0xc0000aa540, 0x0, 0x1)
        C:/Users/[username]/sdk/gotip/src/runtime/chan.go:639 +0x40b fp=0xc000051f90 sp=0xc000051f18 pc=0x2b9e2b
runtime.chanrecv1(0x2ed500?, 0xc000051f76?)
        C:/Users/[username]/sdk/gotip/src/runtime/chan.go:489 +0x12 fp=0xc000051fb8 sp=0xc000051f90 pc=0x2b99f2
runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:1731
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:1734 +0x2f fp=0xc000051fe0 sp=0xc000051fb8 pc=0x2cde8f
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000051fe8 sp=0xc000051fe0 pc=0x329161
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:1729 +0x96

C:\Windows\system32>gotip version
go version devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 windows/amd64

C:\Windows\system32>

@distancesprinter
Copy link

I've been sitting here rebooting and trying to generate a bunch of these. Sometimes things seem really stable--watched 10,000 iterations of gotip version without an error. Then, I'll reboot and immediate failure. Once the error is thrown, things seem pretty stable for a while. Pardon the imprecise language, I can't make any sense of what I'm seeing.

morestack on g0:

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\Windows\system32> gotip version
fatal: morestack on g0
Exception 0x80000003 0x0 0x0 0x198fc1
PC=0x198fc1

runtime.abort()
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1162 +0x1 fp=0xc000035c40 sp=0xc000035c38 pc=0x198fc1
runtime.morestack()
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:598 +0x3d fp=0xc000035c48 sp=0xc000035c40 pc=0x19723d

goroutine 1 gp=0xc0000021c0 m=nil [runnable, locked to thread]:
cmd/vendor/golang.org/x/telemetry/internal/telemetry.init.0()
        C:/Users/[username]/sdk/gotip/src/cmd/vendor/golang.org/x/telemetry/internal/telemetry/dir.go:50 +0xea fp=0xc000035e20 sp=0xc000035e18 pc=0x52a14a
runtime.doInit1(0xcda1b0)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:7303 +0xec fp=0xc000035f50 sp=0xc000035e20 pc=0x16b8cc
runtime.doInit(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:7270
runtime.main()
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:254 +0x325 fp=0xc000035fe0 sp=0xc000035f50 pc=0x15d445
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000035fe8 sp=0xc000035fe0 pc=0x199161

goroutine 2 gp=0xc0000028c0 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc00004dfa8 sp=0xc00004df88 pc=0x191c8e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.forcegchelper()
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:348 +0xb8 fp=0xc00004dfe0 sp=0xc00004dfa8 pc=0x15d6b8
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00004dfe8 sp=0xc00004dfe0 pc=0x199161
created by runtime.init.7 in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:336 +0x1a

goroutine 3 gp=0xc000002c40 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc00004ff80 sp=0xc00004ff60 pc=0x191c8e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.bgsweep(0xc00005a000)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcsweep.go:277 +0x94 fp=0xc00004ffc8 sp=0xc00004ff80 pc=0x146754
runtime.gcenable.gowrap1()
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:203 +0x25 fp=0xc00004ffe0 sp=0xc00004ffc8 pc=0x13ade5
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00004ffe8 sp=0xc00004ffe0 pc=0x199161
created by runtime.gcenable in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000002e00 m=nil [GC scavenge wait]:
runtime.gopark(0xc00005a000?, 0x94a050?, 0x1?, 0x0?, 0xc000002e00?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc000061f78 sp=0xc000061f58 pc=0x191c8e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.(*scavengerState).park(0xd35d20)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000061fa8 sp=0xc000061f78 pc=0x1441e9
runtime.bgscavenge(0xc00005a000)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000061fc8 sp=0xc000061fa8 pc=0x14475c
runtime.gcenable.gowrap2()
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:204 +0x25 fp=0xc000061fe0 sp=0xc000061fc8 pc=0x13ad85
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0x199161
created by runtime.gcenable in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000003180 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc000063e20 sp=0xc000063e00 pc=0x191c8e
runtime.runfinq()
        C:/Users/[username]/sdk/gotip/src/runtime/mfinal.go:193 +0x107 fp=0xc000063fe0 sp=0xc000063e20 pc=0x139ea7
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x199161
created by runtime.createfing in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mfinal.go:163 +0x3d
rax     0x17
rbx     0xd383b8
rcx     0xd380c0
rdx     0x0
rdi     0xcada814000
rsi     0xd380c0
rbp     0xc000035c58
rsp     0xc000035c38
r8      0xc0000358e8
r9      0xc000035ac0
r10     0x0
r11     0x246
r12     0xcadabffd38
r13     0xa8
r14     0xd363a0
r15     0x5
rip     0x198fc1
rflags  0x206
cs      0x33
fs      0x53
gs      0x2b
PS C:\Windows\system32>

unexpected signal during runtime execution:

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\Windows\system32> gotip version
fatal error: unexpected signal during runtime execution
[signal 0xc0000005 code=0x0 addr=0x90 pc=0xd9c040]

runtime stack:
runtime.throw({0x148b894?, 0x4?})
        C:/Users/[username]/sdk/gotip/src/runtime/panic.go:1069 +0x4d fp=0xc1ca1ff598 sp=0xc1ca1ff568 pc=0xdd1b6d
runtime.sigpanic()
        C:/Users/[username]/sdk/gotip/src/runtime/signal_windows.go:395 +0x265 fp=0xc1ca1ff5e0 sp=0xc1ca1ff598 pc=0xdb1385
runtime.isAsyncSafePoint(0x7ffa890bf470?, 0x164?, 0xc1ca1ff690?, 0x1?)
        C:/Users/[username]/sdk/gotip/src/runtime/preempt.go:369 +0x20 fp=0xc1ca1ff640 sp=0xc1ca1ff5e0 pc=0xd9c040
runtime.preemptM(0xd6f276?)
        C:/Users/[username]/sdk/gotip/src/runtime/os_windows.go:1304 +0x2cd fp=0xc1ca1ffba0 sp=0xc1ca1ff640 pc=0xd9800d
runtime.preemptone(0x7ffa890c2570?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:6338 +0x55 fp=0xc1ca1ffbb8 sp=0xc1ca1ffba0 pc=0xdaa075
runtime.retake(0x88d1bd86c)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:6242 +0xb6 fp=0xc1ca1ffc18 sp=0xc1ca1ffbb8 pc=0xda9d56
runtime.sysmon()
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:6181 +0x39a fp=0xc1ca1ffc98 sp=0xc1ca1ffc18 pc=0xda9aba
runtime.mstart1()
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:1845 +0x9d fp=0xc1ca1ffcc0 sp=0xc1ca1ffc98 pc=0xda079d
runtime.mstart0()
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:1802 +0x6a fp=0xc1ca1ffce8 sp=0xc1ca1ffcc0 pc=0xda06ea
runtime.mstart()
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:395 +0x5 fp=0xc1ca1ffcf0 sp=0xc1ca1ffce8 pc=0xdd70a5

goroutine 1 gp=0xc0000021c0 m=nil [runnable, locked to thread]:
runtime.slicebytetostring(0x0?, 0xc000143c43?, 0x5?)
        C:/Users/[username]/sdk/gotip/src/runtime/string.go:124 +0xe5 fp=0xc000143c10 sp=0xc000143c08 pc=0xdb56e5
time.Duration.String(...)
        C:/Users/[username]/sdk/gotip/src/time/time.go:948
flag.(*durationValue).String(0x146ab8f?)
        C:/Users/[username]/sdk/gotip/src/flag/flag.go:296 +0x4d fp=0xc000143c58 sp=0xc000143c10 pc=0xe8c3cd
flag.(*FlagSet).Var(0xc000143d40, {0x1592b68, 0x19bd348}, {0x146ab8f, 0x7}, {0x0, 0x0})
        C:/Users/[username]/sdk/gotip/src/flag/flag.go:1019 +0xc4 fp=0xc000143cf0 sp=0xc000143c58 pc=0xe8f2a4
flag.(*FlagSet).DurationVar(...)
        C:/Users/[username]/sdk/gotip/src/flag/flag.go:932
cmd/go/internal/test.init.1()
        C:/Users/[username]/sdk/gotip/src/cmd/go/internal/test/testflag.go:64 +0x605 fp=0xc000143e20 sp=0xc000143cf0 pc=0x133c365
runtime.doInit1(0x191b040)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:7303 +0xec fp=0xc000143f50 sp=0xc000143e20 pc=0xdab8cc
runtime.doInit(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:7270
runtime.main()
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:254 +0x325 fp=0xc000143fe0 sp=0xc000143f50 pc=0xd9d445
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000143fe8 sp=0xc000143fe0 pc=0xdd9161

goroutine 2 gp=0xc0000028c0 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc00004dfa8 sp=0xc00004df88 pc=0xdd1c8e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.forcegchelper()
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:348 +0xb8 fp=0xc00004dfe0 sp=0xc00004dfa8 pc=0xd9d6b8
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00004dfe8 sp=0xc00004dfe0 pc=0xdd9161
created by runtime.init.7 in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:336 +0x1a

goroutine 3 gp=0xc000002c40 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc00004ff80 sp=0xc00004ff60 pc=0xdd1c8e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.bgsweep(0xc00005a000)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcsweep.go:277 +0x94 fp=0xc00004ffc8 sp=0xc00004ff80 pc=0xd86754
runtime.gcenable.gowrap1()
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:203 +0x25 fp=0xc00004ffe0 sp=0xc00004ffc8 pc=0xd7ade5
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00004ffe8 sp=0xc00004ffe0 pc=0xdd9161
created by runtime.gcenable in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000002e00 m=nil [GC scavenge wait]:
runtime.gopark(0xc00005a000?, 0x158a050?, 0x1?, 0x0?, 0xc000002e00?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc000061f78 sp=0xc000061f58 pc=0xdd1c8e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.(*scavengerState).park(0x1975d20)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000061fa8 sp=0xc000061f78 pc=0xd841e9
runtime.bgscavenge(0xc00005a000)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000061fc8 sp=0xc000061fa8 pc=0xd8475c
runtime.gcenable.gowrap2()
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:204 +0x25 fp=0xc000061fe0 sp=0xc000061fc8 pc=0xd7ad85
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0xdd9161
created by runtime.gcenable in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:204 +0xa5

goroutine 18 gp=0xc0000861c0 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc00005de20 sp=0xc00005de00 pc=0xdd1c8e
runtime.runfinq()
        C:/Users/[username]/sdk/gotip/src/runtime/mfinal.go:193 +0x107 fp=0xc00005dfe0 sp=0xc00005de20 pc=0xd79ea7
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00005dfe8 sp=0xc00005dfe0 pc=0xdd9161
created by runtime.createfing in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mfinal.go:163 +0x3d

goroutine 34 gp=0xc000130000 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc000137f18 sp=0xc000137ef8 pc=0xdd1c8e
runtime.chanrecv(0xc0001225b0, 0x0, 0x1)
        C:/Users/[username]/sdk/gotip/src/runtime/chan.go:639 +0x40b fp=0xc000137f90 sp=0xc000137f18 pc=0xd69e2b
runtime.chanrecv1(0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/chan.go:489 +0x12 fp=0xc000137fb8 sp=0xc000137f90 pc=0xd699f2
runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:1731
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:1734 +0x2f fp=0xc000137fe0 sp=0xc000137fb8 pc=0xd7de8f
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000137fe8 sp=0xc000137fe0 pc=0xdd9161
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:1729 +0x96
PS C:\Windows\system32>

@distancesprinter
Copy link

Several more today. Output below.

As I've stated, I am of course running bhyve and would be happy to make an attempt to reach out to that community if there is evidence that this is a VM issue, but we haven't seen any other issues running Windows 10 or Windows Server 2019 on bhyve on FreeBSD for as long as we've done it.

Do any of the Go engineers have access to Zen2/Zen3 hardware to see if you can reproduce on metal? I don't have that hardware but part of me wants to build a PC just to try to reproduce this for myself to rule out bhyve.

What are the chances this is an AMD firmware or hardware issue? Also haven't had any trouble with FreeBSD. Extremely stable. I only reboot these boxes when there is a kernel update.

Several panics; software eventually ran successfully after repeatedly invoking it:

PS C:\restic> .\backup_prod.cmd
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x19c67d9?, 0xc0001010c0?})
        C:/Users/[username]/sdk/gotip/src/runtime/panic.go:1069 +0x4d fp=0xc000051aa0 sp=0xc000051a70 pc=0xb907ed
runtime.badmcall(0x2457120?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:550 +0x1f fp=0xc000051ac0 sp=0xc000051aa0 pc=0xb5e1df
runtime.badmcall(0xb5c399)
        <autogenerated>:1 +0x25 fp=0xc000051ad8 sp=0xc000051ac0 pc=0xb9cd25
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc000051af0
stack: frame={sp:0xc000051ac0, fp:0xc000051ad8} stack=[0x7fd404000,0x7fd5ffcc0)


goroutine 1 gp=0xc0000021c0 m=nil [runnable, locked to thread]:
google.golang.org/protobuf/internal/impl.(*MessageInfo).initOnce(0xc0001719a8)
        C:/Users/[username]/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/message.go:76 +0x2c5 fp=0xc000051c90 sp=0xc000051c88 pc=0x13d3e45
google.golang.org/protobuf/internal/impl.(*MessageInfo).init(...)
        C:/Users/[username]/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/message.go:72
google.golang.org/protobuf/internal/impl.(*messageState).ProtoMethods(0x1b7ec80?)
        C:/Users/[username]/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/message_reflect_gen.go:31 +0x2e fp=0xc000051cb8 sp=0xc000051c90 pc=0x13de8ee
google.golang.org/protobuf/proto.protoMethods(...)
        C:/Users/[username]/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/proto_methods.go:19
google.golang.org/protobuf/proto.UnmarshalOptions.unmarshal({{}, 0x1, 0x1, 0x0, {0x1b832d8, 0xc000047590}, 0x2710}, {0x2376300, 0x5d, 0x5d}, ...)
        C:/Users/[username]/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/decode.go:95 +0xe2 fp=0xc000051d70 sp=0xc000051cb8 pc=0x136a142
google.golang.org/protobuf/proto.Unmarshal({0x2376300, 0x5d, 0x5d}, {0x1b7ec80?, 0x240e080?})
        C:/Users/[username]/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/decode.go:57 +0x5d fp=0xc000051dc8 sp=0xc000051d70 pc=0x1369dbd
google.golang.org/protobuf/reflect/protodesc.init.0()
        C:/Users/[username]/go/pkg/mod/google.golang.org/protobuf@v1.34.2/reflect/protodesc/editions.go:25 +0x3a fp=0xc000051e20 sp=0xc000051dc8 pc=0x14b969a
runtime.doInit1(0x23729d0)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:7303 +0xec fp=0xc000051f50 sp=0xc000051e20 pc=0xb6bd2c
runtime.doInit(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:7270
runtime.main()
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:254 +0x325 fp=0xc000051fe0 sp=0xc000051f50 pc=0xb5d8a5
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000051fe8 sp=0xc000051fe0 pc=0xb97fa1

goroutine 2 gp=0xc0000028c0 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0xb9090e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.forcegchelper()
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:348 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0xb5db18
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0xb97fa1
created by runtime.init.7 in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:336 +0x1a

goroutine 18 gp=0xc0000861c0 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc000091f80 sp=0xc000091f60 pc=0xb9090e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.bgsweep(0xc000088000)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcsweep.go:277 +0x94 fp=0xc000091fc8 sp=0xc000091f80 pc=0xb46a54
runtime.gcenable.gowrap1()
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:203 +0x25 fp=0xc000091fe0 sp=0xc000091fc8 pc=0xb3b045
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000091fe8 sp=0xc000091fe0 pc=0xb97fa1
created by runtime.gcenable in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:203 +0x66

goroutine 19 gp=0xc000086380 m=nil [GC scavenge wait]:
runtime.gopark(0xc000088000?, 0x1b70730?, 0x1?, 0x0?, 0xc000086380?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc000093f78 sp=0xc000093f58 pc=0xb9090e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.(*scavengerState).park(0x240e6e0)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000093fa8 sp=0xc000093f78 pc=0xb444e9
runtime.bgscavenge(0xc000088000)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000093fc8 sp=0xc000093fa8 pc=0xb44a5c
runtime.gcenable.gowrap2()
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:204 +0x25 fp=0xc000093fe0 sp=0xc000093fc8 pc=0xb3afe5
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000093fe8 sp=0xc000093fe0 pc=0xb97fa1
created by runtime.gcenable in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:204 +0xa5

goroutine 3 gp=0xc000002e00 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc00006be20 sp=0xc00006be00 pc=0xb9090e
runtime.runfinq()
        C:/Users/[username]/sdk/gotip/src/runtime/mfinal.go:193 +0x107 fp=0xc00006bfe0 sp=0xc00006be20 pc=0xb3a107
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xb97fa1
created by runtime.createfing in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mfinal.go:163 +0x3d
PS C:\restic> .\backup_prod.cmd
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x19c67d9?, 0x10?})
        C:/Users/[username]/sdk/gotip/src/runtime/panic.go:1069 +0x4d fp=0xc00029fc30 sp=0xc00029fc00 pc=0xb907ed
runtime.badmcall(0x10?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:550 +0x1f fp=0xc00029fc50 sp=0xc00029fc30 pc=0xb5e1df
runtime.badmcall(0xb5c399)
        <autogenerated>:1 +0x25 fp=0xc00029fc68 sp=0xc00029fc50 pc=0xb9cd25
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc00029fc80
stack: frame={sp:0xc00029fc50, fp:0xc00029fc68} stack=[0x7fcd604000,0x7fcd7ffd90)


goroutine 1 gp=0xc0000021c0 m=nil [runnable, locked to thread]:
vendor/golang.org/x/net/idna.init()
        <autogenerated>:1 +0x97 fp=0xc00029fe20 sp=0xc00029fe18 pc=0xdfcd17
runtime.doInit1(0x23710e0)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:7303 +0xec fp=0xc00029ff50 sp=0xc00029fe20 pc=0xb6bd2c
runtime.doInit(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:7270
runtime.main()
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:254 +0x325 fp=0xc00029ffe0 sp=0xc00029ff50 pc=0xb5d8a5
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00029ffe8 sp=0xc00029ffe0 pc=0xb97fa1

goroutine 2 gp=0xc0000028c0 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0xb9090e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.forcegchelper()
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:348 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0xb5db18
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0xb97fa1
created by runtime.init.7 in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:336 +0x1a

goroutine 3 gp=0xc000002c40 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0xb9090e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.bgsweep(0xc000076000)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcsweep.go:277 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0xb46a54
runtime.gcenable.gowrap1()
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0xb3b045
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xb97fa1
created by runtime.gcenable in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000002e00 m=nil [GC scavenge wait]:
runtime.gopark(0xc000076000?, 0x1b70730?, 0x1?, 0x0?, 0xc000002e00?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc00007df78 sp=0xc00007df58 pc=0xb9090e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.(*scavengerState).park(0x240e6e0)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007dfa8 sp=0xc00007df78 pc=0xb444e9
runtime.bgscavenge(0xc000076000)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00007dfc8 sp=0xc00007dfa8 pc=0xb44a5c
runtime.gcenable.gowrap2()
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:204 +0x25 fp=0xc00007dfe0 sp=0xc00007dfc8 pc=0xb3afe5
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007dfe8 sp=0xc00007dfe0 pc=0xb97fa1
created by runtime.gcenable in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:204 +0xa5

goroutine 18 gp=0xc0001061c0 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc000079e20 sp=0xc000079e00 pc=0xb9090e
runtime.runfinq()
        C:/Users/[username]/sdk/gotip/src/runtime/mfinal.go:193 +0x107 fp=0xc000079fe0 sp=0xc000079e20 pc=0xb3a107
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0xb97fa1
created by runtime.createfing in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mfinal.go:163 +0x3d

goroutine 5 gp=0xc000003340 m=nil [select]:
runtime.gopark(0xc00007ff78?, 0x3?, 0x90?, 0x0?, 0xc00007ff72?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc00007fe10 sp=0xc00007fdf0 pc=0xb9090e
runtime.selectgo(0xc00007ff78, 0xc00007ff6c, 0xc000086a00?, 0x0, 0x0?, 0x1)
        C:/Users/[username]/sdk/gotip/src/runtime/select.go:335 +0x79a fp=0xc00007ff38 sp=0xc00007fe10 pc=0xb6eb1a
go.opencensus.io/stats/view.(*worker).start(0xc000086a00)
        C:/Users/[username]/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f fp=0xc00007ffc8 sp=0xc00007ff38 pc=0x15395bf
go.opencensus.io/stats/view.init.0.gowrap1()
        C:/Users/[username]/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x25 fp=0xc00007ffe0 sp=0xc00007ffc8 pc=0x1538925
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007ffe8 sp=0xc00007ffe0 pc=0xb97fa1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
        C:/Users/[username]/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 6 gp=0xc000003500 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc000359f18 sp=0xc000359ef8 pc=0xb9090e
runtime.chanrecv(0xc0003202a0, 0x0, 0x1)
        C:/Users/[username]/sdk/gotip/src/runtime/chan.go:639 +0x40b fp=0xc000359f90 sp=0xc000359f18 pc=0xb2a12b
runtime.chanrecv1(0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/chan.go:489 +0x12 fp=0xc000359fb8 sp=0xc000359f90 pc=0xb29cf2
runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:1731
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:1734 +0x2f fp=0xc000359fe0 sp=0xc000359fb8 pc=0xb3e0ef
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000359fe8 sp=0xc000359fe0 pc=0xb97fa1
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:1729 +0x96
PS C:\restic> .\backup_prod.cmd
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x19c67d9?, 0xc000375dd0?})
        C:/Users/[username]/sdk/gotip/src/runtime/panic.go:1069 +0x4d fp=0xc00029fc30 sp=0xc00029fc00 pc=0xb907ed
runtime.badmcall(0xc00029fca8?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:550 +0x1f fp=0xc00029fc50 sp=0xc00029fc30 pc=0xb5e1df
runtime.badmcall(0xb5c399)
        <autogenerated>:1 +0x25 fp=0xc00029fc68 sp=0xc00029fc50 pc=0xb9cd25
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc00029fc80
stack: frame={sp:0xc00029fc50, fp:0xc00029fc68} stack=[0x9f36c04000,0x9f36dffa10)


goroutine 1 gp=0xc0000021c0 m=nil [runnable, locked to thread]:
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/shared.init()
        <autogenerated>:1 +0x4b fp=0xc00029fe20 sp=0xc00029fe18 pc=0x12b71ab
runtime.doInit1(0x2371320)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:7303 +0xec fp=0xc00029ff50 sp=0xc00029fe20 pc=0xb6bd2c
runtime.doInit(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:7270
runtime.main()
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:254 +0x325 fp=0xc00029ffe0 sp=0xc00029ff50 pc=0xb5d8a5
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00029ffe8 sp=0xc00029ffe0 pc=0xb97fa1

goroutine 2 gp=0xc0000028c0 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0xb9090e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.forcegchelper()
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:348 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0xb5db18
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0xb97fa1
created by runtime.init.7 in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:336 +0x1a

goroutine 3 gp=0xc000002c40 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0xb9090e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.bgsweep(0xc000076000)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcsweep.go:277 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0xb46a54
runtime.gcenable.gowrap1()
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0xb3b045
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xb97fa1
created by runtime.gcenable in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000002e00 m=nil [GC scavenge wait]:
runtime.gopark(0xc000076000?, 0x1b70730?, 0x1?, 0x0?, 0xc000002e00?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc00007df78 sp=0xc00007df58 pc=0xb9090e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.(*scavengerState).park(0x240e6e0)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007dfa8 sp=0xc00007df78 pc=0xb444e9
runtime.bgscavenge(0xc000076000)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00007dfc8 sp=0xc00007dfa8 pc=0xb44a5c
runtime.gcenable.gowrap2()
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:204 +0x25 fp=0xc00007dfe0 sp=0xc00007dfc8 pc=0xb3afe5
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007dfe8 sp=0xc00007dfe0 pc=0xb97fa1
created by runtime.gcenable in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000003180 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc00007fe20 sp=0xc00007fe00 pc=0xb9090e
runtime.runfinq()
        C:/Users/[username]/sdk/gotip/src/runtime/mfinal.go:193 +0x107 fp=0xc00007ffe0 sp=0xc00007fe20 pc=0xb3a107
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007ffe8 sp=0xc00007ffe0 pc=0xb97fa1
created by runtime.createfing in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mfinal.go:163 +0x3d

goroutine 18 gp=0xc0001da700 m=nil [select]:
runtime.gopark(0xc000079f78?, 0x3?, 0x90?, 0x0?, 0xc000079f72?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc000079e10 sp=0xc000079df0 pc=0xb9090e
runtime.selectgo(0xc000079f78, 0xc000079f6c, 0xc000143700?, 0x0, 0x0?, 0x1)
        C:/Users/[username]/sdk/gotip/src/runtime/select.go:335 +0x79a fp=0xc000079f38 sp=0xc000079e10 pc=0xb6eb1a
go.opencensus.io/stats/view.(*worker).start(0xc000143700)
        C:/Users/[username]/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f fp=0xc000079fc8 sp=0xc000079f38 pc=0x15395bf
go.opencensus.io/stats/view.init.0.gowrap1()
        C:/Users/[username]/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x25 fp=0xc000079fe0 sp=0xc000079fc8 pc=0x1538925
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0xb97fa1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
        C:/Users/[username]/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 19 gp=0xc0001da8c0 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc00007bf18 sp=0xc00007bef8 pc=0xb9090e
runtime.chanrecv(0xc00012c380, 0x0, 0x1)
        C:/Users/[username]/sdk/gotip/src/runtime/chan.go:639 +0x40b fp=0xc00007bf90 sp=0xc00007bf18 pc=0xb2a12b
runtime.chanrecv1(0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/chan.go:489 +0x12 fp=0xc00007bfb8 sp=0xc00007bf90 pc=0xb29cf2
runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:1731
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:1734 +0x2f fp=0xc00007bfe0 sp=0xc00007bfb8 pc=0xb3e0ef
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0xb97fa1
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:1729 +0x96
PS C:\restic> .\backup_prod.cmd
fatal: morestack on g0
Exception 0x80000003 0x0 0x0 0xb97e01
PC=0xb97e01

runtime.abort()
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1162 +0x1 fp=0xc00029fc40 sp=0xc00029fc38 pc=0xb97e01
runtime.morestack()
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:598 +0x3d fp=0xc00029fc48 sp=0xc00029fc40 pc=0xb9607d

goroutine 1 gp=0xc0000021c0 m=nil [runnable, locked to thread]:
github.com/restic/restic/internal/restic.init()
        <autogenerated>:1 +0x23b fp=0xc00029fe20 sp=0xc00029fe18 pc=0xf9199b
runtime.doInit1(0x2372d20)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:7303 +0xec fp=0xc00029ff50 sp=0xc00029fe20 pc=0xb6bd2c
runtime.doInit(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:7270
runtime.main()
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:254 +0x325 fp=0xc00029ffe0 sp=0xc00029ff50 pc=0xb5d8a5
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00029ffe8 sp=0xc00029ffe0 pc=0xb97fa1

goroutine 2 gp=0xc0000028c0 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0xb9090e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.forcegchelper()
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:348 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0xb5db18
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0xb97fa1
created by runtime.init.7 in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:336 +0x1a

goroutine 3 gp=0xc000002c40 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0xb9090e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.bgsweep(0xc000076000)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcsweep.go:277 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0xb46a54
runtime.gcenable.gowrap1()
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0xb3b045
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xb97fa1
created by runtime.gcenable in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000002e00 m=nil [GC scavenge wait]:
runtime.gopark(0xc000076000?, 0x1b70730?, 0x1?, 0x0?, 0xc000002e00?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc00007df78 sp=0xc00007df58 pc=0xb9090e
runtime.goparkunlock(...)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:441
runtime.(*scavengerState).park(0x240e6e0)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007dfa8 sp=0xc00007df78 pc=0xb444e9
runtime.bgscavenge(0xc000076000)
        C:/Users/[username]/sdk/gotip/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00007dfc8 sp=0xc00007dfa8 pc=0xb44a5c
runtime.gcenable.gowrap2()
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:204 +0x25 fp=0xc00007dfe0 sp=0xc00007dfc8 pc=0xb3afe5
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007dfe8 sp=0xc00007dfe0 pc=0xb97fa1
created by runtime.gcenable in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000003180 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc00007fe20 sp=0xc00007fe00 pc=0xb9090e
runtime.runfinq()
        C:/Users/[username]/sdk/gotip/src/runtime/mfinal.go:193 +0x107 fp=0xc00007ffe0 sp=0xc00007fe20 pc=0xb3a107
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007ffe8 sp=0xc00007ffe0 pc=0xb97fa1
created by runtime.createfing in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mfinal.go:163 +0x3d

goroutine 6 gp=0xc0001b3880 m=nil [select]:
runtime.gopark(0xc00006df78?, 0x3?, 0x90?, 0x0?, 0xc00006df72?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc00006de10 sp=0xc00006ddf0 pc=0xb9090e
runtime.selectgo(0xc00006df78, 0xc00006df6c, 0xc0000a5780?, 0x0, 0x0?, 0x1)
        C:/Users/[username]/sdk/gotip/src/runtime/select.go:335 +0x79a fp=0xc00006df38 sp=0xc00006de10 pc=0xb6eb1a
go.opencensus.io/stats/view.(*worker).start(0xc0000a5780)
        C:/Users/[username]/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f fp=0xc00006dfc8 sp=0xc00006df38 pc=0x15395bf
go.opencensus.io/stats/view.init.0.gowrap1()
        C:/Users/[username]/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x25 fp=0xc00006dfe0 sp=0xc00006dfc8 pc=0x1538925
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0xb97fa1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
        C:/Users/[username]/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 7 gp=0xc0001b3a40 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc000079f18 sp=0xc000079ef8 pc=0xb9090e
runtime.chanrecv(0xc000086460, 0x0, 0x1)
        C:/Users/[username]/sdk/gotip/src/runtime/chan.go:639 +0x40b fp=0xc000079f90 sp=0xc000079f18 pc=0xb2a12b
runtime.chanrecv1(0x0?, 0x0?)
        C:/Users/[username]/sdk/gotip/src/runtime/chan.go:489 +0x12 fp=0xc000079fb8 sp=0xc000079f90 pc=0xb29cf2
runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:1731
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:1734 +0x2f fp=0xc000079fe0 sp=0xc000079fb8 pc=0xb3e0ef
runtime.goexit({})
        C:/Users/[username]/sdk/gotip/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0xb97fa1
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
        C:/Users/[username]/sdk/gotip/src/runtime/mgc.go:1729 +0x96
rax     0x17
rbx     0x2410fb8
rcx     0x2410cc0
rdx     0x0
rdi     0xed483a6000
rsi     0x2410cc0
rbp     0xc00029fc58
rsp     0xc00029fc38
r8      0xc00029f8e8
r9      0xc00029fac0
r10     0x0
r11     0x246
r12     0xed485ff758
r13     0x0
r14     0x240f860
r15     0x3fffffffffffff
rip     0xb97e01
rflags  0x206
cs      0x33
fs      0x53
gs      0x2b
PS C:\restic> .\backup_prod.cmd
open repository
fatal error: fatal error: Exception 0xc0000005 0x8 0xb 0xb
PC=0xb

runtime: g 0 gp=0x240f860: unknown pc 0xb
stack: frame={sp:0xc0004a6000, fp:0x0} stack=[0x0,0x54cd7ffbb0)

runtime: g 0 gp=0x240f860: unknown pc 0xb
stack: frame={sp:0xc0004a6000, fp:0x0} stack=[0x0,0x54cd7ffbb0)


goroutine 1 gp=0xc0000021c0 m=nil [select]:
runtime.gopark(0xc0004a5e68?, 0x2?, 0x0?, 0x0?, 0xc0004a5da4?)
        C:/Users/[username]/sdk/gotip/src/runtime/proc.go:435 +0xce fp=0xc0004a5c40 sp=0xc0004a5c20 pc=0xb9090e
runtime.selectgo(0xc0004a5e68, 0xc0004a5da0, 0x180468a02b8?, 0x0, 0x0?, 0x1)
        C:/Users/[username]/sdk/gotip/src/runtime/select.go:335 +0x79a fp=0xc0004a5d68 sp=0xc0004a5c40 pc=0xb6eb1a
net/http.(*Transport).getConn(0xc00014c300, 0xc00040e2d0, {{}, 0x0, {0xc0000e0120, 0x5}, {0xc0000a0330, 0x24}, 0x0})
        C:/Users/[username]/sdk/gotip/src/net/http/transport.go:1479 +0x4bd fp=0xc0004a5f38 sp=0xc0004a5d68 pc=0xe6a21d
net/http.(*Transport).roundTrip(0xc00014c300, 0xc0003e23c0)
        C:/Users/[username]/sdk/gotip/src/net/http/transport.go:644 +0xb34 fp=0xc0004a6118 sp=0xc0004a5f38 pc=0xe65fd4
net/http.(*Transport).RoundTrip(0xc0003e2140?, 0x1b875b0?)
        C:/Users/[username]/sdk/gotip/src/net/http/roundtrip.go:30 +0x13 fp=0xc0004a6138 sp=0xc0004a6118 pc=0xe74ed3
PS C:\restic> .\backup_prod.cmd
open repository
repository 5304de1b opened (version 2, compression level auto)
using parent snapshot 39275226
load index files
[0:00] 100.00%  37 / 37 index files loaded
fatal error: runtime: mcall called on m->g0 stack
PS C:\restic> .\backup_prod.cmd
open repository
repository 5304de1b opened (version 2, compression level auto)
using parent snapshot 39275226
load index files
[0:00] 100.00%  37 / 37 index files loaded
start scan on [K:\]
start backup on [K:\]
creating VSS snapshot for [k:\]
successfully created snapshot for [k:\]
scan finished in 19.271s: 169085 files, 55.465 GiB

Files:          20 new,   802 changed, 168263 unmodified
Dirs:            0 new,    35 changed,  2537 unmodified
Data Blobs:    815 new
Tree Blobs:     26 new
Added to the repository: 94.495 MiB (22.495 MiB stored)

processed 169085 files, 55.465 GiB in 0:37
snapshot 8e874829 saved
PS C:\restic>
PS C:\restic> .\r_prod.cmd version
restic 0.17.1 compiled with devel go1.24-68306ba416 Wed Sep 11 17:15:38 2024 -0400 on windows/amd64
PS C:\restic>

@prattmic
Copy link
Member

This bug could be at multiple different layers:

  1. Bug in Go itself
  2. Bug in Windows (kernel, presumably)
  3. Bug in VMM (Bhyve) [1]
  4. Bug in processor (either fundamental or manufacturing defect)

Honestly, with the minimal information we have, I have a hard time estimating which is most likely.

That said, Windows is a popular Go port, and Zen2/Zen3 are popular processors, so IMO something about this bug must be rare, otherwise we'd have a lot more than 2 user reports of crashes.

I don't know if any of us have a Zen2/Zen3, but GCE has both (N2D instances), so we can try to reproduce in a Windows VM there when we get a chance.

[1] @RichieSams's case would need to be a different bug in this case. Unless bare metal Windows is actually running applications/parts of itself under virtualization. I feel like I read about this a few years ago, but now can't find references. Even if so, this would be a completely different VMM.

@choyri
Copy link

choyri commented Nov 7, 2024

I have encountered a similar error over the past almost a year as follows:

Expand to view details
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0xd646dc?, 0x0?})
  runtime/panic.go:1077 +0x65 fp=0xc0007f3140 sp=0xc0007f3110 pc=0x3bd565
runtime.badmcall(0x149edde0fc8?)
  runtime/proc.go:503 +0x1f fp=0xc0007f3160 sp=0xc0007f3140 pc=0x3c087f
runtime.badmcall(0x3be8d9)
  <autogenerated>:1 +0x25 fp=0xc0007f3178 sp=0xc0007f3160 pc=0x3f6765
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc0007f3190
stack: frame={sp:0xc0007f3160, fp:0xc0007f3178} stack=[0xa87804000,0xa879ffd20)

goroutine 16 [running]:
crypto/x509.(*CertPool).AddCert(0xc00098a2d0?, 0xc000888000?)
  crypto/x509/cert_pool.go:169 +0x308 fp=0xc0007f3330 sp=0xc0007f3328 pc=0x57a848
crypto/tls.(*Conn).verifyServerCertificate(0xc000154a80, {0xc0007f8090, 0x2, 0x2})
  crypto/tls/handshake_client.go:990 +0x7ad fp=0xc0007f35d0 sp=0xc0007f3330 pc=0x5b9f4d
crypto/tls.(*clientHandshakeState).doFullHandshake(0xc0005fa750)
  crypto/tls/handshake_client.go:576 +0x272 fp=0xc0007f3858 sp=0xc0007f35d0 pc=0x5b76f2
crypto/tls.(*clientHandshakeState).handshake(0xc0005fa750)
  crypto/tls/handshake_client.go:493 +0x3a6 fp=0xc0007f3ab0 sp=0xc0007f3858 pc=0x5b6ae6
crypto/tls.(*Conn).clientHandshake(0xc000154a80, {0xeade68?, 0xc000623900})
  crypto/tls/handshake_client.go:274 +0x698 fp=0xc0007f3ce0 sp=0xc0007f3ab0 pc=0x5b55f8
crypto/tls.(*Conn).clientHandshake-fm({0xeade68?, 0xc000623900?})
  <autogenerated>:1 +0x33 fp=0xc0007f3d08 sp=0xc0007f3ce0 pc=0x5eb433
crypto/tls.(*Conn).handshakeContext(0xc000154a80, {0xeaded8, 0xc0005c0c40})
  crypto/tls/conn.go:1552 +0x3d3 fp=0xc0007f3f80 sp=0xc0007f3d08 pc=0x5b3053
crypto/tls.(*Conn).HandshakeContext(...)
  crypto/tls/conn.go:1492
net/http.(*persistConn).addTLS.func2()
  net/http/transport.go:1555 +0x6d fp=0xc0007f3fe0 sp=0xc0007f3f80 pc=0x67b44d
runtime.goexit()
  runtime/asm_amd64.s:1650 +0x1 fp=0xc0007f3fe8 sp=0xc0007f3fe0 pc=0x3f1a21
created by net/http.(*persistConn).addTLS in goroutine 12
  net/http/transport.go:1551 +0x31b

<omit...>
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x28 pc=0xa65dd1]

goroutine 99 [running]:
net.(*Resolver).lookupIP.func1()
  net/lookup_windows.go:151 +0x591
net.(*Resolver).lookupIP.func2()
  net/lookup_windows.go:177 +0x23
created by net.(*Resolver).lookupIP in goroutine 98
  net/lookup_windows.go:176 +0x187
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x55 pc=0x7ff68b8316e0]

goroutine 1 [running]:
google.golang.org/protobuf/internal/impl.pointer.String(...)
  google.golang.org/protobuf/internal/impl/pointer_unsafe.go:109
google.golang.org/protobuf/internal/impl.sizeStringNoZero({0x7ff68e568a58?}, 0x0?, {0x0?})
  google.golang.org/protobuf/internal/impl/codec_gen.go:5079
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc0001732a0, {0x7ff68e568a58?}, {0x0?})
  google.golang.org/protobuf/internal/impl/encode.go:79 +0xd9
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0xc000332900?, {0xc000332990?}, {0x50?})
  google.golang.org/protobuf/internal/impl/encode.go:56 +0x76
google.golang.org/protobuf/internal/impl.(*MessageInfo).size(0x7ff68b85e2ef?, {{}, {0x7ff68e563250?, 0xc000694320?}, 0xa0?})
  google.golang.org/protobuf/internal/impl/encode.go:40 +0x59
google.golang.org/protobuf/proto.MarshalOptions.marshal({{}, 0xc0?, 0x0, 0x0}, {0x0, 0x0, 0x0}, {0x7ff68e563250, 0xc000694320})
  google.golang.org/protobuf/proto/encode.go:184 +0x13b

<omit...>

My application is distributed across at least hundreds of Windows, macOS, and Linux machines, with potentially at least a few hundred runs daily. After analyzing the few errors that occurred, they all happened on the Windows + AMD CPU platform. I feel helpless because, other than retrying, I have no idea how to resolve this low-probability but occasionally occurring issue.

@RichieSams
Copy link
Author

#56774 (comment) reported a fix with 1.21.3. However, for me, this error is still happening as of 1.22.2.

@RichieSams
Copy link
Author

I can try updating to latest and see if we still reproduce

@distancesprinter
Copy link

distancesprinter commented Dec 17, 2024 via email

@distancesprinter
Copy link

That said, Windows is a popular Go port, and Zen2/Zen3 are popular processors, so IMO something about this bug must be rare, otherwise we'd have a lot more than 2 user reports of crashes.

I don't know if any of us have a Zen2/Zen3, but GCE has both (N2D instances), so we can try to reproduce in a Windows VM there when we get a chance.

@prattmic I attempted to test in GCP, --zone=us-east5-c --machine-type=n2d-standard-2 and was not able to reproduce.

However, this is of course using GCP's special SKU: AMD EPYC 7B13 (2 CPUs)

Whereas my hardware is: AMD EPYC 7543P 32-Core Processor (4 CPUs)

Same OS except that GCP is Datacenter and mine is Standard.

System Information
------------------
      Time of this report: 9/16/2024, 15:53:45
             Machine name: INSTANCE-202409
               Machine Id: {FC5A69E8-D396-4D62-BC47-1A0426AF51EA}
         Operating System: Windows Server 2019 Datacenter 64-bit (10.0, Build 17763) (17763.rs5_release.180914-1434)
                 Language: English (Regional Setting: English)
      System Manufacturer: Google
             System Model: Google Compute Engine
                     BIOS: Google (type: UEFI)
                Processor: AMD EPYC 7B13 (2 CPUs), ~2.5GHz
                   Memory: 8192MB RAM
      Available OS Memory: 8188MB RAM
                Page File: 1661MB used, 7551MB available
              Windows Dir: C:\Windows
          DirectX Version: DirectX 12
      DX Setup Parameters: Not found
         User DPI Setting: 96 DPI (100 percent)
       System DPI Setting: 96 DPI (100 percent)
          DWM DPI Scaling: Disabled
                 Miracast: Not Available
Microsoft Graphics Hybrid: Not Supported
           DxDiag Version: 10.00.17763.3113 64bit Unicode

I was thinking of hiring a consultant to try to track down whether this is a bug in bhyve, but I am reluctant since others seem to be reporting this as occurring on metal.

@choyri Can you confirm this is happening on physical machines? What are your processor SKUs and Windows versions? Go versions?

My other thought was to see if I could reproduce in Server 2022. I would be happy to work with a qualified engineer who wants to debug remotely. I would need to supervise, but could set up a meeting to screen share.

Thanks

@choyri
Copy link

choyri commented