cmd/compile: seemingly stuck writing telemetry

[dominikh]

Go version

go version devel go1.23-82c371a307 Fri Jun 28 19:59:35 2024 +0000 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/dominikh/.cache/go-build'
GOENV='/home/dominikh/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/dominikh/prj/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/dominikh/prj'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/dominikh/prj/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/dominikh/prj/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='devel go1.23-82c371a307 Fri Jun 28 19:59:35 2024 +0000'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/dominikh/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/dominikh/prj/src/honnef.co/go/wgpu-demo/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/nix-shell.KkNJRp/go-build3177551663=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Run go tool compile -help

What did you see happen?

The process never terminates and shows high CPU usage, most of which is spent in syscalls.. strace -ff shows a rapid and endless stream of

[pid 701400] openat(AT_FDCWD, "/home/dominikh/.config/go/telemetry/local/compile@devel-devel-linux-amd64-2024-07-04.v1.count", O_RDWR|O_CREAT|O_CLOEXEC, 0666) = 6
[pid 701400] fcntl(6, F_GETFL)          = 0x8002 (flags O_RDWR|O_LARGEFILE)
[pid 701405] <... nanosleep resumed>NULL) = 0
[pid 701400] fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK|O_LARGEFILE <unfinished ...>
[pid 701405] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 701400] <... fcntl resumed>)       = 0
[pid 701400] epoll_ctl(4, EPOLL_CTL_ADD, 6, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1507866546, u64=9223363117715568562}}) = -1 EPERM (Operation not permitted)
[pid 701400] fcntl(6, F_GETFL)          = 0x8802 (flags O_RDWR|O_NONBLOCK|O_LARGEFILE)
[pid 701400] fcntl(6, F_SETFL, O_RDWR|O_LARGEFILE) = 0
[pid 701405] <... nanosleep resumed>NULL) = 0
[pid 701400] fstat(6,  <unfinished ...>
[pid 701405] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 701400] <... fstat resumed>{st_mode=S_IFREG|0644, st_size=16384, ...}) = 0
[pid 701400] fstat(6, {st_mode=S_IFREG|0644, st_size=16384, ...}) = 0
[pid 701400] mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_SHARED, 6, 0) = 0x7ffff7e20000
[pid 701400] munmap(0x7fffb121c000, 16384 <unfinished ...>
[pid 701405] <... nanosleep resumed>NULL) = 0
[pid 701400] <... munmap resumed>)      = 0
[pid 701405] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 701400] close(7)                   = 0
[pid 701400] openat(AT_FDCWD, "/home/dominikh/.config/go/telemetry/local/compile@devel-devel-linux-amd64-2024-07-04.v1.count", O_RDWR|O_CREAT|O_CLOEXEC, 0666) = 7
[pid 701400] fcntl(7, F_GETFL)          = 0x8002 (flags O_RDWR|O_LARGEFILE)
[pid 701405] <... nanosleep resumed>NULL) = 0
[pid 701400] fcntl(7, F_SETFL, O_RDWR|O_NONBLOCK|O_LARGEFILE <unfinished ...>
[pid 701405] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 701400] <... fcntl resumed>)       = 0
[pid 701400] epoll_ctl(4, EPOLL_CTL_ADD, 7, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1507866547, u64=9223363117715568563}}) = -1 EPERM (Operation not permitted)
[pid 701400] fcntl(7, F_GETFL)          = 0x8802 (flags O_RDWR|O_NONBLOCK|O_LARGEFILE)
[pid 701400] fcntl(7, F_SETFL, O_RDWR|O_LARGEFILE) = 0
[pid 701405] <... nanosleep resumed>NULL) = 0
[pid 701400] fstat(7,  <unfinished ...>
[pid 701405] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 701400] <... fstat resumed>{st_mode=S_IFREG|0644, st_size=16384, ...}) = 0
[pid 701400] fstat(7, {st_mode=S_IFREG|0644, st_size=16384, ...}) = 0
[pid 701400] mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_SHARED, 7, 0) = 0x7fffb121c000
[pid 701400] munmap(0x7ffff7e20000, 16384 <unfinished ...>
[pid 701405] <... nanosleep resumed>NULL) = 0
[pid 701400] <... munmap resumed>)      = 0
[pid 701405] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 701400] close(6)                   = 0
[pid 701400] openat(AT_FDCWD, "/home/dominikh/.config/go/telemetry/local/compile@devel-devel-linux-amd64-2024-07-04.v1.count", O_RDWR|O_CREAT|O_CLOEXEC, 0666) = 6
[pid 701400] fcntl(6, F_GETFL)          = 0x8002 (flags O_RDWR|O_LARGEFILE)
[pid 701405] <... nanosleep resumed>NULL) = 0
[pid 701400] fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK|O_LARGEFILE <unfinished ...>
[pid 701405] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 701400] <... fcntl resumed>)       = 0
[pid 701400] epoll_ctl(4, EPOLL_CTL_ADD, 6, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1507866548, u64=9223363117715568564}}) = -1 EPERM (Operation not permitted)
[pid 701400] fcntl(6, F_GETFL)          = 0x8802 (flags O_RDWR|O_NONBLOCK|O_LARGEFILE)
[pid 701400] fcntl(6, F_SETFL, O_RDWR|O_LARGEFILE) = 0
[pid 701405] <... nanosleep resumed>NULL) = 0
[pid 701400] fstat(6,  <unfinished ...>
[pid 701405] nanosleep({tv_sec=0, tv_nsec=40000},  <unfinished ...>
[pid 701400] <... fstat resumed>{st_mode=S_IFREG|0644, st_size=16384, ...}) = 0
[pid 701400] fstat(6, {st_mode=S_IFREG|0644, st_size=16384, ...}) = 0
[pid 701400] mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_SHARED, 6, 0) = 0x7ffff7e20000
[pid 701400] munmap(0x7fffb121c000, 16384) = 0
[pid 701400] close(7 <unfinished ...>
[pid 701405] <... nanosleep resumed>NULL) = 0
[pid 701400] <... close resumed>)       = 0
[pid 701405] nanosleep({tv_sec=0, tv_nsec=80000},  <unfinished ...>
[pid 701400] openat(AT_FDCWD, "/home/dominikh/.config/go/telemetry/local/compile@devel-devel-linux-amd64-2024-07-04.v1.count", O_RDWR|O_CREAT|O_CLOEXEC, 0666) = 7
[pid 701400] fcntl(7, F_GETFL)          = 0x8002 (flags O_RDWR|O_LARGEFILE)
[pid 701400] fcntl(7, F_SETFL, O_RDWR|O_NONBLOCK|O_LARGEFILE) = 0
[pid 701400] epoll_ctl(4, EPOLL_CTL_ADD, 7, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1507866549, u64=9223363117715568565}}) = -1 EPERM (Operation not permitted)
[pid 701400] fcntl(7, F_GETFL)          = 0x8802 (flags O_RDWR|O_NONBLOCK|O_LARGEFILE)
[pid 701400] fcntl(7, F_SETFL, O_RDWR|O_LARGEFILE <unfinished ...>
[pid 701405] <... nanosleep resumed>NULL) = 0
[pid 701400] <... fcntl resumed>)       = 0
[pid 701405] nanosleep({tv_sec=0, tv_nsec=160000},  <unfinished ...>
[pid 701400] fstat(7, {st_mode=S_IFREG|0644, st_size=16384, ...}) = 0
[pid 701400] fstat(7, {st_mode=S_IFREG|0644, st_size=16384, ...}) = 0
[pid 701400] mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_SHARED, 7, 0) = 0x7fffb121c000
[pid 701400] munmap(0x7ffff7e20000, 16384) = 0
[pid 701400] close(6)                   = 0

What did you expect to see?

Not that.

[gabyhelp]

Related Issues

• cmd/compile: "Infinite" loop observed compiling capnproto2 with dev tip #38565 (closed)
• go tool compile: fork/exec /usr/local/go/pkg/tool/linux_amd64/compile: resource temporarily unavailable #50303 (closed)
• cmd/go: test gets hung #27720 (closed)
• cmd/go: go get hangs #16104 (closed)
• cmd/go: test timeouts waiting for os/exec copying goroutines #50187 (closed)
• runtime: go stuck in a dead lock between threads epoll_pwait/futex(..., FUTEX_WAKE_PRIVATE, 1) on arm64 #55120 (closed)
• cmd/compile: when cmd/compile is compiled with `-gcflags = "all=-N -l"`, the output binary panic on start #54412 (closed)
• building on illumos gets stuck occasionally #35261 (closed)
• can't run go run projectname.go #31715 (closed)
• exec.Command().Run() hang on #37498 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[dominikh]

Stacktrace after sending SIGQUIT:

PC=0x47d461 m=0 sigcode=128

goroutine 0 gp=0x1505d20 m=0 mp=0x15085c0 [idle]:
runtime.futex(0x1508700, 0x80, 0x0, 0x0, 0x0, 0x0)
	/home/dominikh/prj/go/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fffffff92d0 sp=0x7fffffff92c8 pc=0x47d461
runtime.futexsleep(0x7fffffff9348?, 0x40feb0?, 0x1fd40c782?)
	/home/dominikh/prj/go/src/runtime/os_linux.go:69 +0x30 fp=0x7fffffff9320 sp=0x7fffffff92d0 pc=0x437cd0
runtime.notesleep(0x1508700)
	/home/dominikh/prj/go/src/runtime/lock_futex.go:170 +0x87 fp=0x7fffffff9358 sp=0x7fffffff9320 pc=0x410047
runtime.mPark(...)
	/home/dominikh/prj/go/src/runtime/proc.go:1866
runtime.stopm()
	/home/dominikh/prj/go/src/runtime/proc.go:2885 +0x8c fp=0x7fffffff9388 sp=0x7fffffff9358 pc=0x44302c
runtime.findRunnable()
	/home/dominikh/prj/go/src/runtime/proc.go:3622 +0xd5c fp=0x7fffffff9500 sp=0x7fffffff9388 pc=0x444a9c
runtime.schedule()
	/home/dominikh/prj/go/src/runtime/proc.go:3995 +0xb1 fp=0x7fffffff9538 sp=0x7fffffff9500 pc=0x445b71
runtime.park_m(0xc0004a2540)
	/home/dominikh/prj/go/src/runtime/proc.go:4102 +0x1eb fp=0x7fffffff9590 sp=0x7fffffff9538 pc=0x445f8b
runtime.mcall()
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:459 +0x4e fp=0x7fffffff95a8 sp=0x7fffffff9590 pc=0x4797ce

goroutine 1 gp=0xc0000061c0 m=nil [runnable]:
syscall.Syscall(0x48, 0x7, 0x3, 0x0)
	/home/dominikh/prj/go/src/syscall/syscall_linux.go:73 +0x25 fp=0xc0001a7448 sp=0xc0001a73e8 pc=0x4c9b05
syscall.fcntl(0x4e3fa5?, 0xc000c82020?, 0xc0001a7401?)
	/home/dominikh/prj/go/src/syscall/zsyscall_linux_amd64.go:432 +0x25 fp=0xc0001a7480 sp=0xc0001a7448 pc=0x4c8025
syscall.SetNonblock(0x7, 0x0)
	/home/dominikh/prj/go/src/syscall/exec_unix.go:106 +0x27 fp=0xc0001a74a8 sp=0xc0001a7480 pc=0x4c40c7
os.newFile(0x7, {0xc00003e540, 0x5d}, 0x1, 0x0)
	/home/dominikh/prj/go/src/os/file_unix.go:238 +0x17f fp=0xc0001a74e8 sp=0xc0001a74a8 pc=0x4f1dff
os.openFileNolog({0xc00003e540, 0x5d}, 0x42, 0xc00000?)
	/home/dominikh/prj/go/src/os/file_unix.go:297 +0x192 fp=0xc0001a7580 sp=0xc0001a74e8 pc=0x4f2032
os.OpenFile({0xc00003e540, 0x5d}, 0x42, 0x1b6)
	/home/dominikh/prj/go/src/os/file.go:385 +0x3e fp=0xc0001a75b0 sp=0xc0001a7580 pc=0x4f0b3e
cmd/vendor/golang.org/x/telemetry/internal/counter.openMapped({0xc00003e540, 0x5d}, {0xc000188510, 0x8e}, 0xc000b00000)
	/home/dominikh/prj/go/src/cmd/vendor/golang.org/x/telemetry/internal/counter/file.go:395 +0xb7 fp=0xc0001a76c8 sp=0xc0001a75b0 pc=0x5265f7
cmd/vendor/golang.org/x/telemetry/internal/counter.(*mappedFile).newCounter(0xc000b00000, {0xddb98c, 0x13})
	/home/dominikh/prj/go/src/cmd/vendor/golang.org/x/telemetry/internal/counter/file.go:609 +0x205 fp=0xc0001a77d8 sp=0xc0001a76c8 pc=0x527425
cmd/vendor/golang.org/x/telemetry/internal/counter.(*file).newCounter1(0x1505800, {0xddb98c, 0x13})
	/home/dominikh/prj/go/src/cmd/vendor/golang.org/x/telemetry/internal/counter/file.go:328 +0x1a8 fp=0xc0001a7890 sp=0xc0001a77d8 pc=0x526148
cmd/vendor/golang.org/x/telemetry/internal/counter.(*file).newCounter(0x0?, {0xddb98c?, 0x0?})
	/home/dominikh/prj/go/src/cmd/vendor/golang.org/x/telemetry/internal/counter/file.go:311 +0x18 fp=0xc0001a78c0 sp=0xc0001a7890 pc=0x525f58
cmd/vendor/golang.org/x/telemetry/internal/counter.(*file).lookup(0xdf1c54?, {0xddb98c?, 0xc000113990?})
	/home/dominikh/prj/go/src/cmd/vendor/golang.org/x/telemetry/internal/counter/file.go:104 +0x2f fp=0xc0001a7910 sp=0xc0001a78c0 pc=0x524b0f
cmd/vendor/golang.org/x/telemetry/internal/counter.(*Counter).releaseLock(0xc000192780, 0xffffffff)
	/home/dominikh/prj/go/src/cmd/vendor/golang.org/x/telemetry/internal/counter/counter.go:224 +0x16b fp=0xc0001a79c0 sp=0xc0001a7910 pc=0x52400b
cmd/vendor/golang.org/x/telemetry/internal/counter.(*Counter).Add(0xc000192780, 0x1)
	/home/dominikh/prj/go/src/cmd/vendor/golang.org/x/telemetry/internal/counter/counter.go:178 +0x38a fp=0xc0001a7a68 sp=0xc0001a79c0 pc=0x523aea
cmd/vendor/golang.org/x/telemetry/internal/counter.(*Counter).Inc(...)
	/home/dominikh/prj/go/src/cmd/vendor/golang.org/x/telemetry/internal/counter/counter.go:130
cmd/vendor/golang.org/x/telemetry/counter.Inc(...)
	/home/dominikh/prj/go/src/cmd/vendor/golang.org/x/telemetry/counter/counter.go:23
cmd/internal/telemetry/counter.Inc(...)
	/home/dominikh/prj/go/src/cmd/internal/telemetry/counter/counter.go:29
cmd/compile/internal/gc.Main(0xe09ff8)
	/home/dominikh/prj/go/src/cmd/compile/internal/gc/main.go:63 +0xdd fp=0xc0001a7ef0 sp=0xc0001a7a68 pc=0xcc0a9d
main.main()
	/home/dominikh/prj/go/src/cmd/compile/main.go:57 +0xf9 fp=0xc0001a7f50 sp=0xc0001a7ef0 pc=0xceb119
runtime.main()
	/home/dominikh/prj/go/src/runtime/proc.go:272 +0x28b fp=0xc0001a7fe0 sp=0xc0001a7f50 pc=0x43e3ab
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0001a7fe8 sp=0xc0001a7fe0 pc=0x47b681

goroutine 2 gp=0xc000006700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0000c2fa8 sp=0xc0000c2f88 pc=0x473b4e
runtime.goparkunlock(...)
	/home/dominikh/prj/go/src/runtime/proc.go:430
runtime.forcegchelper()
	/home/dominikh/prj/go/src/runtime/proc.go:337 +0xb3 fp=0xc0000c2fe0 sp=0xc0000c2fa8 pc=0x43e6f3
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000c2fe8 sp=0xc0000c2fe0 pc=0x47b681
created by runtime.init.7 in goroutine 1
	/home/dominikh/prj/go/src/runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc000006c40 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0000c3780 sp=0xc0000c3760 pc=0x473b4e
runtime.goparkunlock(...)
	/home/dominikh/prj/go/src/runtime/proc.go:430
runtime.bgsweep(0xc0000e4000)
	/home/dominikh/prj/go/src/runtime/mgcsweep.go:317 +0xdf fp=0xc0000c37c8 sp=0xc0000c3780 pc=0x426d5f
runtime.gcenable.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:203 +0x25 fp=0xc0000c37e0 sp=0xc0000c37c8 pc=0x41b405
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000c37e8 sp=0xc0000c37e0 pc=0x47b681
created by runtime.gcenable in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000006e00 m=nil [GC scavenge wait]:
runtime.gopark(0x4e58ac?, 0x4bc268?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0000c3f78 sp=0xc0000c3f58 pc=0x473b4e
runtime.goparkunlock(...)
	/home/dominikh/prj/go/src/runtime/proc.go:430
runtime.(*scavengerState).park(0x1505760)
	/home/dominikh/prj/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc0000c3fa8 sp=0xc0000c3f78 pc=0x424749
runtime.bgscavenge(0xc0000e4000)
	/home/dominikh/prj/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc0000c3fc8 sp=0xc0000c3fa8 pc=0x424cd9
runtime.gcenable.gowrap2()
	/home/dominikh/prj/go/src/runtime/mgc.go:204 +0x25 fp=0xc0000c3fe0 sp=0xc0000c3fc8 pc=0x41b3a5
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000c3fe8 sp=0xc0000c3fe0 pc=0x47b681
created by runtime.gcenable in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000007340 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0xe0b8d0?, 0x10?, 0x23?, 0x1000000010?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0000c2620 sp=0xc0000c2600 pc=0x473b4e
runtime.runfinq()
	/home/dominikh/prj/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0000c27e0 sp=0xc0000c2620 pc=0x41a487
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000c27e8 sp=0xc0000c27e0 pc=0x47b681
created by runtime.createfing in goroutine 1
	/home/dominikh/prj/go/src/runtime/mfinal.go:163 +0x3d

goroutine 6 gp=0xc000007500 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0000c4738 sp=0xc0000c4718 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000c47c8 sp=0xc0000c4738 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0000c47e0 sp=0xc0000c47c8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000c47e8 sp=0xc0000c47e0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 7 gp=0xc0000076c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0000c4f38 sp=0xc0000c4f18 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000c4fc8 sp=0xc0000c4f38 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0000c4fe0 sp=0xc0000c4fc8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000c4fe8 sp=0xc0000c4fe0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 8 gp=0xc000007880 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0000c5738 sp=0xc0000c5718 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000c57c8 sp=0xc0000c5738 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0000c57e0 sp=0xc0000c57c8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000c57e8 sp=0xc0000c57e0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 9 gp=0xc000007a40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0000c5f38 sp=0xc0000c5f18 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000c5fc8 sp=0xc0000c5f38 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0000c5fe0 sp=0xc0000c5fc8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000c5fe8 sp=0xc0000c5fe0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 10 gp=0xc000007c00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0000be738 sp=0xc0000be718 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000be7c8 sp=0xc0000be738 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0000be7e0 sp=0xc0000be7c8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000be7e8 sp=0xc0000be7e0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 11 gp=0xc000007dc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0000bef38 sp=0xc0000bef18 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000befc8 sp=0xc0000bef38 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0000befe0 sp=0xc0000befc8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000befe8 sp=0xc0000befe0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 12 gp=0xc000498000 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0000bf738 sp=0xc0000bf718 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000bf7c8 sp=0xc0000bf738 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0000bf7e0 sp=0xc0000bf7c8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000bf7e8 sp=0xc0000bf7e0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 13 gp=0xc0004981c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0000bff38 sp=0xc0000bff18 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000bffc8 sp=0xc0000bff38 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0000bffe0 sp=0xc0000bffc8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000bffe8 sp=0xc0000bffe0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 14 gp=0xc000498380 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0000c0738 sp=0xc0000c0718 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000c07c8 sp=0xc0000c0738 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0000c07e0 sp=0xc0000c07c8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000c07e8 sp=0xc0000c07e0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 15 gp=0xc000498540 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0000c0f38 sp=0xc0000c0f18 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000c0fc8 sp=0xc0000c0f38 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0000c0fe0 sp=0xc0000c0fc8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000c0fe8 sp=0xc0000c0fe0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 16 gp=0xc000498700 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0000c1738 sp=0xc0000c1718 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000c17c8 sp=0xc0000c1738 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0000c17e0 sp=0xc0000c17c8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000c17e8 sp=0xc0000c17e0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 17 gp=0xc0004988c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0000c1f38 sp=0xc0000c1f18 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000c1fc8 sp=0xc0000c1f38 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0000c1fe0 sp=0xc0000c1fc8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000c1fe8 sp=0xc0000c1fe0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 18 gp=0xc000498a80 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc00049e738 sp=0xc00049e718 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc00049e7c8 sp=0xc00049e738 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00049e7e0 sp=0xc00049e7c8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00049e7e8 sp=0xc00049e7e0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 19 gp=0xc000498c40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc00049ef38 sp=0xc00049ef18 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc00049efc8 sp=0xc00049ef38 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00049efe0 sp=0xc00049efc8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00049efe8 sp=0xc00049efe0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 20 gp=0xc000498e00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc00049f738 sp=0xc00049f718 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc00049f7c8 sp=0xc00049f738 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00049f7e0 sp=0xc00049f7c8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00049f7e8 sp=0xc00049f7e0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 21 gp=0xc000498fc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc00049ff38 sp=0xc00049ff18 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc00049ffc8 sp=0xc00049ff38 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00049ffe0 sp=0xc00049ffc8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00049ffe8 sp=0xc00049ffe0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 22 gp=0xc000499180 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0004a0738 sp=0xc0004a0718 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0004a07c8 sp=0xc0004a0738 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0004a07e0 sp=0xc0004a07c8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0004a07e8 sp=0xc0004a07e0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 23 gp=0xc000499340 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0004a0f38 sp=0xc0004a0f18 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0004a0fc8 sp=0xc0004a0f38 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0004a0fe0 sp=0xc0004a0fc8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0004a0fe8 sp=0xc0004a0fe0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 24 gp=0xc000499500 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0004a1738 sp=0xc0004a1718 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0004a17c8 sp=0xc0004a1738 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0004a17e0 sp=0xc0004a17c8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0004a17e8 sp=0xc0004a17e0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 25 gp=0xc0004996c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0004a1f38 sp=0xc0004a1f18 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0004a1fc8 sp=0xc0004a1f38 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0004a1fe0 sp=0xc0004a1fc8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0004a1fe8 sp=0xc0004a1fe0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 26 gp=0xc000499880 m=nil [GC worker (idle)]:
runtime.gopark(0x48fdfa016780?, 0x3?, 0x34?, 0x5?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc00049a738 sp=0xc00049a718 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc00049a7c8 sp=0xc00049a738 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00049a7e0 sp=0xc00049a7c8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00049a7e8 sp=0xc00049a7e0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 27 gp=0xc000499a40 m=nil [GC worker (idle)]:
runtime.gopark(0x48fdfd40b712?, 0x1?, 0x5?, 0xd1?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc00049af38 sp=0xc00049af18 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc00049afc8 sp=0xc00049af38 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00049afe0 sp=0xc00049afc8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00049afe8 sp=0xc00049afe0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 28 gp=0xc000499c00 m=nil [GC worker (idle)]:
runtime.gopark(0x48fdfd40b3b4?, 0x3?, 0x7?, 0x5d?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc00049b738 sp=0xc00049b718 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc00049b7c8 sp=0xc00049b738 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00049b7e0 sp=0xc00049b7c8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00049b7e8 sp=0xc00049b7e0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 29 gp=0xc000499dc0 m=nil [GC worker (idle)]:
runtime.gopark(0x153ce60?, 0x1?, 0x61?, 0xf0?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc00049bf38 sp=0xc00049bf18 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc00049bfc8 sp=0xc00049bf38 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00049bfe0 sp=0xc00049bfc8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00049bfe8 sp=0xc00049bfe0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 30 gp=0xc0004a2000 m=nil [GC worker (idle)]:
runtime.gopark(0x48fdfd4096fa?, 0x1?, 0x35?, 0x9f?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc00049c738 sp=0xc00049c718 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc00049c7c8 sp=0xc00049c738 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00049c7e0 sp=0xc00049c7c8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00049c7e8 sp=0xc00049c7e0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 31 gp=0xc0004a21c0 m=nil [GC worker (idle)]:
runtime.gopark(0x48fc7cc3039e?, 0x1?, 0x70?, 0x49?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc00049cf38 sp=0xc00049cf18 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc00049cfc8 sp=0xc00049cf38 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00049cfe0 sp=0xc00049cfc8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00049cfe8 sp=0xc00049cfe0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 32 gp=0xc0004a2380 m=nil [GC worker (idle)]:
runtime.gopark(0x48fdfd40b039?, 0x1?, 0x39?, 0xcb?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc00049d738 sp=0xc00049d718 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc00049d7c8 sp=0xc00049d738 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00049d7e0 sp=0xc00049d7c8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00049d7e8 sp=0xc00049d7e0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 33 gp=0xc0004a2540 m=nil [GC worker (idle)]:
runtime.gopark(0x153ce60?, 0x1?, 0x67?, 0xf5?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc00049df38 sp=0xc00049df18 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc00049dfc8 sp=0xc00049df38 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00049dfe0 sp=0xc00049dfc8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00049dfe8 sp=0xc00049dfe0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 34 gp=0xc0004a2700 m=nil [GC worker (idle)]:
runtime.gopark(0x48fdfd43572a?, 0x1?, 0x81?, 0x2?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0004a8738 sp=0xc0004a8718 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0004a87c8 sp=0xc0004a8738 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0004a87e0 sp=0xc0004a87c8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0004a87e8 sp=0xc0004a87e0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 35 gp=0xc0004a28c0 m=nil [GC worker (idle)]:
runtime.gopark(0x48fdfd4096f0?, 0x1?, 0xb3?, 0x48?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0004a8f38 sp=0xc0004a8f18 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0004a8fc8 sp=0xc0004a8f38 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0004a8fe0 sp=0xc0004a8fc8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0004a8fe8 sp=0xc0004a8fe0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 36 gp=0xc0004a2a80 m=nil [GC worker (idle)]:
runtime.gopark(0x48fdfd42a521?, 0x3?, 0x5c?, 0x84?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0004a9738 sp=0xc0004a9718 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0004a97c8 sp=0xc0004a9738 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0004a97e0 sp=0xc0004a97c8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0004a97e8 sp=0xc0004a97e0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

goroutine 37 gp=0xc0004a2c40 m=nil [GC worker (idle)]:
runtime.gopark(0x153ce60?, 0x3?, 0x91?, 0x6c?, 0x0?)
	/home/dominikh/prj/go/src/runtime/proc.go:424 +0xce fp=0xc0004a9f38 sp=0xc0004a9f18 pc=0x473b4e
runtime.gcBgMarkWorker(0xc00003a620)
	/home/dominikh/prj/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0004a9fc8 sp=0xc0004a9f38 pc=0x41d729
runtime.gcBgMarkStartWorkers.gowrap1()
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0004a9fe0 sp=0xc0004a9fc8 pc=0x41d605
runtime.goexit({})
	/home/dominikh/prj/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0004a9fe8 sp=0xc0004a9fe0 pc=0x47b681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/dominikh/prj/go/src/runtime/mgc.go:1279 +0x105

rax    0xca
rbx    0x0
rcx    0x47d463
rdx    0x0
rdi    0x1508700
rsi    0x80
rbp    0x7fffffff9310
rsp    0x7fffffff92c8
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x3
r13    0x1
r14    0x1505d20
r15    0x1d
rip    0x47d461
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

[dominikh]

I don't know how to reproduce this. I've used this version of Go for quite a bit, and it suddenly started behaving this way, so I suspect it depends on some system state. It currently reproduces every time with this Go version, and doesn't reproduce with Go 1.22. I've avoided perturbing the system so it stays reproducible for me.

git bisect in go points, unhelpfully, to

commit efc347348e965f8db4f036ac0c9c7bf214b7a929
Author: Michael Matloob <matloob@golang.org>
Date:   Tue May 14 14:48:52 2024 -0400

    cmd: pull in golang.org/x/telemetry@7d78a97
    
    Commands run:
        go get golang.org/x/telemetry@7d78a97
        go mod tidy
        go mod vendor
    
    Change-Id: Iea0fbe11bcc69fac35fba0ee3d02cd14344a3ee9
    Reviewed-on: https://go-review.googlesource.com/c/go/+/585455
    LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
    Reviewed-by: Robert Findley <rfindley@google.com>

I'll bisect golang.org/x/telemetry next. I'm afraid I can't quite figure out the combination of modules and vendoring in go and how I'd bisect that dependency, especially because of transitive dependencies on newer versions of it.

[dmitshur]

That CL pulled in 7 changes from x/telemetry: https://go.googlesource.com/telemetry/+log/51e8b5d718eb..7d78a974cc44. Skipping ones that were test-only or modifying services (and not vendored to the main repo), it leaves 3 I think:

• golang/telemetry@f87de6f internal/upload: correct debug log filename construction
• golang/telemetry@15c10ce interal/upload: improve error messages
• golang/telemetry@4a0682c internal/configgen: add special handling for toolchain program versions

It should be possible to temporarily swap out the vendored x/telemetry version arbitrarily by using go mod edit -replace with a local path (pointing to intended x/telemetry commit checked out, with local edits if desired) or a pseudo-version, then go mod vendor.

[dominikh]

Aha. I tried adding the replace manually in go.mod, but either I forgot to go mod vendor afterwards, or go mod edit -replace does more work than modifying the go.mod when vendoring is in use. Either way, this works, thanks.

It bisects to

commit 4a0682cf430de51aba1059af8bd30fb06e7ac4b1 (HEAD)
Author: Rob Findley <rfindley@google.com>
Date:   Mon May 13 16:48:50 2024 +0000

    internal/configgen: add special handling for toolchain program versions

[findleyr]

Thanks very much @dominikh. I also can't reproduce this, so it seems very important that you are able to preserve your system state.

That https://go.dev/cl/585198 is implicated is probably a red herring: that CL changed the telemetry file path, by detecting when the version of toolchain programs is really the Go version. My first guess is that there's something corrupted about your counter file, and reverting to an older CL changes the path to not point to that corrupted file.

One quick test for this would be: does this reproduce today? (If it was some sort of corrupted file, it wouldn't reproduce the next day).

• Can you share ls -l ~/.config/go/telemetry/local?
• Can you please copy ~/.config/go/telemetry to a backup location? For example:

cp -R ~/.config/go/telemetry ~/.config/go/telemetry.2024-07-07

Unfortunately, since I am only seeing this a few days later, we have a ~3/7 chance that a weekly telemetry aggregation operation will have wiped away your old counter file :(. In that case, we'd have very little to go on, but could make the code in question more defensive.

CC @golang/telemetry

[dominikh]

does this reproduce today?

It does not currently reproduce; I tried at the same commit as from my initial report.

Unfortunately, since I am only seeing this a few days later, we have a ~3/7 chance that a weekly telemetry aggregation operation will have wiped away your old counter file :(.

I should have an abundance of ZFS snapshots covering that directory for the day I reported the issue.

Can you share ls -l ~/.config/go/telemetry/local?

$ ls -l ~/.config/go/telemetry/local
total 681
-rw-r--r-- 1 dominikh users 16384 Jul  3 15:38 'asm@devel-devel-linux-amd64-2024-07-03.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  8 00:43 'asm@devel-devel-linux-amd64-2024-07-07.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  7 13:54 'asm@go1.23rc1-go1.23rc1-linux-amd64-2024-07-07.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  3 17:53 'cgo@devel-devel-linux-amd64-2024-07-03.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  8 00:43 'cgo@devel-devel-linux-amd64-2024-07-07.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  7 13:54 'cgo@go1.23rc1-go1.23rc1-linux-amd64-2024-07-07.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  3 17:53 'compile@devel-devel-linux-amd64-2024-07-03.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  4 18:37 'compile@devel-devel-linux-amd64-2024-07-04.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  8 00:44 'compile@devel-devel-linux-amd64-2024-07-07.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  5 00:23  compile-devel-linux-amd64-2024-07-04.v1.count
-rw-r--r-- 1 dominikh users 16384 Jul  5 23:32  compile-devel-linux-amd64-2024-07-05.v1.count
-rw-r--r-- 1 dominikh users 16384 Jul  7 01:59  compile-devel-linux-amd64-2024-07-06.v1.count
-rw-r--r-- 1 dominikh users 16384 Jul  8 00:06  compile-devel-linux-amd64-2024-07-07.v1.count
-rw-r--r-- 1 dominikh users 16384 Jul  7 13:54 'compile@go1.23rc1-go1.23rc1-linux-amd64-2024-07-07.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  4 18:47 'doc@devel-devel-linux-amd64-2024-07-04.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  3 17:53 'go@devel-devel-linux-amd64-2024-07-03.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  4 19:06 'go@devel-devel-linux-amd64-2024-07-04.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  8 00:44 'go@devel-devel-linux-amd64-2024-07-07.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  5 00:23  go-devel-linux-amd64-2024-07-04.v1.count
-rw-r--r-- 1 dominikh users 16384 Jul  5 23:32  go-devel-linux-amd64-2024-07-05.v1.count
-rw-r--r-- 1 dominikh users 16384 Jul  7 01:59  go-devel-linux-amd64-2024-07-06.v1.count
-rw-r--r-- 1 dominikh users 16384 Jul  8 00:06  go-devel-linux-amd64-2024-07-07.v1.count
-rw-r--r-- 1 dominikh users 16384 Jul  3 17:53 'gofmt@devel-devel-linux-amd64-2024-07-03.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  4 19:06 'gofmt@devel-devel-linux-amd64-2024-07-04.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  7 13:54 'go@go1.23rc1-go1.23rc1-linux-amd64-2024-07-07.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  4 18:37 'gopls@devel-devel-linux-amd64-2024-07-03.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  6 19:57 'gopls@devel-devel-linux-amd64-2024-07-04.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  6 03:05 'gopls@devel-devel-linux-amd64-2024-07-05.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  7 14:37 'gopls@devel-devel-linux-amd64-2024-07-06.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  8 00:07 'gopls@devel-devel-linux-amd64-2024-07-07.v1.count'
-rw-r--r-- 1 dominikh users 16384 Dec  7  2023 'gopls@devel-go1.21.0-linux-amd64-2023-12-07.v1.count'
-rw-r--r-- 1 dominikh users 16384 Dec 29  2023 'gopls@devel-go1.21.0-linux-amd64-2023-12-29.v1.count'
-rw-r--r-- 1 dominikh users 16384 Mar 21 04:29 'gopls@devel-go1.22.0-linux-amd64-2024-03-21.v1.count'
-rw-r--r-- 1 dominikh users 16384 May  9 20:57 'gopls@devel-go1.22.0-linux-amd64-2024-05-09.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  3 17:53 'link@devel-devel-linux-amd64-2024-07-03.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  8 00:43 'link@devel-devel-linux-amd64-2024-07-07.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  5 00:15  link-devel-linux-amd64-2024-07-04.v1.count
-rw-r--r-- 1 dominikh users 16384 Jul  5 14:54  link-devel-linux-amd64-2024-07-05.v1.count
-rw-r--r-- 1 dominikh users 16384 Jul  6 23:16  link-devel-linux-amd64-2024-07-06.v1.count
-rw-r--r-- 1 dominikh users 16384 Jul  7 23:05  link-devel-linux-amd64-2024-07-07.v1.count
-rw-r--r-- 1 dominikh users 16384 Jul  7 13:54 'link@go1.23rc1-go1.23rc1-linux-amd64-2024-07-07.v1.count'
-rw-r--r-- 1 dominikh users  1510 Nov 15  2023  local.2023-11-15.json
-rw-r--r-- 1 dominikh users  3202 Nov 24  2023  local.2023-11-22.json
-rw-r--r-- 1 dominikh users  1388 Nov 30  2023  local.2023-11-29.json
-rw-r--r-- 1 dominikh users  1689 Dec 10  2023  local.2023-12-06.json
-rw-r--r-- 1 dominikh users  3166 Dec 13  2023  local.2023-12-13.json
-rw-r--r-- 1 dominikh users  3254 Dec 21  2023  local.2023-12-20.json
-rw-r--r-- 1 dominikh users  3140 Jan 14 02:07  local.2023-12-27.json
-rw-r--r-- 1 dominikh users  4137 Jan 25 14:42  local.2024-01-03.json
-rw-r--r-- 1 dominikh users  1259 Jan 25 14:42  local.2024-01-10.json
-rw-r--r-- 1 dominikh users  4358 Jan 25 14:42  local.2024-01-24.json
-rw-r--r-- 1 dominikh users  3118 Jan 31 17:38  local.2024-01-31.json
-rw-r--r-- 1 dominikh users  3990 Feb  8 14:15  local.2024-02-07.json
-rw-r--r-- 1 dominikh users  2419 Feb 14 14:29  local.2024-02-14.json
-rw-r--r-- 1 dominikh users  4440 Feb 21 04:10  local.2024-02-21.json
-rw-r--r-- 1 dominikh users  1667 Mar  8 02:44  local.2024-02-28.json
-rw-r--r-- 1 dominikh users  1122 Mar  8 02:44  local.2024-03-06.json
-rw-r--r-- 1 dominikh users  2086 Mar 13 01:58  local.2024-03-13.json
-rw-r--r-- 1 dominikh users  3543 Mar 20 01:04  local.2024-03-20.json
-rw-r--r-- 1 dominikh users  2059 Mar 27 03:33  local.2024-03-27.json
-rw-r--r-- 1 dominikh users  2686 Apr  4 10:34  local.2024-04-03.json
-rw-r--r-- 1 dominikh users  2599 Apr 12 19:34  local.2024-04-10.json
-rw-r--r-- 1 dominikh users  1813 Apr 17 03:46  local.2024-04-17.json
-rw-r--r-- 1 dominikh users  2120 Apr 24 02:27  local.2024-04-24.json
-rw-r--r-- 1 dominikh users  2650 May  2 00:38  local.2024-05-01.json
-rw-r--r-- 1 dominikh users  1668 May  8 02:59  local.2024-05-08.json
-rw-r--r-- 1 dominikh users  5633 May 15 19:12  local.2024-05-15.json
-rw-r--r-- 1 dominikh users  6380 May 22 20:07  local.2024-05-22.json
-rw-r--r-- 1 dominikh users  5024 May 30 22:10  local.2024-05-29.json
-rw-r--r-- 1 dominikh users  5249 Jun  5 17:08  local.2024-06-05.json
-rw-r--r-- 1 dominikh users  6589 Jun 12 21:22  local.2024-06-12.json
-rw-r--r-- 1 dominikh users  6099 Jun 19 02:24  local.2024-06-19.json
-rw-r--r-- 1 dominikh users  7599 Jun 26 15:35  local.2024-06-26.json
-rw-r--r-- 1 dominikh users 11755 Jul  3 15:18  local.2024-07-03.json
-rw-r--r-- 1 dominikh users 16384 Jul  4 19:39 'preprofile@devel-devel-linux-amd64-2024-07-04.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  8 00:43 'preprofile@devel-devel-linux-amd64-2024-07-07.v1.count'
-rw-r--r-- 1 dominikh users     0 Jul  7 23:50  upload.token
-rw-r--r-- 1 dominikh users 16384 Jul  3 17:53 'vet@devel-devel-linux-amd64-2024-07-03.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  4 18:37 'vet@devel-devel-linux-amd64-2024-07-04.v1.count'
-rw-r--r-- 1 dominikh users 16384 Jul  7 13:54 'vet@go1.23rc1-go1.23rc1-linux-amd64-2024-07-07.v1.count'
-rw-r--r-- 1 dominikh users     2 Nov 10  2023  weekends