Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: panic: unexpected return pc for runtime.heapBitsSetType #38851

Open
fasaxc opened this issue May 4, 2020 · 3 comments
Open

runtime: panic: unexpected return pc for runtime.heapBitsSetType #38851

fasaxc opened this issue May 4, 2020 · 3 comments
Milestone

Comments

@fasaxc
Copy link

@fasaxc fasaxc commented May 4, 2020

What version of Go are you using (go version)?

$ go version
go version go1.13.7 linux/amd64

Does this issue reproduce with the latest release?

I don't know yet, it's intermittent

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/user/.cache/go-build"
GOENV="/home/user/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="0"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build203255452=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Ran a large test FV suite for our application (approximately 900 tests, each of which starts and stops multiple copies of our application).

What did you expect to see?

A clean run.

What did you see instead?

One of the tests panicked in the go runtime very soon after our app started; full stack trace from the panic is in the details below. I don't really understand the message of the panic so it's hard to guess what other diagnostics might be useful.

Looking at the chunk of code that generates the panic there's a comment

// This happens if you get a profiling interrupt at just the wrong time.
/// In that context it is okay to stop early.
/// But if callback is set, we're doing a garbage collection and must
/// get everything, so crash loudly.

Does that mean that this might be caused by a profiling interrupt during a GC? Does that refer to go's built-in profiling mechanisms; I think we run with memory profiling turned on by default since it's useful and low overhead but if it causes random panics then we may be forced to turn it off.

runtime: unexpected return pc for runtime.heapBitsSetType called from 0x7273752f3d485441
stack: frame={sp:0xc000694388, fp:0xc000694390} stack=[0xc000690000,0xc000698000)
000000c000694288:  0000000000000000  0000009b00000008 
000000c000694298:  0000000000000000  000000c000458800 
000000c0006942a8:  0000000000000001  0000000000000010 
000000c0006942b8:  0000009c00000008  0000000000000000 
000000c0006942c8:  000000c000458804  0000000000000001 
000000c0006942d8:  000000000000000f  0000009d00000008 
000000c0006942e8:  0000000000000000  000000c000458808 
000000c0006942f8:  0000000000000001  000000000000000e 
000000c000694308:  0000009e00000008  0000000000000000 
000000c000694318:  000000c00045880c  0000000000000001 
000000c000694328:  000000000000000d  0000009f00000008 
000000c000694338:  0000000000000000  000000c000458810 
000000c000694348:  0000000000000001  000000000000000c 
000000c000694358:  000000a000000008  0000000000000000 
000000c000694368:  000000c000458814  0000000000000001 
000000c000694378:  000000000000000b  000000a100000008 
000000c000694388: <7273752f3d485441 >732f6c61636f6c2f 
000000c000694398:  0000000000000000  0000000000000000 
000000c0006943a8:  0000000001a00ac0  0000000000000000 
000000c0006943b8:  0000000000000000  0000000000000000 
000000c0006943c8:  0000000000000008  000000c000010990 
000000c0006943d8:  0000000000000000  0000000000000000 
000000c0006943e8:  000000c000568000  00007f7b481d0d98 
000000c0006943f8:  0000000000000000  0000000000000000 
000000c000694408:  0000000000000000  0000000000000000 
000000c000694418:  0000000000000000  000000c000826450 
000000c000694428:  000000000040ef88   0000000000000008 
000000c000694438:  0000000001a00ac0  0000000000000001 
000000c000694448:  0000000000000000  000000c000826530 
000000c000694458:  000000000044143c   0000000001a00ac0 
000000c000694468:  0000000000000000  0000000000000000 
000000c000694478:  0000000000000000  0000000000000000 
000000c000694488:  0000000000000000 
fatal error: unknown caller pc

runtime stack:
runtime.throw(0x1d4253f, 0x11)
/usr/local/go-cgo/src/runtime/panic.go:774 +0x72
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc00054d200, 0x0, 0x0, 0x7fffffff, 0x1e06f00, 0x7f7b3effcbc0, 0x0, ...)
/usr/local/go-cgo/src/runtime/traceback.go:273 +0x19e5
runtime.copystack(0xc00054d200, 0x8000, 0x7f7b00000001)
/usr/local/go-cgo/src/runtime/stack.go:886 +0x25a
runtime.newstack()
/usr/local/go-cgo/src/runtime/stack.go:1055 +0x2fb
runtime.morestack()
/usr/local/go-cgo/src/runtime/asm_amd64.s:449 +0x8f

goroutine 143 [copystack]:
runtime: unexpected return pc for runtime.heapBitsSetType called from 0x7273752f3d485441
stack: frame={sp:0xc000694388, fp:0xc000694390} stack=[0xc000690000,0xc000698000)
000000c000694288: 0000000000000000 0000009b00000008
000000c000694298: 0000000000000000 000000c000458800
000000c0006942a8: 0000000000000001 0000000000000010
000000c0006942b8: 0000009c00000008 0000000000000000
000000c0006942c8: 000000c000458804 0000000000000001
000000c0006942d8: 000000000000000f 0000009d00000008
000000c0006942e8: 0000000000000000 000000c000458808
000000c0006942f8: 0000000000000001 000000000000000e
000000c000694308: 0000009e00000008 0000000000000000
000000c000694318: 000000c00045880c 0000000000000001
000000c000694328: 000000000000000d 0000009f00000008
000000c000694338: 0000000000000000 000000c000458810
000000c000694348: 0000000000000001 000000000000000c
000000c000694358: 000000a000000008 0000000000000000
000000c000694368: 000000c000458814 0000000000000001
000000c000694378: 000000000000000b 000000a100000008
000000c000694388: <7273752f3d485441 >732f6c61636f6c2f
000000c000694398: 0000000000000000 0000000000000000
000000c0006943a8: 0000000001a00ac0 0000000000000000
000000c0006943b8: 0000000000000000 0000000000000000
000000c0006943c8: 0000000000000008 000000c000010990
000000c0006943d8: 0000000000000000 0000000000000000
000000c0006943e8: 000000c000568000 00007f7b481d0d98
000000c0006943f8: 0000000000000000 0000000000000000
000000c000694408: 0000000000000000 0000000000000000
000000c000694418: 0000000000000000 000000c000826450
000000c000694428: 000000000040ef88 <runtime.newobject+56> 0000000000000008
000000c000694438: 0000000001a00ac0 0000000000000001
000000c000694448: 0000000000000000 000000c000826530
000000c000694458: 000000000044143c <reflect.typelinks+108> 0000000001a00ac0
000000c000694468: 0000000000000000 0000000000000000
000000c000694478: 0000000000000000 0000000000000000
000000c000694488: 0000000000000000
runtime.heapBitsSetType(0x732f6c61636f6c2f, 0x0, 0x0, 0x1a00ac0)
/usr/local/go-cgo/src/runtime/mbitmap.go:938 +0xa4e fp=0xc000694390 sp=0xc000694388 pc=0x418bbe
created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c

goroutine 1 [select]:
github.com/projectcalico/felix/daemon.monitorAndManageShutdown(0xc000470060, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/projectcalico/felix/daemon/daemon.go:670 +0x281
github.com/projectcalico/felix/daemon.Run(0xc00018615b, 0x15, 0x20722a0, 0x20, 0x20a5be0, 0x28, 0x206bf50, 0x18)
/go/src/github.com/projectcalico/felix/daemon/daemon.go:615 +0x2cbd
main.main()
/go/src/github.com/projectcalico/felix/cmd/calico-felix/calico-felix.go:51 +0x1ca

goroutine 19 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go-cgo/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
/usr/local/go-cgo/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
/usr/local/go-cgo/src/os/signal/signal_unix.go:29 +0x41

goroutine 35 [chan receive]:
k8s.io/klog.(*loggingT).flushDaemon(0x2f5b2e0)
/go/pkg/mod/k8s.io/klog@v0.4.0/klog.go:1018 +0x8b
created by k8s.io/klog.init.0
/go/pkg/mod/k8s.io/klog@v0.4.0/klog.go:404 +0x6c

goroutine 71 [IO wait]:
internal/poll.runtime_pollWait(0x7f7b44760e38, 0x72, 0xffffffffffffffff)
/usr/local/go-cgo/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc00028d418, 0x72, 0xa00, 0xa91, 0xffffffffffffffff)
/usr/local/go-cgo/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go-cgo/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00028d400, 0xc000546400, 0xa91, 0xa91, 0x0, 0x0, 0x0)
/usr/local/go-cgo/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc00028d400, 0xc000546400, 0xa91, 0xa91, 0x203000, 0x0, 0xa57)
/usr/local/go-cgo/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000d00f0, 0xc000546400, 0xa91, 0xa91, 0x0, 0x0, 0x0)
/usr/local/go-cgo/src/net/net.go:184 +0x68
crypto/tls.(*atLeastReader).Read(0xc0000c42a0, 0xc000546400, 0xa91, 0xa91, 0x50, 0xc0000e5d60, 0xc000423970)
/usr/local/go-cgo/src/crypto/tls/conn.go:780 +0x60
bytes.(*Buffer).ReadFrom(0xc00009fe58, 0x206c840, 0xc0000c42a0, 0x40c6f5, 0x1ab8c40, 0x1cacac0)
/usr/local/go-cgo/src/bytes/buffer.go:204 +0xb4
crypto/tls.(*Conn).readFromUntil(0xc00009fc00, 0x206efc0, 0xc0000d00f0, 0x5, 0xc0000d00f0, 0x203000)
/usr/local/go-cgo/src/crypto/tls/conn.go:802 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc00009fc00, 0x0, 0x0, 0xc0000c4240)
/usr/local/go-cgo/src/crypto/tls/conn.go:609 +0x124
crypto/tls.(*Conn).readRecord(...)
/usr/local/go-cgo/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc00009fc00, 0xc0005d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go-cgo/src/crypto/tls/conn.go:1255 +0x161
bufio.(*Reader).Read(0xc0000945a0, 0xc0001f42d8, 0x9, 0x9, 0xc000423d10, 0x0, 0x924d55)
/usr/local/go-cgo/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x206c6a0, 0xc0000945a0, 0xc0001f42d8, 0x9, 0x9, 0x9, 0xc0000b6050, 0x0, 0x206ca20)
/usr/local/go-cgo/src/io/io.go:310 +0x87
io.ReadFull(...)
/usr/local/go-cgo/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0001f42d8, 0x9, 0x9, 0x206c6a0, 0xc0000945a0, 0x0, 0x0, 0xc000618e40, 0x0)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0001f42a0, 0xc000618e40, 0x0, 0x0, 0x0)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/frame.go:492 +0xa1
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000423fb8, 0x407295, 0xc000094720)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1746 +0x8c
golang.org/x/net/http2.(*ClientConn).readLoop(0xc0000a1500)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1674 +0xa3
created by golang.org/x/net/http2.(*Transport).newClientConn
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:674 +0x62f

goroutine 73 [chan receive]:
github.com/projectcalico/libcalico-go/lib/logutils.(*Destination).LoopWritingLogs(0xc0002b1440)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/logutils/logutils.go:323 +0x83
created by github.com/projectcalico/libcalico-go/lib/logutils.(*BackgroundHook).Start
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/logutils/logutils.go:468 +0x5e

goroutine 114 [sleep]:
runtime.goparkunlock(...)
/usr/local/go-cgo/src/runtime/proc.go:310
time.Sleep(0x62f5b97)
/usr/local/go-cgo/src/runtime/time.go:105 +0x157
github.com/projectcalico/felix/jitter.(*Ticker).loop(0xc000518420, 0xc0000d2de0)
/go/src/github.com/projectcalico/felix/jitter/jittered_ticker.go:53 +0x4e
created by github.com/projectcalico/felix/jitter.NewTicker
/go/src/github.com/projectcalico/felix/jitter/jittered_ticker.go:46 +0xeb

goroutine 53 [runnable]:
syscall.readlen(0x10, 0xc00066f878, 0x8, 0x8, 0xc0003605b0, 0x19)
/usr/local/go-cgo/src/syscall/zsyscall_linux_amd64.go:1025 +0xef
syscall.forkExec(0xc0005072e0, 0x10, 0xc0003aa770, 0x7, 0x7, 0xc00066f998, 0x18, 0x3e98d35b00000200, 0xc00012ea80)
/usr/local/go-cgo/src/syscall/exec_unix.go:210 +0x3c6
syscall.StartProcess(...)
/usr/local/go-cgo/src/syscall/exec_unix.go:248
os.startProcess(0xc0005072e0, 0x10, 0xc0003aa770, 0x7, 0x7, 0xc00066fb30, 0x0, 0x0, 0x0)
/usr/local/go-cgo/src/os/exec_posix.go:51 +0x2b0
os.StartProcess(0xc0005072e0, 0x10, 0xc0003aa770, 0x7, 0x7, 0xc00066fb30, 0x18, 0x0, 0x1ed)
/usr/local/go-cgo/src/os/exec.go:102 +0x7c
os/exec.(*Cmd).Start(0xc0004ef760, 0x1, 0xc0006f6db0)
/usr/local/go-cgo/src/os/exec/exec.go:416 +0x50c
os/exec.(*Cmd).Run(0xc0004ef760, 0xc0006f6db0, 0xc0004ef760)
/usr/local/go-cgo/src/os/exec/exec.go:338 +0x2b
os/exec.(*Cmd).CombinedOutput(0xc0004ef760, 0x7, 0xc0003aa710, 0x6, 0x6, 0xc0004ef760)
/usr/local/go-cgo/src/os/exec/exec.go:561 +0x91
github.com/projectcalico/felix/bpf.(*PinnedMap).Iter(0xc000630cb0, 0xc0006f2d60, 0x0, 0x0)
/go/src/github.com/projectcalico/felix/bpf/maps.go:179 +0x11e
github.com/projectcalico/felix/bpf/nat.LoadBackendMap(0x20e03a0, 0xc000630cb0, 0xc000802ab0, 0x0, 0x0)
/go/src/github.com/projectcalico/felix/bpf/nat/maps.go:282 +0x44
github.com/projectcalico/felix/bpf/proxy.(*Syncer).loadOrigs(0xc000814000, 0xc000814000, 0xc00048ccc0)
/go/src/github.com/projectcalico/felix/bpf/proxy/syncer.go:190 +0x73
github.com/projectcalico/felix/bpf/proxy.NewSyncer(0xc000802a20, 0x2, 0x2, 0x20e03a0, 0xc000630c40, 0x20e03a0, 0xc000630cb0, 0x20e03a0, 0xc000630d90, 0x20810a0, ...)
/go/src/github.com/projectcalico/felix/bpf/proxy/syncer.go:176 +0x1a2
github.com/projectcalico/felix/bpf/proxy.(*KubeProxy).run(0xc000626000, 0xc0007f4fc0, 0x1, 0x1, 0x0, 0x0)
/go/src/github.com/projectcalico/felix/bpf/proxy/kube-proxy.go:98 +0x1e9
github.com/projectcalico/felix/bpf/proxy.(*KubeProxy).start(0xc000626000, 0xc0000d3320, 0xc0005667b0)
/go/src/github.com/projectcalico/felix/bpf/proxy/kube-proxy.go:120 +0xb2
github.com/projectcalico/felix/bpf/proxy.StartKubeProxy.func1(0xc000626000)
/go/src/github.com/projectcalico/felix/bpf/proxy/kube-proxy.go:67 +0x2b
created by github.com/projectcalico/felix/bpf/proxy.StartKubeProxy
/go/src/github.com/projectcalico/felix/bpf/proxy/kube-proxy.go:66 +0x25c

goroutine 111 [select]:
github.com/projectcalico/felix/dataplane/linux.(*InternalDataplane).loopUpdatingDataplane(0xc000183600)
/go/src/github.com/projectcalico/felix/dataplane/linux/int_dataplane.go:1128 +0x3d5
created by github.com/projectcalico/felix/dataplane/linux.(*InternalDataplane).Start
/go/src/github.com/projectcalico/felix/dataplane/linux/int_dataplane.go:773 +0x51

goroutine 112 [sleep]:
runtime.goparkunlock(...)
/usr/local/go-cgo/src/runtime/proc.go:310
time.Sleep(0x2540be400)
/usr/local/go-cgo/src/runtime/time.go:105 +0x157
github.com/projectcalico/felix/dataplane/linux.(*InternalDataplane).loopReportingStatus(0xc000183600)
/go/src/github.com/projectcalico/felix/dataplane/linux/int_dataplane.go:1469 +0xa4
created by github.com/projectcalico/felix/dataplane/linux.(*InternalDataplane).Start
/go/src/github.com/projectcalico/felix/dataplane/linux/int_dataplane.go:774 +0x73

goroutine 131 [chan receive]:
github.com/projectcalico/felix/calc.(*SyncerCallbacksDecoupler).SendTo(0xc0005c0038, 0x20811e0, 0xc000347f10)
/go/src/github.com/projectcalico/felix/calc/async_decoupler.go:40 +0x50
created by github.com/projectcalico/felix/daemon.Run
/go/src/github.com/projectcalico/felix/daemon/daemon.go:560 +0x2b44

goroutine 81 [sleep]:
runtime.goparkunlock(...)
/usr/local/go-cgo/src/runtime/proc.go:310
time.Sleep(0x162ba721e4)
/usr/local/go-cgo/src/runtime/time.go:105 +0x157
github.com/projectcalico/felix/jitter.(*Ticker).loop(0xc000518320, 0xc0000d2d80)
/go/src/github.com/projectcalico/felix/jitter/jittered_ticker.go:53 +0x4e
created by github.com/projectcalico/felix/jitter.NewTicker
/go/src/github.com/projectcalico/felix/jitter/jittered_ticker.go:46 +0xeb

goroutine 80 [sleep]:
runtime.goparkunlock(...)
/usr/local/go-cgo/src/runtime/proc.go:310
time.Sleep(0x2649aa928)
/usr/local/go-cgo/src/runtime/time.go:105 +0x157
github.com/projectcalico/felix/jitter.(*Ticker).loop(0xc000518240, 0xc0000d2d20)
/go/src/github.com/projectcalico/felix/jitter/jittered_ticker.go:53 +0x4e
created by github.com/projectcalico/felix/jitter.NewTicker
/go/src/github.com/projectcalico/felix/jitter/jittered_ticker.go:46 +0xeb

goroutine 113 [select]:
github.com/projectcalico/felix/ifacemonitor.(*InterfaceMonitor).MonitorInterfaces(0xc0000d2a80)
/go/src/github.com/projectcalico/felix/ifacemonitor/iface_monitor.go:111 +0x597
created by github.com/projectcalico/felix/dataplane/linux.(*InternalDataplane).Start
/go/src/github.com/projectcalico/felix/dataplane/linux/int_dataplane.go:775 +0x9c

goroutine 130 [chan receive]:
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run(0xc0000953e0, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:149 +0x2f2
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).Start.func1(0xc0000953e0, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:102 +0x43
created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).Start
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:101 +0x139

goroutine 56 [chan receive]:
github.com/vishvananda/netlink.linkSubscribeAt.func1(0xc000186600, 0xc00058c510)
/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/link_linux.go:1642 +0x34
created by github.com/vishvananda/netlink.linkSubscribeAt
/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/link_linux.go:1641 +0x36b

goroutine 57 [syscall]:
syscall.Syscall6(0x2d, 0xb, 0xc0006b0000, 0x10000, 0x0, 0xc00017e0e0, 0xc0005f05f4, 0x0, 0xc0005f0590, 0x40ef88)
/usr/local/go-cgo/src/syscall/asm_linux_amd64.s:44 +0x5
golang.org/x/sys/unix.recvfrom(0xb, 0xc0006b0000, 0x10000, 0x10000, 0x0, 0xc00017e0e0, 0xc0005f05f4, 0xc0005f0608, 0x4479ec, 0x10000)
/go/pkg/mod/golang.org/x/sys@v0.0.0-20200202164722-d101bd2416d5/unix/zsyscall_linux_amd64.go:2414 +0xa3
golang.org/x/sys/unix.Recvfrom(0xb, 0xc0006b0000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x4302fa)
/go/pkg/mod/golang.org/x/sys@v0.0.0-20200202164722-d101bd2416d5/unix/syscall_unix.go:307 +0x94
github.com/vishvananda/netlink/nl.(*NetlinkSocket).Receive(0xc00058c510, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/nl/nl_linux.go:626 +0x93
github.com/vishvananda/netlink.linkSubscribeAt.func2(0xc000466000, 0xc00058c510, 0x0)
/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/link_linux.go:1658 +0xf3
created by github.com/vishvananda/netlink.linkSubscribeAt
/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/link_linux.go:1655 +0x204

goroutine 58 [chan receive]:
github.com/vishvananda/netlink.addrSubscribeAt.func1(0xc000186600, 0xc00058c570)
/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/addr_linux.go:308 +0x34
created by github.com/vishvananda/netlink.addrSubscribeAt
/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/addr_linux.go:307 +0x36e

goroutine 59 [syscall]:
syscall.Syscall6(0x2d, 0xc, 0xc0006f8000, 0x10000, 0x0, 0xc0001e4310, 0xc0005f1d24, 0xc0005f1ef0, 0xc0005f1cc0, 0x40ef88)
/usr/local/go-cgo/src/syscall/asm_linux_amd64.s:44 +0x5
golang.org/x/sys/unix.recvfrom(0xc, 0xc0006f8000, 0x10000, 0x10000, 0x0, 0xc0001e4310, 0xc0005f1d24, 0xc0005f1d38, 0x4479ec, 0x10000)
/go/pkg/mod/golang.org/x/sys@v0.0.0-20200202164722-d101bd2416d5/unix/zsyscall_linux_amd64.go:2414 +0xa3
golang.org/x/sys/unix.Recvfrom(0xc, 0xc0006f8000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x4302fa)
/go/pkg/mod/golang.org/x/sys@v0.0.0-20200202164722-d101bd2416d5/unix/syscall_unix.go:307 +0x94
github.com/vishvananda/netlink/nl.(*NetlinkSocket).Receive(0xc00058c570, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/nl/nl_linux.go:626 +0x93
github.com/vishvananda/netlink.addrSubscribeAt.func2(0xc000466060, 0xc00058c570, 0x0)
/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/addr_linux.go:324 +0xf9
created by github.com/vishvananda/netlink.addrSubscribeAt
/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/addr_linux.go:321 +0x207

goroutine 132 [select]:
github.com/projectcalico/felix/calc.(*AsyncCalcGraph).loop(0xc000095620)
/go/src/github.com/projectcalico/felix/calc/async_calc_graph.go:139 +0x192
created by github.com/projectcalico/felix/calc.(*AsyncCalcGraph).Start
/go/src/github.com/projectcalico/felix/calc/async_calc_graph.go:237 +0xb3

goroutine 133 [chan receive]:
github.com/projectcalico/felix/daemon.(*DataplaneConnector).sendMessagesToDataplaneDriver(0xc00062c780)
/go/src/github.com/projectcalico/felix/daemon/daemon.go:1103 +0xb0
created by github.com/projectcalico/felix/daemon.(*DataplaneConnector).Start
/go/src/github.com/projectcalico/felix/daemon/daemon.go:1185 +0x3f

goroutine 134 [semacquire]:
github.com/sirupsen/logrus.NewEntry(...)
/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/entry.go:52
github.com/sirupsen/logrus.(*Logger).newEntry(0xc0000d2120, 0xc0005c0018)
/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/logger.go:83 +0x6f
github.com/sirupsen/logrus.(*Logger).Info(0xc0000d2120, 0xc00007bf98, 0x1, 0x1)
/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/logger.go:188 +0x41
github.com/sirupsen/logrus.Info(...)
/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/exported.go:85
github.com/projectcalico/felix/daemon.(*DataplaneConnector).readMessagesFromDataplane(0xc00062c780)
/go/src/github.com/projectcalico/felix/daemon/daemon.go:939 +0xbe
created by github.com/projectcalico/felix/daemon.(*DataplaneConnector).Start
/go/src/github.com/projectcalico/felix/daemon/daemon.go:1188 +0x61

goroutine 135 [select]:
github.com/projectcalico/felix/daemon.(*DataplaneConnector).handleWireguardStatUpdateFromDataplane(0xc00062c780)
/go/src/github.com/projectcalico/felix/daemon/daemon.go:1070 +0xc8
created by github.com/projectcalico/felix/daemon.(*DataplaneConnector).Start
/go/src/github.com/projectcalico/felix/daemon/daemon.go:1191 +0x83

goroutine 136 [runnable]:
runtime.goparkunlock(...)
/usr/local/go-cgo/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0004ef0c0, 0x0)
/usr/local/go-cgo/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0004ef0b0)
/usr/local/go-cgo/src/sync/cond.go:56 +0x9d
golang.org/x/net/http2.(*pipe).Read(0xc0004ef0a8, 0xc000658200, 0x200, 0x200, 0x0, 0x0, 0x0)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/pipe.go:65 +0xa6
golang.org/x/net/http2.transportResponseBody.Read(0xc0004ef080, 0xc000658200, 0x200, 0x200, 0x0, 0x0, 0x0)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:2019 +0xac
bytes.(*Buffer).ReadFrom(0xc0007c63f8, 0x7f7b44507518, 0xc0004ef080, 0x40b5ab, 0xc00003c000, 0x1ab8c40)
/usr/local/go-cgo/src/bytes/buffer.go:204 +0xb4
io/ioutil.readAll(0x7f7b44507518, 0xc0004ef080, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go-cgo/src/io/ioutil/ioutil.go:36 +0x100
io/ioutil.ReadAll(...)
/usr/local/go-cgo/src/io/ioutil/ioutil.go:45
k8s.io/client-go/rest.(*Request).transformResponse(0xc00067e000, 0xc00018cab0, 0xc000382c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:861 +0x6a7
k8s.io/client-go/rest.(*Request).Do.func1(0xc000382c00, 0xc00018cab0)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:829 +0x5f
k8s.io/client-go/rest.(*Request).request.func2(0xc00018cab0, 0xc0007c6820, 0xa, 0xc00067e000, 0xc0007c6948, 0xc000382c00, 0xc0004b0730, 0x49, 0xc000382b00)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:805 +0xd1
k8s.io/client-go/rest.(*Request).request(0xc00067e000, 0xc0007c6948, 0x0, 0x0)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:807 +0x6ba
k8s.io/client-go/rest.(*Request).Do(0xc00067e000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:828 +0xd8
github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*customK8sResourceClient).List(0xc0004889a0, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014ef80, 0x0, 0x0, 0x30, 0x30, 0x1c5ca00)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/customresource.go:287 +0x9dc
github.com/projectcalico/libcalico-go/lib/backend/k8s.(*KubeClient).List(0xc000668b80, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014ef80, 0x0, 0x0, 0x46, 0x0, 0x0)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/k8s.go:592 +0x2f9
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).resyncAndCreateWatcher(0xc0001165a0, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:203 +0x7fb
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc0001165a0, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:81 +0xb9
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc0001165a0)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43
created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c

goroutine 137 [runnable]:
golang.org/x/net/http2.(*ClientConn).roundTrip(0xc0000a1500, 0xc000616d00, 0x0, 0x0, 0x0, 0x0)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1096 +0x9a0
golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc0002b9500, 0xc000616d00, 0xc00078e000, 0x72db95, 0x0, 0xc000207200)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:448 +0x172
golang.org/x/net/http2.(*Transport).RoundTrip(...)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:409
golang.org/x/net/http2.noDialH2RoundTripper.RoundTrip(0xc0002b9500, 0xc000616d00, 0xc0001255e0, 0x5, 0xc000207288)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:2603 +0x3e
net/http.(*Transport).roundTrip(0xc0002b43c0, 0xc000616d00, 0x4119fe, 0x2039510, 0x0)
/usr/local/go-cgo/src/net/http/transport.go:485 +0xdc1
net/http.(*Transport).RoundTrip(0xc0002b43c0, 0xc000616d00, 0xc0002b43c0, 0x0, 0x0)
/usr/local/go-cgo/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc000616d00, 0x206f100, 0xc0002b43c0, 0x0, 0x0, 0x0, 0xc0000105e8, 0xc0005cfa80, 0x1, 0x0)
/usr/local/go-cgo/src/net/http/client.go:250 +0x443
net/http.(*Client).send(0xc0006637d0, 0xc000616d00, 0x0, 0x0, 0x0, 0xc0000105e8, 0x0, 0x1, 0xc000804810)
/usr/local/go-cgo/src/net/http/client.go:174 +0xfa
net/http.(*Client).do(0xc0006637d0, 0xc000616d00, 0x0, 0x0, 0x0)
/usr/local/go-cgo/src/net/http/client.go:641 +0x3ce
net/http.(*Client).Do(0xc0006637d0, 0xc000616d00, 0x0, 0x3, 0xc0001255e0)
/usr/local/go-cgo/src/net/http/client.go:509 +0x35
k8s.io/client-go/rest.(*Request).request(0xc0005e8a80, 0xc00078e948, 0x0, 0x0)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:754 +0x367
k8s.io/client-go/rest.(*Request).Do(0xc0005e8a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:828 +0xd8
github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*customK8sResourceClient).List(0xc0004888f0, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014efc0, 0x0, 0x0, 0x30, 0x30, 0x1c5ca00)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/customresource.go:287 +0x9dc
github.com/projectcalico/libcalico-go/lib/backend/k8s.(*KubeClient).List(0xc000668b80, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014efc0, 0x0, 0x0, 0x46, 0x0, 0x0)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/k8s.go:592 +0x2f9
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).resyncAndCreateWatcher(0xc000116630, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:203 +0x7fb
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc000116630, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:81 +0xb9
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc000116630)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43
created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c

goroutine 138 [runnable]:
sync.runtime_SemacquireMutex(0xc0000a1554, 0xc00078a100, 0x1)
/usr/local/go-cgo/src/runtime/sema.go:71 +0x47
sync.(*Mutex).lockSlow(0xc0000a1550)
/usr/local/go-cgo/src/sync/mutex.go:138 +0xfc
sync.(*Mutex).Lock(...)
/usr/local/go-cgo/src/sync/mutex.go:81
golang.org/x/net/http2.transportResponseBody.Read(0xc000280580, 0xc000658400, 0x200, 0x200, 0xcc, 0x0, 0x0)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:2042 +0x584
bytes.(*Buffer).ReadFrom(0xc00078a3f8, 0x7f7b44507518, 0xc000280580, 0x40b5ab, 0xc00003c000, 0x1ab8c40)
/usr/local/go-cgo/src/bytes/buffer.go:204 +0xb4
io/ioutil.readAll(0x7f7b44507518, 0xc000280580, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go-cgo/src/io/ioutil/ioutil.go:36 +0x100
io/ioutil.ReadAll(...)
/usr/local/go-cgo/src/io/ioutil/ioutil.go:45
k8s.io/client-go/rest.(*Request).transformResponse(0xc0005e8480, 0xc00018ca20, 0xc000616800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:861 +0x6a7
k8s.io/client-go/rest.(*Request).Do.func1(0xc000616800, 0xc00018ca20)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:829 +0x5f
k8s.io/client-go/rest.(*Request).request.func2(0xc00018ca20, 0xc00078a820, 0xa, 0xc0005e8480, 0xc00078a948, 0xc000616800, 0xc0001246e0, 0x4b, 0xc000616700)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:805 +0xd1
k8s.io/client-go/rest.(*Request).request(0xc0005e8480, 0xc00078a948, 0x0, 0x0)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:807 +0x6ba
k8s.io/client-go/rest.(*Request).Do(0xc0005e8480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:828 +0xd8
github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*customK8sResourceClient).List(0xc0004882c0, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f000, 0x0, 0x0, 0xbfa4231b92b10143, 0x71e74b0, 0x2f5ae40)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/customresource.go:287 +0x9dc
github.com/projectcalico/libcalico-go/lib/backend/k8s.(*KubeClient).List(0xc000668b80, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f000, 0x0, 0x0, 0x48, 0x0, 0x0)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/k8s.go:592 +0x2f9
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).resyncAndCreateWatcher(0xc0001166c0, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:203 +0x7fb
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc0001166c0, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:81 +0xb9
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc0001166c0)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43
created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c

goroutine 139 [select]:
golang.org/x/net/http2.(*ClientConn).roundTrip(0xc0000a1500, 0xc000382600, 0x0, 0x0, 0x0, 0x0)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1096 +0x9a0
golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc0002b9500, 0xc000382600, 0xc0006f0000, 0x72db95, 0x0, 0xc000207200)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:448 +0x172
golang.org/x/net/http2.(*Transport).RoundTrip(...)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:409
golang.org/x/net/http2.noDialH2RoundTripper.RoundTrip(0xc0002b9500, 0xc000382600, 0xc0004b1cc0, 0x5, 0xc000207288)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:2603 +0x3e
net/http.(*Transport).roundTrip(0xc0002b43c0, 0xc000382600, 0x0, 0xc0006f03a8, 0x10000000040ef88)
/usr/local/go-cgo/src/net/http/transport.go:485 +0xdc1
net/http.(*Transport).RoundTrip(0xc0002b43c0, 0xc000382600, 0xc0002b43c0, 0x0, 0x0)
/usr/local/go-cgo/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc000382600, 0x206f100, 0xc0002b43c0, 0x0, 0x0, 0x0, 0xc0005c0010, 0xc0007ced60, 0x1, 0x0)
/usr/local/go-cgo/src/net/http/client.go:250 +0x443
net/http.(*Client).send(0xc0006637d0, 0xc000382600, 0x0, 0x0, 0x0, 0xc0005c0010, 0x0, 0x1, 0xc000802db0)
/usr/local/go-cgo/src/net/http/client.go:174 +0xfa
net/http.(*Client).do(0xc0006637d0, 0xc000382600, 0x0, 0x0, 0x0)
/usr/local/go-cgo/src/net/http/client.go:641 +0x3ce
net/http.(*Client).Do(0xc0006637d0, 0xc000382600, 0x0, 0x3, 0xc0004b1cc0)
/usr/local/go-cgo/src/net/http/client.go:509 +0x35
k8s.io/client-go/rest.(*Request).request(0xc00067e600, 0xc0006f0948, 0x0, 0x0)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:754 +0x367
k8s.io/client-go/rest.(*Request).Do(0xc00067e600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:828 +0xd8
github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*customK8sResourceClient).List(0xc000488580, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f040, 0x0, 0x0, 0x30, 0x30, 0xc000802c60)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/customresource.go:287 +0x9dc
github.com/projectcalico/libcalico-go/lib/backend/k8s.(*KubeClient).List(0xc000668b80, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f040, 0x0, 0x0, 0x44, 0x0, 0x0)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/k8s.go:592 +0x2f9
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).resyncAndCreateWatcher(0xc000116750, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:203 +0x7fb
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc000116750, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:81 +0xb9
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc000116750)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43
created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c

goroutine 140 [runnable]:
github.com/sirupsen/logrus.Entry.log(0xc0000d2120, 0xc00057ff20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5, ...)
/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/entry.go:91 +0x3f
github.com/sirupsen/logrus.(*Entry).Debug(0xc0004b3400, 0xc00007a920, 0x1, 0x1)
/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/entry.go:143 +0x10f
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc0001167e0, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:80 +0x8e
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc0001167e0)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43
created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c

goroutine 141 [runnable]:
github.com/json-iterator/go.describeStruct(0xc0006f6ba0, 0x20f8c00, 0xc000662b40, 0x1c20e80)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_extension.go:393 +0x61c
github.com/json-iterator/go.encoderOfStruct(0xc0006f6ba0, 0x20f8c00, 0xc000662b40, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_struct_encoder.go:18 +0x5a
github.com/json-iterator/go._createEncoderOfType(0xc0006f6ba0, 0x20f8c00, 0xc000662b40, 0xc000194658, 0x1bfa120)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:275 +0x441
github.com/json-iterator/go.createEncoderOfType(0xc0006f6ba0, 0x20f8c00, 0xc000662b40, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:245 +0x10b
github.com/json-iterator/go.encoderOfType(0xc0006f6ba0, 0x20f8c00, 0xc000662b40, 0x1d2df08, 0x1)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:227 +0x78
github.com/json-iterator/go.describeStruct(0xc00084de10, 0x20f8c00, 0xc0002e8a20, 0x7f7b0002d330)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_extension.go:384 +0x8da
github.com/json-iterator/go.decoderOfStruct(0xc00084de10, 0x20f8c00, 0xc0002e8a20, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_struct_decoder.go:14 +0xb6
github.com/json-iterator/go._createDecoderOfType(0xc0007c1e10, 0x20f8c00, 0xc0002e8a20, 0xc0004e0208, 0x1bfa120)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:176 +0x4aa
github.com/json-iterator/go.createDecoderOfType(0xc0007c1e10, 0x20f8c00, 0xc0002e8a20, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:142 +0x10b
github.com/json-iterator/go.decoderOfType(0xc00084de10, 0x20f8c00, 0xc0002e8a20, 0x1d2df08, 0x1)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:124 +0x78
github.com/json-iterator/go.decoderOfSlice(0xc000619fb0, 0x20f8b60, 0xc0000d9100, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_slice.go:12 +0x105
github.com/json-iterator/go._createDecoderOfType(0xc000619fb0, 0x20f8b60, 0xc0000d9100, 0xc0004e01f8, 0xc0001c2c60)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:180 +0x4ff
github.com/json-iterator/go.createDecoderOfType(0xc000619fb0, 0x20f8b60, 0xc0000d9100, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:142 +0x10b
github.com/json-iterator/go.decoderOfType(0xc000619fb0, 0x20f8b60, 0xc0000d9100, 0x1d2df08, 0x1)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:124 +0x78
github.com/json-iterator/go.describeStruct(0xc00084e550, 0x20f8c00, 0xc000619200, 0x2d330)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_extension.go:380 +0xa4f
github.com/json-iterator/go.decoderOfStruct(0xc00084e550, 0x20f8c00, 0xc000619200, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_struct_decoder.go:14 +0xb6
github.com/json-iterator/go._createDecoderOfType(0xc0007c2550, 0x20f8c00, 0xc000619200, 0xc0004e01a8, 0xc0001c2c60)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:176 +0x4aa
github.com/json-iterator/go.createDecoderOfType(0xc0007c2550, 0x20f8c00, 0xc000619200, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:142 +0x10b
github.com/json-iterator/go.decoderOfType(0xc00084e550, 0x20f8c00, 0xc000619200, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:124 +0x78
github.com/json-iterator/go.(*frozenConfig).DecoderOf(0xc000175ea0, 0x20f8ac0, 0xc000618f90, 0x20f8ac0, 0xc000618f90)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:114 +0x111
github.com/json-iterator/go.(*Iterator).ReadVal(0xc00018cd80, 0x1cd8f40, 0xc0003d2000)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:72 +0x1ce
github.com/json-iterator/go.(*frozenConfig).Unmarshal(0xc000175ea0, 0xc00054e000, 0x52f, 0xe00, 0x1cd8f40, 0xc0003d2000, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/config.go:348 +0xd2
k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode(0xc00014ea40, 0xc00054e000, 0x52f, 0xe00, 0x0, 0x2085460, 0xc0003d2000, 0x1, 0x0, 0x206f080, ...)
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/runtime/serializer/json/json.go:275 +0x7d5
k8s.io/apimachinery/pkg/runtime.WithoutVersionDecoder.Decode(0x206e680, 0xc00014ea40, 0xc00054e000, 0x52f, 0xe00, 0x0, 0x2085460, 0xc0003d2000, 0x0, 0xc000616e00, ...)
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/runtime/helper.go:252 +0x97
k8s.io/client-go/rest.Result.Into(0xc00054e000, 0x52f, 0xe00, 0xc00014a250, 0x10, 0x0, 0x0, 0xc8, 0x2070ec0, 0xc0002aeef0, ...)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:1133 +0xba
k8s.io/client-go/kubernetes/typed/core/v1.(*nodes).List(0xc00081e1d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/kubernetes/typed/core/v1/node.go:88 +0x1d2
github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*nodeClient).List(0xc0002af810, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f180, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/node.go:156 +0x4d3
github.com/projectcalico/libcalico-go/lib/backend/k8s.(*KubeClient).List(0xc000668b80, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f180, 0x0, 0x0, 0x38, 0x0, 0x0)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/k8s.go:592 +0x2f9
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).resyncAndCreateWatcher(0xc000116870, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:203 +0x7fb
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc000116870, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:81 +0xb9
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc000116870)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43
created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c

goroutine 142 [select]:
golang.org/x/net/http2.(*ClientConn).roundTrip(0xc0000a1500, 0xc000616500, 0x0, 0x0, 0x0, 0x0)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1096 +0x9a0
golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc0002b9500, 0xc000616500, 0xc000678100, 0x72db95, 0x0, 0xc000207200)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:448 +0x172
golang.org/x/net/http2.(*Transport).RoundTrip(...)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:409
golang.org/x/net/http2.noDialH2RoundTripper.RoundTrip(0xc0002b9500, 0xc000616500, 0xc0004fe300, 0x5, 0xc000207288)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:2603 +0x3e
net/http.(*Transport).roundTrip(0xc0002b43c0, 0xc000616500, 0x1a005e0, 0x1, 0xc0000f8f20)
/usr/local/go-cgo/src/net/http/transport.go:485 +0xdc1
net/http.(*Transport).RoundTrip(0xc0002b43c0, 0xc000616500, 0x1d36edf, 0xa, 0xc0000f8fc0)
/usr/local/go-cgo/src/net/http/roundtrip.go:17 +0x35
k8s.io/client-go/transport.(*userAgentRoundTripper).RoundTrip(0xc0001f3b20, 0xc000616400, 0xc0001f3b20, 0x0, 0x0)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/transport/round_trippers.go:159 +0x1c0
net/http.send(0xc000616400, 0x206eea0, 0xc0001f3b20, 0x0, 0x0, 0x0, 0xc000010108, 0xc0003d4439, 0x1, 0x0)
/usr/local/go-cgo/src/net/http/client.go:250 +0x443
net/http.(*Client).send(0xc000523fb0, 0xc000616400, 0x0, 0x0, 0x0, 0xc000010108, 0x0, 0x1, 0xc000618540)
/usr/local/go-cgo/src/net/http/client.go:174 +0xfa
net/http.(*Client).do(0xc000523fb0, 0xc000616400, 0x0, 0x0, 0x0)
/usr/local/go-cgo/src/net/http/client.go:641 +0x3ce
net/http.(*Client).Do(0xc000523fb0, 0xc000616400, 0x0, 0x3, 0xc0004fe300)
/usr/local/go-cgo/src/net/http/client.go:509 +0x35
k8s.io/client-go/rest.(*Request).request(0xc0005e9200, 0xc0006789c0, 0x0, 0x0)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:754 +0x367
k8s.io/client-go/rest.(*Request).Do(0xc0005e9200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:828 +0xd8
k8s.io/client-go/kubernetes/typed/core/v1.(*namespaces).List(0xc0004d07d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/kubernetes/typed/core/v1/namespace.go:86 +0x179
github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*profileClient).List(0xc00003aba0, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f1c0, 0x0, 0x0, 0xbfa4231b93345c5b, 0x7a1cfc3, 0x2f5ae40)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/profile.go:192 +0x85b
github.com/projectcalico/libcalico-go/lib/backend/k8s.(*KubeClient).List(0xc000668b80, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f1c0, 0x0, 0x0, 0x3b, 0x0, 0x0)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/k8s.go:592 +0x2f9
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).resyncAndCreateWatcher(0xc000116900, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:203 +0x7fb
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc000116900, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:81 +0xb9
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc000116900)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43
created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c

goroutine 144 [runnable]:
github.com/json-iterator/go.createEncoderOfType(0xc0006f7890, 0x20f8ac0, 0xc000618ae0, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:244 +0xc8
github.com/json-iterator/go.encoderOfType(0xc0006f7890, 0x20f8ac0, 0xc000618ae0, 0x1d2df08, 0x1)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:227 +0x78
github.com/json-iterator/go.describeStruct(0xc00082ea68, 0x20f8c00, 0xc000618780, 0x2d330)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_extension.go:384 +0x8da
github.com/json-iterator/go.decoderOfStruct(0xc00082ea68, 0x20f8c00, 0xc000618780, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_struct_decoder.go:14 +0xb6
github.com/json-iterator/go._createDecoderOfType(0xc00082ea68, 0x20f8c00, 0xc000618780, 0xc00080c628, 0x1bfa120)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:176 +0x4aa
github.com/json-iterator/go.createDecoderOfType(0xc00082ea68, 0x20f8c00, 0xc000618780, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:142 +0x10b
github.com/json-iterator/go.decoderOfType(0xc00082ea68, 0x20f8c00, 0xc000618780, 0x1d2df08, 0x1)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:124 +0x78
github.com/json-iterator/go.decoderOfSlice(0xc0006f7650, 0x20f8b60, 0xc0002b1b80, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_slice.go:12 +0x105
github.com/json-iterator/go._createDecoderOfType(0xc0006f7650, 0x20f8b60, 0xc0002b1b80, 0xc0001e33e8, 0x1bfa120)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:180 +0x4ff
github.com/json-iterator/go.createDecoderOfType(0xc0006f7650, 0x20f8b60, 0xc0002b1b80, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:142 +0x10b
github.com/json-iterator/go.decoderOfType(0xc0006f7650, 0x20f8b60, 0xc0002b1b80, 0x1d2df08, 0x1)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:124 +0x78
github.com/json-iterator/go.describeStruct(0xc000803a40, 0x20f8c00, 0xc0004971a0, 0x2d330)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_extension.go:380 +0xa4f
github.com/json-iterator/go.decoderOfStruct(0xc000803a40, 0x20f8c00, 0xc0004971a0, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_struct_decoder.go:14 +0xb6
github.com/json-iterator/go._createDecoderOfType(0xc000803a40, 0x20f8c00, 0xc0004971a0, 0xc00080c338, 0x1bfa120)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:176 +0x4aa
github.com/json-iterator/go.createDecoderOfType(0xc000803a40, 0x20f8c00, 0xc0004971a0, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:142 +0x10b
github.com/json-iterator/go.decoderOfType(0xc000803a40, 0x20f8c00, 0xc0004971a0, 0x1d2df08, 0x1)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:124 +0x78
github.com/json-iterator/go.describeStruct(0xc00082f720, 0x20f8c00, 0xc000803800, 0x7f7b00009e51)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_extension.go:380 +0xa4f
github.com/json-iterator/go.decoderOfStruct(0xc00082f720, 0x20f8c00, 0xc000803800, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_struct_decoder.go:14 +0xb6
github.com/json-iterator/go._createDecoderOfType(0xc000823720, 0x20f8c00, 0xc000803800, 0xc00080c328, 0x1bfa120)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:176 +0x4aa
github.com/json-iterator/go.createDecoderOfType(0xc000823720, 0x20f8c00, 0xc000803800, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:142 +0x10b
github.com/json-iterator/go.decoderOfType(0xc00082f720, 0x20f8c00, 0xc000803800, 0x1d2df08, 0x1)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:124 +0x78
github.com/json-iterator/go.decoderOfSlice(0xc000803710, 0x20f8b60, 0xc00014ecc0, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_slice.go:12 +0x105
github.com/json-iterator/go._createDecoderOfType(0xc000803710, 0x20f8b60, 0xc00014ecc0, 0xc00080c318, 0xc0001c2c60)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:180 +0x4ff
github.com/json-iterator/go.createDecoderOfType(0xc000803710, 0x20f8b60, 0xc00014ecc0, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:142 +0x10b
github.com/json-iterator/go.decoderOfType(0xc000803710, 0x20f8b60, 0xc00014ecc0, 0x1d2df08, 0x1)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:124 +0x78
github.com/json-iterator/go.describeStruct(0xc00082fe60, 0x20f8c00, 0xc000802f60, 0x1ae42)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_extension.go:380 +0xa4f
github.com/json-iterator/go.decoderOfStruct(0xc00082fe60, 0x20f8c00, 0xc000802f60, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_struct_decoder.go:14 +0xb6
github.com/json-iterator/go._createDecoderOfType(0xc000823e60, 0x20f8c00, 0xc000802f60, 0xc00080c2c8, 0xc0001c2c60)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:176 +0x4aa
github.com/json-iterator/go.createDecoderOfType(0xc000823e60, 0x20f8c00, 0xc000802f60, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:142 +0x10b
github.com/json-iterator/go.decoderOfType(0xc00082fe60, 0x20f8c00, 0xc000802f60, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:124 +0x78
github.com/json-iterator/go.(*frozenConfig).DecoderOf(0xc000175ea0, 0x20f8ac0, 0xc000802ea0, 0x20f8ac0, 0xc000802ea0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:114 +0x111
github.com/json-iterator/go.(*Iterator).ReadVal(0xc00018c990, 0x1c55340, 0xc0003c0460)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:72 +0x1ce
github.com/json-iterator/go.(*frozenConfig).Unmarshal(0xc000175ea0, 0xc00067a600, 0xc0, 0x600, 0x1c55340, 0xc0003c0460, 0x0, 0x0)
/go/pkg/mod/github.com/json-iterator/go@v1.1.8/config.go:348 +0xd2
k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode(0xc00014ea40, 0xc00067a600, 0xc0, 0x600, 0x0, 0x2081f20, 0xc0003c0460, 0x1, 0x0, 0x206f080, ...)
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/runtime/serializer/json/json.go:275 +0x7d5
k8s.io/apimachinery/pkg/runtime.WithoutVersionDecoder.Decode(0x206e680, 0xc00014ea40, 0xc00067a600, 0xc0, 0x600, 0x0, 0x2081f20, 0xc0003c0460, 0xc000617100, 0xc000617000, ...)
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/runtime/helper.go:252 +0x97
k8s.io/client-go/rest.Result.Into(0xc00067a600, 0xc0, 0x600, 0xc00014a250, 0x10, 0x0, 0x0, 0xc8, 0x2070ec0, 0xc0002af7f0, ...)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:1133 +0xba
github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*customK8sResourceClient).List(0xc000488630, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00083a000, 0x0, 0x0, 0xffffffffffffffff, 0xc000830e70, 0x15cc0f5)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/customresource.go:287 +0xa5a
github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*networkPolicyClient).List(0xc0006639b0, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f240, 0x0, 0x0, 0xbfa4231b9311f897, 0x77f6c0d, 0x2f5ae40)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/networkpolicy.go:243 +0x4dc
github.com/projectcalico/libcalico-go/lib/backend/k8s.(*KubeClient).List(0xc000668b80, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f240, 0x0, 0x0, 0x42, 0x0, 0x0)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/k8s.go:592 +0x2f9
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).resyncAndCreateWatcher(0xc000116a20, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:203 +0x7fb
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc000116a20, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:81 +0xb9
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc000116a20)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43
created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c

goroutine 145 [select]:
golang.org/x/net/http2.(*ClientConn).roundTrip(0xc0000a1500, 0xc000616300, 0x0, 0x0, 0x0, 0x0)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1096 +0x9a0
golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc0002b9500, 0xc000616300, 0xc000674000, 0x72db95, 0x0, 0xc000207200)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:448 +0x172
golang.org/x/net/http2.(*Transport).RoundTrip(...)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:409
golang.org/x/net/http2.noDialH2RoundTripper.RoundTrip(0xc0002b9500, 0xc000616300, 0xc000140550, 0x5, 0xc000207288)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:2603 +0x3e
net/http.(*Transport).roundTrip(0xc0002b43c0, 0xc000616300, 0xc0004d0f10, 0xc0006743a8, 0x40000000040ef88)
/usr/local/go-cgo/src/net/http/transport.go:485 +0xdc1
net/http.(*Transport).RoundTrip(0xc0002b43c0, 0xc000616300, 0xc0002b43c0, 0x0, 0x0)
/usr/local/go-cgo/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc000616300, 0x206f100, 0xc0002b43c0, 0x0, 0x0, 0x0, 0xc000010098, 0xc0000502e0, 0x1, 0x0)
/usr/local/go-cgo/src/net/http/client.go:250 +0x443
net/http.(*Client).send(0xc0006637d0, 0xc000616300, 0x0, 0x0, 0x0, 0xc000010098, 0x0, 0x1, 0xc0004c5e60)
/usr/local/go-cgo/src/net/http/client.go:174 +0xfa
net/http.(*Client).do(0xc0006637d0, 0xc000616300, 0x0, 0x0, 0x0)
/usr/local/go-cgo/src/net/http/client.go:641 +0x3ce
net/http.(*Client).Do(0xc0006637d0, 0xc000616300, 0x0, 0x3, 0xc000140550)
/usr/local/go-cgo/src/net/http/client.go:509 +0x35
k8s.io/client-go/rest.(*Request).request(0xc0005e9380, 0xc000674948, 0x0, 0x0)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:754 +0x367
k8s.io/client-go/rest.(*Request).Do(0xc0005e9380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:828 +0xd8
github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*customK8sResourceClient).List(0xc0004886e0, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f280, 0x0, 0x0, 0xbfa4231b9336d048, 0x7a443d8, 0x2f5ae40)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/customresource.go:287 +0x9dc
github.com/projectcalico/libcalico-go/lib/backend/k8s.(*KubeClient).List(0xc000668b80, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f280, 0x0, 0x0, 0x3e, 0x0, 0x0)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/k8s.go:592 +0x2f9
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).resyncAndCreateWatcher(0xc000116ab0, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:203 +0x7fb
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc000116ab0, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:81 +0xb9
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc000116ab0)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43
created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c

goroutine 146 [runnable]:
fmt.glob..func1(0x2ed2a40, 0x3)
/usr/local/go-cgo/src/fmt/print.go:132 +0x2d
sync.(*Pool).Get(0x2ed2a40, 0x0, 0x0)
/usr/local/go-cgo/src/sync/pool.go:148 +0xa6
fmt.newPrinter(0x0)
/usr/local/go-cgo/src/fmt/print.go:137 +0x31
fmt.Sprint(0xc00077e920, 0x1, 0x1, 0x0, 0x0)
/usr/local/go-cgo/src/fmt/print.go:248 +0x26
github.com/sirupsen/logrus.(*Entry).Debug(0xc0004b35e0, 0xc00077e920, 0x1, 0x1)
/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/entry.go:143 +0x7d
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc000116b40, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:80 +0x8e
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc000116b40)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43
created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c

goroutine 147 [select]:
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc000116bd0, 0x20c15e0, 0xc00014f300)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:91 +0x211
github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc000116bd0)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43
created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c

goroutine 149 [runnable]:
github.com/projectcalico/felix/daemon.servePrometheusMetrics(0xc0004bb800)
/go/src/github.com/projectcalico/felix/daemon/daemon.go:622 +0x1c8
created by github.com/projectcalico/felix/daemon.Run
/go/src/github.com/projectcalico/felix/daemon/daemon.go:607 +0x2f68

goroutine 151 [chan receive]:
github.com/projectcalico/felix/logutils.RegisterProfilingSignalHandlers.func2(0xc000095aa0, 0xc0004bb800)
/go/src/github.com/projectcalico/felix/logutils/dump.go:106 +0x34
created by github.com/projectcalico/felix/logutils.RegisterProfilingSignalHandlers
/go/src/github.com/projectcalico/felix/logutils/dump.go:104 +0xdc

goroutine 60 [runnable]:
github.com/sirupsen/logrus.NewEntry(...)
/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/entry.go:52
github.com/sirupsen/logrus.(*Logger).newEntry(0xc0000d2120, 0xc0005f9cd8)
/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/logger.go:83 +0x56
github.com/sirupsen/logrus.(*Logger).Info(0xc0000d2120, 0xc0005eefb0, 0x1, 0x1)
/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/logger.go:188 +0x41
github.com/sirupsen/logrus.Info(...)
/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/exported.go:85
github.com/projectcalico/felix/daemon.monitorAndManageShutdown.func1(0x0, 0xc0001868a0)
/go/src/github.com/projectcalico/felix/daemon/daemon.go:658 +0x137
created by github.com/projectcalico/felix/daemon.monitorAndManageShutdown
/go/src/github.com/projectcalico/felix/daemon/daemon.go:656 +0x1a2

goroutine 63 [runnable]:
golang.org/x/net/http2.awaitRequestCancel(0xc000382c00, 0xc00048c5a0, 0x1, 0x0)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:288 +0x120
golang.org/x/net/http2.(*clientStream).awaitRequestCancel(0xc0004ef080, 0xc000382c00)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:314 +0x40
created by golang.org/x/net/http2.(*clientConnReadLoop).handleResponse
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1965 +0x6b0

goroutine 115 [runnable]:
golang.org/x/net/http2.(*clientStream).awaitRequestCancel(0xc000280dc0, 0xc000616900)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:313
created by golang.org/x/net/http2.(*clientConnReadLoop).handleResponse
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1965 +0x6b0

goroutine 116 [runnable]:
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000669240)
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/watch/streamwatcher.go:99
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/watch/streamwatcher.go:71 +0xbe

goroutine 117 [select]:
github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*k8sWatcherConverter).processK8sEvents(0xc000154a00)
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/watcher.go:103 +0x5fb
created by github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.newK8sWatcherConverterOneToMany
/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/watcher.go:60 +0x181

goroutine 162 [runnable]:
golang.org/x/net/http2.(*clientStream).awaitRequestCancel(0xc0002811e0, 0xc000616d00)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:313
created by golang.org/x/net/http2.(*clientConnReadLoop).handleResponse
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1965 +0x6b0

goroutine 163 [runnable]:
golang.org/x/net/http2.(*clientStream).awaitRequestCancel(0xc000281e40, 0xc000617100)
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:313
created by golang.org/x/net/http2.(*clientConnReadLoop).handleResponse
/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1965 +0x6b0

@ianlancetaylor ianlancetaylor changed the title Panic: runtime: unexpected return pc for runtime.heapBitsSetType runtime: panic: unexpected return pc for runtime.heapBitsSetType May 4, 2020
@ianlancetaylor ianlancetaylor added this to the Go1.15 milestone May 4, 2020
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented May 4, 2020

Is it feasible for you to test the current release (1.14.2)? Thanks.

@fasaxc
Copy link
Author

@fasaxc fasaxc commented May 5, 2020

Pretty easy to try but we've only seen this crash once so far; I don't have a reliable repro scenario. Any hints as to what might cause it?

@fasaxc
Copy link
Author

@fasaxc fasaxc commented May 7, 2020

Have done 5 or so runs on go v1.14 without repro yet. I did get another repro on 1.13.7 with the same panic but a different named function.

I checked how we use profiling and it looks like we don't enable by default; it's only enabled when sending a signal.

The area of code that seems to trigger the panic is deep in a library that's a transient dependency. Looks like we call through the kubernetes client to json-iter, which uses reflect2, which has a scary name that makes me wonder if it's doing something unsafe and trashing the stack somehow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.