Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic from fatal error: concurrent map read and map write #1101

Closed
adammw opened this issue Jul 23, 2020 · 6 comments
Closed

Panic from fatal error: concurrent map read and map write #1101

adammw opened this issue Jul 23, 2020 · 6 comments
Labels
bug

Comments

@adammw
Copy link
Contributor

adammw commented Jul 23, 2020

Running AWS CNI v1.6.2 on our own nodes. Panic seen in the logs causing a restart of the CNI container.

fatal error: concurrent map read and map write

goroutine 3726875 [running]:
runtime.throw(0x5629d4b94f4d, 0x21)
	/usr/local/go/src/runtime/panic.go:774 +0x74 fp=0xc001253290 sp=0xc001253260 pc=0x5629d3994f44
runtime.mapaccess2(0x5629d55c8460, 0xc000d25bf0, 0xc00152c110, 0xc00152c110, 0xc00088d20a)
	/usr/local/go/src/runtime/map.go:470 +0x27a fp=0xc0012532d8 sp=0xc001253290 pc=0x5629d39745fa
reflect.mapaccess(0x5629d55c8460, 0xc000d25bf0, 0xc00152c110, 0x5629d4b826b7)
	/usr/local/go/src/runtime/map.go:1319 +0x41 fp=0xc001253310 sp=0xc0012532d8 pc=0x5629d3976931
reflect.Value.MapIndex(0x5629d55c8460, 0xc001770058, 0x95, 0x5629d550f920, 0xc00152c110, 0x98, 0x5629d55da7c0, 0xc00117c9f0, 0x16)
	/usr/local/go/src/reflect/value.go:1176 +0x16f fp=0xc001253388 sp=0xc001253310 pc=0x5629d3a2bebf
encoding/json.mapEncoder.encode(0xc000f36910, 0xc0003be000, 0x5629d55c8460, 0xc001770058, 0x95, 0x100)
	/usr/local/go/src/encoding/json/encode.go:706 +0x304 fp=0xc0012534d0 sp=0xc001253388 pc=0x5629d3ac8754
encoding/json.mapEncoder.encode-fm(0xc0003be000, 0x5629d55c8460, 0xc001770058, 0x95, 0x100)
	/usr/local/go/src/encoding/json/encode.go:682 +0x66 fp=0xc001253510 sp=0xc0012534d0 pc=0x5629d3ad4286
encoding/json.structEncoder.encode(0xc000a12000, 0x5, 0x8, 0xc000a5ba10, 0xc0003be000, 0x5629d57b1e20, 0xc001770000, 0x99, 0x100)
	/usr/local/go/src/encoding/json/encode.go:664 +0x308 fp=0xc0012535b0 sp=0xc001253510 pc=0x5629d3ac8208
encoding/json.structEncoder.encode-fm(0xc0003be000, 0x5629d57b1e20, 0xc001770000, 0x99, 0xc001520100)
	/usr/local/go/src/encoding/json/encode.go:635 +0x81 fp=0xc001253608 sp=0xc0012535b0 pc=0x5629d3ad4201
encoding/json.mapEncoder.encode(0xc000a5ba40, 0xc0003be000, 0x5629d55c98a0, 0xc0009eca90, 0x195, 0x100)
	/usr/local/go/src/encoding/json/encode.go:706 +0x353 fp=0xc001253750 sp=0xc001253608 pc=0x5629d3ac87a3
encoding/json.mapEncoder.encode-fm(0xc0003be000, 0x5629d55c98a0, 0xc0009eca90, 0x195, 0x100)
	/usr/local/go/src/encoding/json/encode.go:682 +0x66 fp=0xc001253790 sp=0xc001253750 pc=0x5629d3ad4286
encoding/json.structEncoder.encode(0xc000242240, 0x3, 0x4, 0xc000a5ba70, 0xc0003be000, 0x5629d56e1640, 0xc0009eca80, 0x199, 0x5629d3ac0100)
	/usr/local/go/src/encoding/json/encode.go:664 +0x308 fp=0xc001253830 sp=0xc001253790 pc=0x5629d3ac8208
encoding/json.structEncoder.encode-fm(0xc0003be000, 0x5629d56e1640, 0xc0009eca80, 0x199, 0xc0009e0100)
	/usr/local/go/src/encoding/json/encode.go:635 +0x81 fp=0xc001253888 sp=0xc001253830 pc=0x5629d3ad4201
encoding/json.ptrEncoder.encode(0xc000a5baa0, 0xc0003be000, 0x5629d54a61a0, 0xc0009eca80, 0x16, 0x5629d54a0100)
	/usr/local/go/src/encoding/json/encode.go:810 +0xb3 fp=0xc0012538d0 sp=0xc001253888 pc=0x5629d3ac96a3
encoding/json.ptrEncoder.encode-fm(0xc0003be000, 0x5629d54a61a0, 0xc0009eca80, 0x16, 0x7f07c9f70100)
	/usr/local/go/src/encoding/json/encode.go:805 +0x66 fp=0xc001253910 sp=0xc0012538d0 pc=0x5629d3ad4406
encoding/json.(*encodeState).reflectValue(0xc0003be000, 0x5629d54a61a0, 0xc0009eca80, 0x16, 0x5629d3970100)
	/usr/local/go/src/encoding/json/encode.go:337 +0x84 fp=0xc001253948 sp=0xc001253910 pc=0x5629d3ac5a24
encoding/json.(*encodeState).marshal(0xc0003be000, 0x5629d54a61a0, 0xc0009eca80, 0x5629d65c0100, 0x0, 0x0)
	/usr/local/go/src/encoding/json/encode.go:309 +0x10d fp=0xc0012539c8 sp=0xc001253948 pc=0x5629d3ac54cd
encoding/json.Marshal(0x5629d54a61a0, 0xc0009eca80, 0x5629d656f441, 0xc00153e100, 0x0, 0xc00118bab8, 0x5629d454d34c)
	/usr/local/go/src/encoding/json/encode.go:161 +0x54 fp=0xc001253a40 sp=0xc0012539c8 pc=0x5629d3ac4a84
github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd.eniV1RequestHandler.func1(0x5629d590cc00, 0xc0011d00e0, 0xc0008b6000)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd/introspect.go:134 +0x44 fp=0xc001253aa0 sp=0xc001253a40 pc=0x5629d4b670a4
net/http.HandlerFunc.ServeHTTP(0xc000f1a100, 0x5629d590cc00, 0xc0011d00e0, 0xc0008b6000)
	/usr/local/go/src/net/http/server.go:2007 +0x46 fp=0xc001253ac8 sp=0xc001253aa0 pc=0x5629d3d71ec6
net/http.(*ServeMux).ServeHTTP(0xc0003a8400, 0x5629d590cc00, 0xc0011d00e0, 0xc0008b6000)
	/usr/local/go/src/net/http/server.go:2387 +0x1bf fp=0xc001253b28 sp=0xc001253ac8 pc=0x5629d3d73dcf
github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd.LoggingHandler.ServeHTTP(0x5629d58d4900, 0xc0003a8400, 0x5629d590cc00, 0xc0011d00e0, 0xc0008b6000)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd/introspect.go:51 +0x17c fp=0xc001253b80 sp=0xc001253b28 pc=0x5629d4b58b3c
github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd.(*LoggingHandler).ServeHTTP(0xc000f1a1d0, 0x5629d590cc00, 0xc0011d00e0, 0xc0008b6000)
	<autogenerated>:1 +0x65 fp=0xc001253bb8 sp=0xc001253b80 pc=0x5629d4b682f5
net/http.(*ServeMux).ServeHTTP(0xc0003a8440, 0x5629d590cc00, 0xc0011d00e0, 0xc0008b6000)
	/usr/local/go/src/net/http/server.go:2387 +0x1bf fp=0xc001253c18 sp=0xc001253bb8 pc=0x5629d3d73dcf
net/http.serverHandler.ServeHTTP(0xc00064a540, 0x5629d590cc00, 0xc0011d00e0, 0xc0008b6000)
	/usr/local/go/src/net/http/server.go:2802 +0xa6 fp=0xc001253c48 sp=0xc001253c18 pc=0x5629d3d75346
net/http.(*conn).serve(0xc000b78000, 0x5629d5912640, 0xc00104c000)
	/usr/local/go/src/net/http/server.go:1890 +0x877 fp=0xc001253fc8 sp=0xc001253c48 pc=0x5629d3d70ca7
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc001253fd0 sp=0xc001253fc8 pc=0x5629d39c2551
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2928 +0x386

goroutine 1 [IO wait]:
internal/poll.runtime_pollWait(0x7f07c9f0cbc8, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc0006f7998, 0x72, 0x0, 0x0, 0x5629d4b6cbcf)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0006f7980, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1fa
net.(*netFD).accept(0xc0006f7980, 0x37e11d600, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x44
net.(*TCPListener).accept(0xc000e645c0, 0xc000b20c58, 0xc000b20c60, 0x18)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x34
net.(*TCPListener).Accept(0xc000e645c0, 0x5629d5887e78, 0xc00010c340, 0x5629d5926980, 0xc0007aa3b0)
	/usr/local/go/src/net/tcpsock.go:261 +0x49
google.golang.org/grpc.(*Server).Serve(0xc00010c340, 0x5629d590c940, 0xc000e645c0, 0x0, 0x0)
	/go/pkg/mod/google.golang.org/grpc@v1.29.0/server.go:621 +0x230
github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd.(*IPAMContext).RunRPCHandler(0xc000178900, 0x5629d58866e8, 0xc000178900)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd/rpc_handler.go:124 +0x3fa
main._main(0xc0000fc058)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/cmd/aws-k8s-agent/main.go:73 +0x46f
main.main()
	/go/src/github.com/aws/amazon-vpc-cni-k8s/cmd/aws-k8s-agent/main.go:30 +0x24

goroutine 6 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x5629d65c4ba0)
	/go/pkg/mod/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x8d
created by github.com/golang/glog.init.0
	/go/pkg/mod/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x271

goroutine 44 [syscall, 10215 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:147 +0x9e
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x24
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:29 +0x43

goroutine 72 [IO wait]:
internal/poll.runtime_pollWait(0x7f07c9f0cf08, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc00068c318, 0x72, 0xcf00, 0xcfde, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00068c300, 0xc001086000, 0xcfde, 0xcfde, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc00068c300, 0xc001086000, 0xcfde, 0xcfde, 0x203000, 0x0, 0xcfd9)
	/usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc00000e008, 0xc001086000, 0xcfde, 0xcfde, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x6a
crypto/tls.(*atLeastReader).Read(0xc000bdade0, 0xc001086000, 0xcfde, 0xcfde, 0x4000, 0x800, 0xc000b1c970)
	/usr/local/go/src/crypto/tls/conn.go:780 +0x62
bytes.(*Buffer).ReadFrom(0xc0000c0cd8, 0x5629d58d28a0, 0xc000bdade0, 0x5629d39706b7, 0x5629d5633960, 0x5629d5823400)
	/usr/local/go/src/bytes/buffer.go:204 +0xb6
crypto/tls.(*Conn).readFromUntil(0xc0000c0a80, 0x5629d58d4860, 0xc00000e008, 0x5, 0xc00000e008, 0x203000)
	/usr/local/go/src/crypto/tls/conn.go:802 +0xee
crypto/tls.(*Conn).readRecordOrCCS(0xc0000c0a80, 0x0, 0x0, 0xc000bdadc0)
	/usr/local/go/src/crypto/tls/conn.go:609 +0x126
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc0000c0a80, 0xc00002c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1255 +0x163
bufio.(*Reader).Read(0xc000144de0, 0xc00064a498, 0x9, 0x9, 0x2863, 0x38d7, 0x0)
	/usr/local/go/src/bufio/bufio.go:226 +0x26c
io.ReadAtLeast(0x5629d58d2700, 0xc000144de0, 0xc00064a498, 0x9, 0x9, 0x9, 0x0, 0x0, 0xc000b1cf78)
	/usr/local/go/src/io/io.go:310 +0x89
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc00064a498, 0x9, 0x9, 0x5629d58d2700, 0xc000144de0, 0x0, 0x562900000000, 0xc000bd4e40, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/frame.go:237 +0x89
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00064a460, 0xc000bd4e40, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/frame.go:492 +0xa3
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000b1cfb8, 0xc00047ffa8, 0x5629d3dafaba)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1746 +0x8e
golang.org/x/net/http2.(*ClientConn).readLoop(0xc000001b00)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1674 +0xa5
created by golang.org/x/net/http2.(*Transport).newClientConn
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:674 +0x631

goroutine 239 [sync.Cond.Wait]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0006478e8, 0x15d86)
	/usr/local/go/src/runtime/sema.go:510 +0xfc
sync.(*Cond).Wait(0xc0006478d8)
	/usr/local/go/src/sync/cond.go:56 +0x9f
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0006478c0, 0xc000431d40, 0x0, 0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/delta_fifo.go:431 +0xa8
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc000441d00)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/controller.go:150 +0x42
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000344fb0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:133 +0x60
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000344fb0, 0x3b9aca00, 0x0, 0xc00055ae01, 0xc000508780)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:134 +0xfa
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*controller).Run(0xc000441d00, 0xc000508780)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/controller.go:124 +0x2ff
created by github.com/aws/amazon-vpc-cni-k8s/pkg/k8sapi.(*Controller).run
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/k8sapi/discovery.go:309 +0xdf

goroutine 217 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc000518aa0, 0x0, 0xc0003ab080, 0xc00041bf20)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:278 +0x17a
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:272 +0x995

goroutine 546 [runnable]:
github.com/aws/aws-sdk-go/aws/request.New(0x0, 0xc0007d0510, 0x0, 0x5629d58d5360, 0xc00048e240, 0x0, 0xc0004d1670, 0x0, 0x5629d65c42c0, 0xc0006d78f8, ...)
	/go/pkg/mod/github.com/aws/aws-sdk-go@v1.30.4/aws/request/request.go:117 +0x79b
github.com/aws/aws-sdk-go/aws/client.(*Client).NewRequest(...)
	/go/pkg/mod/github.com/aws/aws-sdk-go@v1.30.4/aws/client/client.go:85
github.com/aws/aws-sdk-go/service/ec2.(*EC2).newRequest(0xc0004d6388, 0xc00104c180, 0x5629d56bdca0, 0xc000a060e0, 0x5629d561f4a0, 0x5629d65e2098, 0x0)
	/go/pkg/mod/github.com/aws/aws-sdk-go@v1.30.4/service/ec2/service.go:90 +0x191
github.com/aws/aws-sdk-go/service/ec2.(*EC2).UnassignPrivateIpAddressesRequest(0xc0004d6388, 0xc000a060e0, 0x8, 0x5629d5526060)
	/go/pkg/mod/github.com/aws/aws-sdk-go@v1.30.4/service/ec2/api.go:36762 +0xe1
github.com/aws/aws-sdk-go/service/ec2.(*EC2).UnassignPrivateIpAddressesWithContext(0xc0004d6388, 0x5629d5912680, 0xc000042048, 0xc000a060e0, 0xc00014c038, 0x1, 0x1, 0x1, 0xc000f2a3d0, 0xc000f2a3c0)
	/go/pkg/mod/github.com/aws/aws-sdk-go@v1.30.4/service/ec2/api.go:36793 +0x3b
github.com/aws/amazon-vpc-cni-k8s/pkg/awsutils.(*EC2InstanceMetadataCache).DeallocIPAddresses(0xc000337000, 0xc000fe0600, 0x15, 0xc000f2a3e0, 0x1, 0x1, 0x0, 0x1)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/awsutils/awsutils.go:1147 +0x3a7
github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd.(*IPAMContext).tryUnassignIPsFromAll(0xc000178900)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd/ipamd.go:600 +0x2a4
github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd.(*IPAMContext).decreaseIPPool(0xc000178900, 0x6fc23ac00)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd/ipamd.go:538 +0x308
github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd.(*IPAMContext).updateIPPoolIfRequired(0xc000178900)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd/ipamd.go:516 +0xb6
github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd.(*IPAMContext).StartNodeIPPoolManager(0xc000178900)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd/ipamd.go:506 +0x3b
created by main._main
	/go/src/github.com/aws/amazon-vpc-cni-k8s/cmd/aws-k8s-agent/main.go:64 +0x41d

goroutine 549 [chan receive, 10215 minutes]:
github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd.(*IPAMContext).shutdownListener(0xc000178900)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd/rpc_handler.go:141 +0x119
created by github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd.(*IPAMContext).RunRPCHandler
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd/rpc_handler.go:123 +0x3d8

goroutine 548 [IO wait]:
internal/poll.runtime_pollWait(0x7f07c9f0caf8, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc0003c4118, 0x72, 0x0, 0x0, 0x5629d4b6cbcf)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0003c4100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1fa
net.(*netFD).accept(0xc0003c4100, 0xc00033fd08, 0x5629d3d762f6, 0xc00064a5e0)
	/usr/local/go/src/net/fd_unix.go:238 +0x44
net.(*TCPListener).accept(0xc000f3a1a0, 0x5f11f560, 0xc00033fd08, 0x5629d39ee008)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x34
net.(*TCPListener).Accept(0xc000f3a1a0, 0xc00033fd58, 0x18, 0xc0004b1c80, 0x5629d3d75816)
	/usr/local/go/src/net/tcpsock.go:261 +0x49
net/http.(*Server).Serve(0xc00064a540, 0x5629d590c940, 0xc000f3a1a0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2896 +0x282
github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd.(*IPAMContext).ServeIntrospection.func1(0x0, 0x0)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd/introspect.go:75 +0xe6
github.com/aws/amazon-vpc-cni-k8s/pkg/utils/retry.RetryWithBackoffCtx(0x5629d5912680, 0xc000042048, 0x5629d58e1b40, 0xc001114180, 0xc00033ffb8, 0x0, 0x0)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/utils/retry/retry.go:43 +0x87
github.com/aws/amazon-vpc-cni-k8s/pkg/utils/retry.RetryWithBackoff(...)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/utils/retry/retry.go:28
github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd.(*IPAMContext).ServeIntrospection(0xc000178900)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd/introspect.go:63 +0x105
created by main._main
	/go/src/github.com/aws/amazon-vpc-cni-k8s/cmd/aws-k8s-agent/main.go:70 +0x461

goroutine 547 [IO wait]:
internal/poll.runtime_pollWait(0x7f07c9f0ca28, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc0003c4018, 0x72, 0x0, 0x0, 0x5629d4b6cbcf)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0003c4000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1fa
net.(*netFD).accept(0xc0003c4000, 0xc000b21cb8, 0x5629d3d762f6, 0xc00064a180)
	/usr/local/go/src/net/fd_unix.go:238 +0x44
net.(*TCPListener).accept(0xc000d01c80, 0x5f11f55a, 0xc000b21cb8, 0x5629d39ee008)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x34
net.(*TCPListener).Accept(0xc000d01c80, 0xc000b21d08, 0x18, 0xc00048c300, 0x5629d3d75816)
	/usr/local/go/src/net/tcpsock.go:261 +0x49
net/http.(*Server).Serve(0xc00064a0e0, 0x5629d590c940, 0xc000d01c80, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2896 +0x282
net/http.(*Server).ListenAndServe(0xc00064a0e0, 0x0, 0x30)
	/usr/local/go/src/net/http/server.go:2825 +0xb9
github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd.(*IPAMContext).ServeMetrics.func1(0x0, 0x0)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd/metrics.go:46 +0x39
github.com/aws/amazon-vpc-cni-k8s/pkg/utils/retry.RetryWithBackoffCtx(0x5629d5912680, 0xc000042048, 0x5629d58e1b40, 0xc001114150, 0xc000b21fb0, 0x1, 0x0)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/utils/retry/retry.go:43 +0x87
github.com/aws/amazon-vpc-cni-k8s/pkg/utils/retry.RetryWithBackoff(...)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/utils/retry/retry.go:28
github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd.(*IPAMContext).ServeMetrics(0xc000178900)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd/metrics.go:45 +0x1a0
created by main._main
	/go/src/github.com/aws/amazon-vpc-cni-k8s/cmd/aws-k8s-agent/main.go:67 +0x43f

goroutine 274 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc00034e000)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/shared_informer.go:517 +0x18f
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00051ea70, 0xc000228010)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:71 +0x5b
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:69 +0x64

goroutine 191 [chan receive, 10215 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc00051e620, 0xc00072e4e0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/shared_informer.go:425 +0x4b
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:54 +0x30
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000042e40, 0xc000684ba0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:71 +0x5b
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:69 +0x64

goroutine 107 [chan receive, 1 minutes]:
github.com/operator-framework/operator-sdk/pkg/k8sclient.(*resourceClientFactory).runBackgroundCacheReset.func1(0xc000696410, 0xc000430ed0)
	/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.0.7/pkg/k8sclient/client.go:162 +0x7a
created by github.com/operator-framework/operator-sdk/pkg/k8sclient.(*resourceClientFactory).runBackgroundCacheReset
	/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.0.7/pkg/k8sclient/client.go:161 +0x59

goroutine 111 [chan receive (nil chan), 10215 minutes]:
github.com/operator-framework/operator-sdk/pkg/sdk.Run(0x5629d5912680, 0xc000042070)
	/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.0.7/pkg/sdk/api.go:72 +0xcc
github.com/aws/amazon-vpc-cni-k8s/pkg/eniconfig.(*ENIConfigController).Start(0xc0000dc660)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/eniconfig/eniconfig.go:160 +0x24b
created by main._main
	/go/src/github.com/aws/amazon-vpc-cni-k8s/cmd/aws-k8s-agent/main.go:53 +0x577

goroutine 237 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000696c80)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/util/workqueue/delaying_queue.go:206 +0x346
created by k8s.io/client-go/util/workqueue.newDelayingQueue
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/util/workqueue/delaying_queue.go:54 +0x29c

goroutine 110 [select (no cases), 10215 minutes]:
github.com/aws/amazon-vpc-cni-k8s/pkg/k8sapi.(*Controller).DiscoverK8SPods(0xc00051e0e0, 0xc000146f40)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/k8sapi/discovery.go:170 +0x536
github.com/aws/amazon-vpc-cni-k8s/pkg/k8sapi.(*Controller).DiscoverLocalK8SPods(0xc00051e0e0)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/k8sapi/discovery.go:127 +0x164
created by main._main
	/go/src/github.com/aws/amazon-vpc-cni-k8s/cmd/aws-k8s-agent/main.go:49 +0x2eb

goroutine 189 [runnable]:
k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc00072e480, 0x5f5e100, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:374 +0x190
created by k8s.io/apimachinery/pkg/util/wait.poller.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:357 +0x8e

goroutine 238 [chan receive, 10215 minutes]:
github.com/aws/amazon-vpc-cni-k8s/pkg/k8sapi.(*Controller).run(0xc00051e0e0, 0x1, 0xc000508780)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/k8sapi/discovery.go:326 +0x2a8
created by github.com/aws/amazon-vpc-cni-k8s/pkg/k8sapi.(*Controller).DiscoverK8SPods
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/k8sapi/discovery.go:167 +0x531

goroutine 188 [sync.Cond.Wait]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0001f0188, 0x1de2a)
	/usr/local/go/src/runtime/sema.go:510 +0xfc
sync.(*Cond).Wait(0xc0001f0178)
	/usr/local/go/src/sync/cond.go:56 +0x9f
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0001f0160, 0xc00066e860, 0x0, 0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/delta_fifo.go:431 +0xa8
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc0003c5200)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/controller.go:150 +0x42
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0007c6e00)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:133 +0x60
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0007c6e00, 0x3b9aca00, 0x0, 0xc0004d6101, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:134 +0xfa
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*controller).Run(0xc0003c5200, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/controller.go:124 +0x2ff
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc000236f30, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/shared_informer.go:227 +0x481
created by github.com/operator-framework/operator-sdk/pkg/sdk.(*informer).Run
	/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.0.7/pkg/sdk/informer.go:91 +0x174

goroutine 240 [runnable]:
k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc0005087e0, 0x5f5e100, 0x0, 0xc000508780)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:374 +0x190
created by k8s.io/apimachinery/pkg/util/wait.poller.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:357 +0x8e

goroutine 241 [IO wait, 10215 minutes]:
internal/poll.runtime_pollWait(0x7f07c9f0cd68, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc000492018, 0x72, 0x0, 0x0, 0x5629d4b6cbcf)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000492000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1fa
net.(*netFD).accept(0xc000492000, 0xc00033ed50, 0xc00040ea80, 0x7f07c9f686d0)
	/usr/local/go/src/net/fd_unix.go:238 +0x44
net.(*TCPListener).accept(0xc0004da060, 0xc00033ed80, 0x5629d3972fba, 0x30)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x34
net.(*TCPListener).Accept(0xc0004da060, 0x5629d5762b80, 0xc0003e07b0, 0x5629d55e3100, 0x5629d654a250)
	/usr/local/go/src/net/tcpsock.go:261 +0x49
net/http.(*Server).Serve(0xc00064a000, 0x5629d590c940, 0xc0004da060, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2896 +0x282
net/http.(*Server).ListenAndServe(0xc00064a000, 0xc00064a000, 0x5629d58d2a40)
	/usr/local/go/src/net/http/server.go:2825 +0xb9
net/http.ListenAndServe(0xc000470ce6, 0x6, 0x0, 0x0, 0x6, 0xc000521f40)
	/usr/local/go/src/net/http/server.go:3081 +0x76
created by github.com/operator-framework/operator-sdk/pkg/sdk.ExposeMetricsPort
	/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.0.7/pkg/sdk/metrics.go:30 +0x113

goroutine 243 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000697040)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/util/workqueue/delaying_queue.go:206 +0x346
created by k8s.io/client-go/util/workqueue.newDelayingQueue
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/util/workqueue/delaying_queue.go:54 +0x29c

goroutine 244 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000697360)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/util/workqueue/delaying_queue.go:206 +0x346
created by k8s.io/client-go/util/workqueue.newDelayingQueue
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/util/workqueue/delaying_queue.go:54 +0x29c

goroutine 245 [chan receive (nil chan), 10215 minutes]:
github.com/operator-framework/operator-sdk/pkg/sdk.(*informer).Run(0xc00051e540, 0x5629d5912680, 0xc000042070)
	/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.0.7/pkg/sdk/informer.go:100 +0x323
created by github.com/operator-framework/operator-sdk/pkg/sdk.Run
	/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.0.7/pkg/sdk/api.go:70 +0x8f

goroutine 246 [chan receive (nil chan), 10215 minutes]:
github.com/operator-framework/operator-sdk/pkg/sdk.(*informer).Run(0xc00051e9a0, 0x5629d5912680, 0xc000042070)
	/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.0.7/pkg/sdk/informer.go:100 +0x323
created by github.com/operator-framework/operator-sdk/pkg/sdk.Run
	/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.0.7/pkg/sdk/api.go:70 +0x8f

goroutine 247 [sync.Cond.Wait]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0006a8028, 0x2ce8b)
	/usr/local/go/src/runtime/sema.go:510 +0xfc
sync.(*Cond).Wait(0xc0006a8018)
	/usr/local/go/src/sync/cond.go:56 +0x9f
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0006a8000, 0xc000129b60, 0x0, 0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/delta_fifo.go:431 +0xa8
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc0003ec000)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/controller.go:150 +0x42
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000796e00)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:133 +0x60
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000796e00, 0x3b9aca00, 0x0, 0xc0004d6201, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:134 +0xfa
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*controller).Run(0xc0003ec000, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/controller.go:124 +0x2ff
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc000236fc0, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/shared_informer.go:227 +0x481
created by github.com/operator-framework/operator-sdk/pkg/sdk.(*informer).Run
	/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.0.7/pkg/sdk/informer.go:91 +0x174

goroutine 248 [runnable]:
k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc000508c00, 0x5f5e100, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:374 +0x190
created by k8s.io/apimachinery/pkg/util/wait.poller.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:357 +0x8e

goroutine 249 [chan receive, 10215 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1(0xc000508780, 0xc000441d00)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/controller.go:103 +0x36
created by k8s.io/client-go/tools/cache.(*controller).Run
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/controller.go:102 +0xb0

goroutine 250 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc0001770e0, 0x5629d58e7a80, 0xc001407d40, 0xc000947be0, 0xc000681c80, 0xc000508780, 0x0, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:373 +0x270
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0001770e0, 0xc000508780, 0x0, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:339 +0xf31
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:204 +0x35
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00047fef0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:133 +0x60
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000947ef0, 0x3b9aca00, 0x0, 0x5629d450eb01, 0xc000508780)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:134 +0xfa
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0001770e0, 0xc000508780)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:203 +0x16b
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:54 +0x30
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0006d7320, 0xc000147e40)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:71 +0x5b
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:69 +0x64

goroutine 273 [chan receive]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1.1(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/shared_informer.go:549 +0x56
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x989680, 0x3ff0000000000000, 0x3fb999999999999a, 0x5, 0xc000790dd8, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:203 +0xe0
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/shared_informer.go:548 +0x8b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0006bbf40)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:133 +0x60
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000790f40, 0xdf8475800, 0x0, 0x5629d3994501, 0xc0006da060)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:134 +0xfa
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*processorListener).run(0xc00034e000)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/shared_informer.go:546 +0x9d
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00051ea70, 0xc000228000)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:71 +0x5b
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:69 +0x64

goroutine 258 [chan receive, 10215 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc00051ea10, 0xc0006ac000)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/shared_informer.go:425 +0x4b
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:54 +0x30
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00041c010, 0xc000392040)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:71 +0x5b
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:69 +0x64

goroutine 259 [chan receive (nil chan), 10215 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1(0x0, 0xc0003ec000)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/controller.go:103 +0x36
created by k8s.io/client-go/tools/cache.(*controller).Run
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/controller.go:102 +0xb0

goroutine 260 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc00011c320, 0x5629d58e7a80, 0xc000d30480, 0xc0004a1be0, 0xc000dd2000, 0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:373 +0x270
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc00011c320, 0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:339 +0xf31
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:204 +0x35
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00048aef0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:133 +0x60
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0004a1ef0, 0x3b9aca00, 0x0, 0x1, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:134 +0xfa
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc00011c320, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:203 +0x16b
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:54 +0x30
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00041c040, 0xc0003920a0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:71 +0x5b
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:69 +0x64

goroutine 275 [chan receive]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1.1(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/shared_informer.go:549 +0x56
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x989680, 0x3ff0000000000000, 0x3fb999999999999a, 0x5, 0xc000792dd8, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:203 +0xe0
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/shared_informer.go:548 +0x8b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0006bbf40)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:133 +0x60
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000792f40, 0xdf8475800, 0x0, 0x5629d3994501, 0xc0006da0c0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:134 +0xfa
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000441f80)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/shared_informer.go:546 +0x9d
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00051e680, 0xc000228020)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:71 +0x5b
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:69 +0x64

goroutine 276 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc000441f80)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/shared_informer.go:517 +0x18f
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00051e680, 0xc000228030)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:71 +0x5b
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:69 +0x64

goroutine 192 [chan receive (nil chan), 10215 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1(0x0, 0xc0003c5200)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/controller.go:103 +0x36
created by k8s.io/client-go/tools/cache.(*controller).Run
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/controller.go:102 +0xb0

goroutine 289 [select, 6 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc000518aa0, 0x5629d58e7a80, 0xc000a5b830, 0xc001591be0, 0xc00041bf20, 0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:373 +0x270
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000518aa0, 0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:339 +0xf31
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:204 +0x35
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000483ef0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:133 +0x60
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc001591ef0, 0x3b9aca00, 0x0, 0x1, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:134 +0xfa
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000518aa0, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:203 +0x16b
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:54 +0x30
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000043030, 0xc000684bc0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:71 +0x5b
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:69 +0x64

goroutine 278 [sync.Cond.Wait]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0000fb0d0, 0xc00001de8a)
	/usr/local/go/src/runtime/sema.go:510 +0xfc
sync.(*Cond).Wait(0xc0000fb0c0)
	/usr/local/go/src/sync/cond.go:56 +0x9f
k8s.io/client-go/util/workqueue.(*Type).Get(0xc000696fa0, 0x0, 0x0, 0x5629d550f900)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/util/workqueue/queue.go:124 +0xa2
github.com/operator-framework/operator-sdk/pkg/sdk.(*informer).processNextItem(0xc00051e540, 0x0)
	/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.0.7/pkg/sdk/informer-sync.go:42 +0x53
github.com/operator-framework/operator-sdk/pkg/sdk.(*informer).runWorker(0xc00051e540)
	/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.0.7/pkg/sdk/informer-sync.go:36 +0x2d
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0002280e0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:133 +0x60
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0002280e0, 0x3b9aca00, 0x0, 0x1, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:134 +0xfa
k8s.io/apimachinery/pkg/util/wait.Until(0xc0002280e0, 0x3b9aca00, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:88 +0x4f
created by github.com/operator-framework/operator-sdk/pkg/sdk.(*informer).Run
	/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.0.7/pkg/sdk/informer.go:98 +0x25e

goroutine 337 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc00011c320, 0x0, 0xc000dae2a0, 0xc000dd2000)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:278 +0x17a
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:272 +0x995

goroutine 218 [sync.Cond.Wait]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc000678510, 0xc00002cee6)
	/usr/local/go/src/runtime/sema.go:510 +0xfc
sync.(*Cond).Wait(0xc000678500)
	/usr/local/go/src/sync/cond.go:56 +0x9f
k8s.io/client-go/util/workqueue.(*Type).Get(0xc000697270, 0x0, 0x0, 0x5629d550f900)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/util/workqueue/queue.go:124 +0xa2
github.com/operator-framework/operator-sdk/pkg/sdk.(*informer).processNextItem(0xc00051e9a0, 0xc00023fe00)
	/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.0.7/pkg/sdk/informer-sync.go:42 +0x53
github.com/operator-framework/operator-sdk/pkg/sdk.(*informer).runWorker(0xc00051e9a0)
	/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.0.7/pkg/sdk/informer-sync.go:36 +0x2d
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000cda7e0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:133 +0x60
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000cda7e0, 0x3b9aca00, 0x0, 0x1, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:134 +0xfa
k8s.io/apimachinery/pkg/util/wait.Until(0xc000cda7e0, 0x3b9aca00, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:88 +0x4f
created by github.com/operator-framework/operator-sdk/pkg/sdk.(*informer).Run
	/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.0.7/pkg/sdk/informer.go:98 +0x25e

goroutine 308 [select, 10215 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc0001770e0, 0xc000508780, 0xc0006ac240, 0xc000681c80)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:278 +0x17a
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:272 +0x995

goroutine 353 [sync.Cond.Wait]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc000521bd0, 0xc000015d55)
	/usr/local/go/src/runtime/sema.go:510 +0xfc
sync.(*Cond).Wait(0xc000521bc0)
	/usr/local/go/src/sync/cond.go:56 +0x9f
k8s.io/client-go/util/workqueue.(*Type).Get(0xc000696be0, 0x0, 0x0, 0x5629d550f900)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/util/workqueue/queue.go:124 +0xa2
github.com/aws/amazon-vpc-cni-k8s/pkg/k8sapi.(*Controller).processNextItem(0xc00051e0e0, 0x0)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/k8sapi/discovery.go:206 +0x57
github.com/aws/amazon-vpc-cni-k8s/pkg/k8sapi.(*Controller).runWorker(0xc00051e0e0)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/k8sapi/discovery.go:331 +0x2d
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000fe62d0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:133 +0x60
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000fe62d0, 0x3b9aca00, 0x0, 0x1, 0xc000508780)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:134 +0xfa
k8s.io/apimachinery/pkg/util/wait.Until(0xc000fe62d0, 0x3b9aca00, 0xc000508780)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:88 +0x4f
created by github.com/aws/amazon-vpc-cni-k8s/pkg/k8sapi.(*Controller).run
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/k8sapi/discovery.go:323 +0x216

goroutine 433 [select, 33 minutes]:
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000fb80e0, 0x34630b8a000, 0x0, 0x1, 0xc0000fc360)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:145 +0x183
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:88
k8s.io/apimachinery/pkg/util/wait.Forever(0xc000fb80e0, 0x34630b8a000)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:79 +0x51
created by github.com/aws/amazon-vpc-cni-k8s/pkg/awsutils.New
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/awsutils/awsutils.go:258 +0x50c

goroutine 526 [chan receive, 10215 minutes]:
gopkg.in/natefinch/lumberjack%2ev2.(*Logger).millRun(0xc00041ad80)
	/go/pkg/mod/gopkg.in/natefinch/lumberjack.v2@v2.0.0/lumberjack.go:379 +0x5a
created by gopkg.in/natefinch/lumberjack%2ev2.(*Logger).mill.func1
	/go/pkg/mod/gopkg.in/natefinch/lumberjack.v2@v2.0.0/lumberjack.go:390 +0x7b

goroutine 3736876 [runnable]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc000143a60, 0xc000000013)
	/usr/local/go/src/runtime/sema.go:510 +0xfc
sync.(*Cond).Wait(0xc000143a50)
	/usr/local/go/src/sync/cond.go:56 +0x9f
golang.org/x/net/http2.(*pipe).Read(0xc000143a48, 0xc000cf6001, 0x3dff, 0x3dff, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/pipe.go:65 +0xa8
golang.org/x/net/http2.transportResponseBody.Read(0xc000143a20, 0xc000cf6001, 0x3dff, 0x3dff, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:2019 +0xae
encoding/json.(*Decoder).refill(0xc000143e40, 0xc00093be0a, 0x9)
	/usr/local/go/src/encoding/json/stream.go:161 +0xed
encoding/json.(*Decoder).readValue(0xc000143e40, 0x0, 0x0, 0x5629d55e77e0)
	/usr/local/go/src/encoding/json/stream.go:136 +0x1de
encoding/json.(*Decoder).Decode(0xc000143e40, 0x5629d561ca20, 0xc00093be40, 0x0, 0x0)
	/usr/local/go/src/encoding/json/stream.go:63 +0x7b
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc001407d10, 0xc001594000, 0x4000, 0x4a80, 0x5629d58e5c00, 0x0, 0x38)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/framer/framer.go:150 +0x1a3
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc0014399a0, 0x0, 0x5629d58e7500, 0xc000d70500, 0x0, 0x0, 0x0, 0xc0010aead8, 0x5629d39bc3e0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/runtime/serializer/streaming/streaming.go:77 +0x8b
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000cef920, 0xc0012a9fa8, 0x8, 0x5629d58e5c00, 0xc0004a6e00, 0x0, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/rest/watch/decoder.go:49 +0x7e
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc001407d40)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/watch/streamwatcher.go:93 +0x177
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/watch/streamwatcher.go:60 +0xa1

goroutine 3736796 [sync.Cond.Wait]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0009d9a60, 0xc00000000e)
	/usr/local/go/src/runtime/sema.go:510 +0xfc
sync.(*Cond).Wait(0xc0009d9a50)
	/usr/local/go/src/sync/cond.go:56 +0x9f
golang.org/x/net/http2.(*pipe).Read(0xc0009d9a48, 0xc00081e001, 0x3dff, 0x3dff, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/pipe.go:65 +0xa8
golang.org/x/net/http2.transportResponseBody.Read(0xc0009d9a20, 0xc00081e001, 0x3dff, 0x3dff, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:2019 +0xae
encoding/json.(*Decoder).refill(0xc000916dc0, 0xc0009fa60a, 0x9)
	/usr/local/go/src/encoding/json/stream.go:161 +0xed
encoding/json.(*Decoder).readValue(0xc000916dc0, 0x0, 0x0, 0x5629d55e77e0)
	/usr/local/go/src/encoding/json/stream.go:136 +0x1de
encoding/json.(*Decoder).Decode(0xc000916dc0, 0x5629d561ca20, 0xc0009fa640, 0xc001257d68, 0x5629d3fe8b4a)
	/usr/local/go/src/encoding/json/stream.go:63 +0x7b
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000d30420, 0xc00136aa80, 0x4000, 0x4a80, 0xc001630450, 0x5629d58e7500, 0x38)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/framer/framer.go:150 +0x1a3
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc0010c4370, 0x0, 0x5629d58e7500, 0xc000e09480, 0x0, 0x0, 0x0, 0xc0010aef58, 0x5629d39bc3e0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/runtime/serializer/streaming/streaming.go:77 +0x8b
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000bf3c20, 0xc001257fa8, 0x8, 0x5629d58e7540, 0xc00076bdf8, 0x0, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/rest/watch/decoder.go:49 +0x7e
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000d30480)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/watch/streamwatcher.go:93 +0x177
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/watch/streamwatcher.go:60 +0xa1

goroutine 3735354 [sync.Cond.Wait, 6 minutes]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0003af220, 0x0)
	/usr/local/go/src/runtime/sema.go:510 +0xfc
sync.(*Cond).Wait(0xc0003af210)
	/usr/local/go/src/sync/cond.go:56 +0x9f
golang.org/x/net/http2.(*pipe).Read(0xc0003af208, 0xc0006ee200, 0x200, 0x200, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/pipe.go:65 +0xa8
golang.org/x/net/http2.transportResponseBody.Read(0xc0003af1e0, 0xc0006ee200, 0x200, 0x200, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:2019 +0xae
encoding/json.(*Decoder).refill(0xc0003af4a0, 0xc000d01aa0, 0x203000)
	/usr/local/go/src/encoding/json/stream.go:161 +0xed
encoding/json.(*Decoder).readValue(0xc0003af4a0, 0x0, 0x0, 0x5629d55e77e0)
	/usr/local/go/src/encoding/json/stream.go:136 +0x1de
encoding/json.(*Decoder).Decode(0xc0003af4a0, 0x5629d561ca20, 0xc000d01aa0, 0x300c0015bb528, 0x0)
	/usr/local/go/src/encoding/json/stream.go:63 +0x7b
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000a5b800, 0xc0011cd000, 0x400, 0x400, 0x9, 0x9, 0x38)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/framer/framer.go:150 +0x1a3
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000de3720, 0x0, 0x5629d58e7500, 0xc0014bc280, 0xc0015bb680, 0x203000, 0x203000, 0x5629d3993cdc, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/runtime/serializer/streaming/streaming.go:77 +0x8b
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000d01a80, 0x0, 0x28, 0x8, 0xc0015bb640, 0x5629d49f00b7, 0x5629d5888de8)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/rest/watch/decoder.go:49 +0x7e
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000a5b830)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/watch/streamwatcher.go:93 +0x177
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/watch/streamwatcher.go:60 +0xa1

goroutine 3726876 [IO wait]:
internal/poll.runtime_pollWait(0x7f07c9f0c548, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc000fea018, 0x72, 0x0, 0x1, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000fea000, 0xc00153e0d1, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc000fea000, 0xc00153e0d1, 0x1, 0x1, 0xc000865f58, 0x5629d39d2be6, 0xc000f96a20)
	/usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc0014a6000, 0xc00153e0d1, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x6a
net/http.(*connReader).backgroundRead(0xc00153e0c0)
	/usr/local/go/src/net/http/server.go:677 +0x5a
created by net/http.(*connReader).startBackgroundRead
	/usr/local/go/src/net/http/server.go:673 +0xd6
@mogren
Copy link
Contributor

mogren commented Aug 19, 2020

Thanks for reporting @adammw! There is clearly an issue in GetENIInfos() that we have to fix.

// GetENIInfos provides ENI IP information to introspection endpoint
func (ds *DataStore) GetENIInfos() *ENIInfos {
	ds.lock.Lock()
	defer ds.lock.Unlock()

	var eniInfos = ENIInfos{
		TotalIPs:    ds.total,
		AssignedIPs: ds.assigned,
		ENIIPPools:  make(map[string]ENIIPPool, len(ds.eniIPPools)),
	}

	for eni, eniInfo := range ds.eniIPPools {
		eniInfos.ENIIPPools[eni] = *eniInfo
	}
	return &eniInfos
}

@SaranBalaji90
Copy link
Contributor

Seems to be like same code path as the one reported here #1100

@SaranBalaji90
Copy link
Contributor

SaranBalaji90 commented Aug 28, 2020 •
edited
Loading

I tried seq 1 2020 | xargs -n1 -P15 curl localhost:61679/v1/enis but I'm not able to repro. I'm guessing as a first step we can add some logging to see what is being returned from GetENIInfos() to investigate further.

@mogren mogren mentioned this issue Aug 28, 2020
Merged
@SaranBalaji90
Copy link
Contributor

SaranBalaji90 commented Aug 30, 2020 •
edited
Loading

But I’m still not sure how we are seeing concurrent read/write error though.

Update: looks like we need to perform deep copy. #1179 addresses the issue.

This was referenced Aug 30, 2020
Merged
Merged
@mogren mogren closed this as completed Sep 2, 2020
@mogren
Copy link
Contributor

mogren commented Sep 4, 2020

v1.7.2-rc1 includes this fix.

@mogren
Copy link
Contributor

mogren commented Sep 15, 2020

@adammw v1.7.2 is out! 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mogren @adammw @SaranBalaji90