Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nodeagent panic (1.1.3-pre) #13325

Closed
mandarjog opened this issue Apr 15, 2019 · 6 comments

Comments

6 participants
@mandarjog
Copy link
Contributor

commented Apr 15, 2019

image: gcr.io/istio-release/node-agent-k8s:release-1.1-20190413-09-16

2019-04-15T05:52:46.913328Z info  SDS: push key/cert pair from node agent to proxy: "sidecar~10.16.3.145~svc00-0v2-5b7c99db7b-jjsm8.service-graph00~service-graph00.svc.cluster.local-13"

fatal error: concurrent map writes

goroutine 347 [running]:
runtime.throw(0x155181c, 0x15)
  /usr/local/go/src/runtime/panic.go:616 +0x81 fp=0xc420f33d38 sp=0xc420f33d18 pc=0x429581
runtime.mapassign(0x1383c40, 0xc42020acc0, 0xc420f33ec8, 0xc420a26e00)
  /usr/local/go/src/runtime/hashmap.go:607 +0x553 fp=0xc420f33dc8 sp=0xc420f33d38 pc=0x407b73
istio.io/istio/security/pkg/nodeagent/cache.(*SecretCache).rotate.func1.1(0xc420045200, 0xc42052e680, 0xc42093cea0, 0xc420a3c070, 0x63, 0xc420a3c070, 0x63, 0xc420432180, 0x7, 0xbf2524eba61a2cbb, ...)
  /workspace/go/src/istio.io/istio/security/pkg/nodeagent/cache/secretcache.go:465 +0x5a5 fp=0xc420f33f78 sp=0xc420f33dc8 pc=0x1000a45
runtime.goexit()
  /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420f33f80 sp=0xc420f33f78 pc=0x455cd1
created by istio.io/istio/security/pkg/nodeagent/cache.(*SecretCache).rotate.func1
  /workspace/go/src/istio.io/istio/security/pkg/nodeagent/cache/secretcache.go:437 +0x326
2019-04-15T05:54:15.307935Z info  SDS: push key/cert pair from node agent to proxy: "sidecar~10.16.20.66~svc01-0-1-0-67b96bb649-h4lpl.service-graph01~service-graph01.svc.cluster.local-13"

fatal error: bad map state

goroutine 60 [running]:
runtime.throw(0x1547e1d, 0xd)
  /usr/local/go/src/runtime/panic.go:616 +0x81 fp=0xc4209a9c58 sp=0xc4209a9c38 pc=0x429581
runtime.evacuate(0x1383c40, 0xc420c4b2f0, 0x2)
  /usr/local/go/src/runtime/hashmap.go:1039 +0x634 fp=0xc4209a9d10 sp=0xc4209a9c58 pc=0x409244
runtime.growWork(0x1383c40, 0xc420c4b2f0, 0x2)
  /usr/local/go/src/runtime/hashmap.go:985 +0x61 fp=0xc4209a9d38 sp=0xc4209a9d10 pc=0x408bc1
runtime.mapassign(0x1383c40, 0xc420c4b2f0, 0xc4209a9ec8, 0xc420b0b880)
  /usr/local/go/src/runtime/hashmap.go:535 +0x4c5 fp=0xc4209a9dc8 sp=0xc4209a9d38 pc=0x407ae5
istio.io/istio/security/pkg/nodeagent/cache.(*SecretCache).rotate.func1.1(0xc420c787a0, 0xc420682f70, 0xc4201bbe60, 0xc42023a4d0, 0x63, 0xc42023a4d0, 0x63, 0xc420c78489, 0x7, 0xbf252501a7272900, ...)
  /workspace/go/src/istio.io/istio/security/pkg/nodeagent/cache/secretcache.go:465 +0x5a5 fp=0xc4209a9f78 sp=0xc4209a9dc8 pc=0x1000a45
runtime.goexit()
  /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4209a9f80 sp=0xc4209a9f78 pc=0x455cd1
created by istio.io/istio/security/pkg/nodeagent/cache.(*SecretCache).rotate.func1
  /workspace/go/src/istio.io/istio/security/pkg/nodeagent/cache/secretcache.go:437 +0x326

@mandarjog mandarjog added this to the 1.2 milestone Apr 15, 2019

@mandarjog

This comment has been minimized.

Copy link
Contributor Author

commented Apr 15, 2019

@duderino

This comment has been minimized.

Copy link
Contributor

commented Apr 15, 2019

@JimmyCYJ please take a look too

@duderino

This comment has been minimized.

Copy link
Contributor

commented Apr 15, 2019

I'm adding a note for this to the 1.1.3 release notes. I'm describing it as a non-blocker since the feature is alpha, but saying we will try to fix it this week in time for 1.1.4

@JimmyCYJ

This comment has been minimized.

Copy link
Contributor

commented Apr 15, 2019

It looks like a race condition during cert rotation.

@quanjielin

This comment has been minimized.

Copy link
Contributor

commented Apr 15, 2019

the issue was original introduced in #9055, just sent a fix PR #13333

@wenchenglu wenchenglu assigned quanjielin and unassigned lei-tang Apr 15, 2019

@kramerul

This comment has been minimized.

Copy link
Contributor

commented May 15, 2019

It seems that this issue is solved. Therefore, I'm closing it.

@kramerul kramerul closed this May 15, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.