Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump gRPC dependency to v1.29.x #2443

Closed
jpkrohling opened this issue Sep 1, 2020 · 0 comments · Fixed by #2445 or jaegertracing/jaeger-operator#1178
Closed

Bump gRPC dependency to v1.29.x #2443

jpkrohling opened this issue Sep 1, 2020 · 0 comments · Fixed by #2445 or jaegertracing/jaeger-operator#1178
Labels

Comments

@jpkrohling
Copy link
Contributor

jpkrohling commented Sep 1, 2020

Describe the bug
The current version of the gRPC client used has a behavior where it won't update the list of hosts if the initial list is empty. See jaegertracing/jaeger-operator#1175, which is likely caused by grpc/grpc-go#3353. Bumping the gRPC dependency to the latest (v1.31.1) makes it work.

To Reproduce
Steps to reproduce the behavior:

  1. Using the Jaeger Operator, deploy the example agent-as-daemonset
  2. Get the logs for the daemonset: $ kubectl logs daemonsets/agent-as-daemonset-agent-daemonset -f
  3. See that the list of hosts isn't updated even after a couple of minutes:
$ kubectl logs daemonsets/agent-as-daemonset-agent-daemonset  -f
2020/09/01 15:13:00 maxprocs: Leaving GOMAXPROCS=6: CPU quota undefined
{"level":"info","ts":1598973180.944437,"caller":"flags/service.go:116","msg":"Mounting metrics handler on admin server","route":"/metrics"}
{"level":"info","ts":1598973180.9445577,"caller":"flags/admin.go:120","msg":"Mounting health check on admin server","route":"/"}
{"level":"info","ts":1598973180.9446096,"caller":"flags/admin.go:126","msg":"Starting admin HTTP server","http-addr":":14271"}
{"level":"info","ts":1598973180.944623,"caller":"flags/admin.go:112","msg":"Admin server started","http.host-port":"[::]:14271","health-status":"unavailable"}
{"level":"info","ts":1598973180.9454503,"caller":"grpc/builder.go:66","msg":"Agent requested insecure grpc connection to collector(s)"}
{"level":"info","ts":1598973180.9454787,"caller":"grpc@v1.28.0-pre/clientconn.go:106","msg":"parsed scheme: \"dns\"","system":"grpc","grpc_log":true}
{"level":"info","ts":1598973180.9479086,"caller":"command-line-arguments/main.go:78","msg":"Starting agent"}
{"level":"info","ts":1598973180.9479551,"caller":"healthcheck/handler.go:128","msg":"Health Check state change","status":"ready"}
{"level":"info","ts":1598973180.947995,"caller":"app/agent.go:69","msg":"Starting jaeger-agent HTTP server","http-port":5778}
{"level":"info","ts":1598973180.9573705,"caller":"dns/dns_resolver.go:212","msg":"ccResolverWrapper: sending update to cc: {[] <nil> <nil>}","system":"grpc","grpc_log":true}
{"level":"info","ts":1598973180.9574049,"caller":"grpc@v1.28.0-pre/clientconn.go:948","msg":"ClientConn switching balancer to \"round_robin\"","system":"grpc","grpc_log":true}

Bumping gRPC to the latest, after 30s in this state, the list of servers is updated:

$ kubectl logs daemonsets/agent-as-daemonset-agent-daemonset  -f
2020/09/01 15:16:04 maxprocs: Leaving GOMAXPROCS=6: CPU quota undefined
{"level":"info","ts":1598973364.7745779,"caller":"flags/service.go:116","msg":"Mounting metrics handler on admin server","route":"/metrics"}
{"level":"info","ts":1598973364.7747014,"caller":"flags/admin.go:120","msg":"Mounting health check on admin server","route":"/"}
{"level":"info","ts":1598973364.774737,"caller":"flags/admin.go:126","msg":"Starting admin HTTP server","http-addr":":14271"}
{"level":"info","ts":1598973364.7747464,"caller":"flags/admin.go:112","msg":"Admin server started","http.host-port":"[::]:14271","health-status":"unavailable"}
{"level":"info","ts":1598973364.7751777,"caller":"grpc/builder.go:66","msg":"Agent requested insecure grpc connection to collector(s)"}
{"level":"info","ts":1598973364.7751987,"caller":"grpc@v1.27.1/clientconn.go:106","msg":"parsed scheme: \"dns\"","system":"grpc","grpc_log":true}
{"level":"info","ts":1598973364.776601,"caller":"command-line-arguments/main.go:78","msg":"Starting agent"}
{"level":"info","ts":1598973364.776632,"caller":"healthcheck/handler.go:128","msg":"Health Check state change","status":"ready"}
{"level":"info","ts":1598973364.7766716,"caller":"app/agent.go:69","msg":"Starting jaeger-agent HTTP server","http-port":5778}
{"level":"info","ts":1598973364.7881553,"caller":"dns/dns_resolver.go:212","msg":"ccResolverWrapper: sending update to cc: {[] <nil> <nil>}","system":"grpc","grpc_log":true}
{"level":"info","ts":1598973364.7881958,"caller":"grpc@v1.27.1/clientconn.go:948","msg":"ClientConn switching balancer to \"round_robin\"","system":"grpc","grpc_log":true}
{"level":"info","ts":1598973394.8044846,"caller":"dns/dns_resolver.go:212","msg":"ccResolverWrapper: sending update to cc: {[{10.244.0.14:14250  <nil> 0 <nil>}] <nil> <nil>}","system":"grpc","grpc_log":true}
{"level":"info","ts":1598973394.8048198,"caller":"base/balancer.go:196","msg":"roundrobinPicker: newPicker called with info: {map[]}","system":"grpc","grpc_log":true}
{"level":"info","ts":1598973394.8062818,"caller":"base/balancer.go:196","msg":"roundrobinPicker: newPicker called with info: {map[0xc0004121c0:{{10.244.0.14:14250  <nil> 0 <nil>}}]}","system":"grpc","grpc_log":true}

Expected behavior
The gRPC client should be able to get DNS updates, especially if the initial list is empty.

Version (please complete the following information):

  • Jaeger version: master as of now, v1.19.2
  • Deployment: Kubernetes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant