Add distributed tracing via Zipkin. #354

mdemirhan · 2018-03-13T21:25:16Z

No description provided.

mdemirhan · 2018-03-13T21:54:40Z

/retest

grantr · 2018-03-13T21:55:10Z

@mdemirhan General question: how does Zipkin overlap with Jaeger? Will we install both?

mdemirhan · 2018-03-13T22:17:42Z

@grantr Both Zipkin and Jaeger implement OpenTracing (http://opentracing.io/documentation/pages/supported-tracers). However; as far as I understand; OpenTracing doesn't really define the wire format or the backend contract and these different implementations aren't necessarily compatible. That being said, Jaeger is supposed to be Zipkin compatible and we should be able to use Jaeger instead of Zipkin if necessary.

Istio supports both Zipkin and Jaeger and we can replace Zipkin with Jaeger if there are features in Jaeger that you prefer over Zipkin. Let me know and I will be happy to take a look.

mattmoor · 2018-03-13T22:12:15Z

sample/telemetrysample/telemetrysample.go

@@ -55,32 +61,106 @@ func main() {
 	flag.Parse()
 	glog.Info("Telemetry sample started.")

+	// Zipkin setup
+	reporter := httpreporter.NewReporter("http://zipkin.istio-system.svc.cluster.local:9411/api/v2/spans")


I'm curious about the choice to run zipkin in istio-system instead of either the monitoring or ela-system namespaces?

Istio has a hard-coded URL to zipkin and it assumes that zipkin is installed in istio-system with a service named zipkin. I will open an issue to them and ask them to provide us an override. In the meantime, I will add comments in the code and YAML mentioning that this is a workaround for the istio issue.

mattmoor · 2018-03-13T22:16:45Z

sample/telemetrysample/telemetrysample.go

-
-	// Add a random delay to simulate some actual work here
-	time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
+func rootHandler(client *zipkinhttp.Client) http.HandlerFunc {


These changes seem complicated. I don't think it's immediately obvious which parts of the code are necessary setup vs. demonstrating the tracing capabilities. Can you add detailed comments explaining the various parts?

Sure thing. I will add extra comments explaining what each section is doing.

grantr · 2018-03-13T22:23:42Z

True, the individual tracer implementation defines trace propagation. I have no experience with either (though I do have experience with LightStep) but even so, Jaeger seems like a better choice than Zipkin:

owned by the CNCF
active community (though https://github.com/openzipkin/zipkin seems active as well)
compatible with both Jaeger and Zipkin traces (presumably Zipkin is only compatible with Zipkin)
written in Go (its Go client, which we'd use for Elafros, is likely to be very good)

mdemirhan · 2018-03-13T22:48:29Z

@grantr All good points. The reason I chose Zipkin over Jaeger was my impression on its higher usage. I couldn't really find a good reference that compares both to make a good informed decision. I will look into Jaeger more and see if we can replace Zipkin. Though, I would like to check this in now and do that investigation next week. Does that sound reasonable?

grantr · 2018-03-13T22:52:49Z

Does that sound reasonable?

Definitely, I didn't mean to stop this PR; was just curious.

mdemirhan · 2018-03-13T23:18:38Z

/retest

vaikas · 2018-03-14T18:00:56Z

I just have a clarifying question about this. It seems to me like we're instrumenting the application to talk directly to a particular tool, and I had been thinking that we should be able to get some of these traces by preserving the headers and not actually requiring the application to have to choose which of these tools is being used.
https://istio.io/docs/tasks/telemetry/distributed-tracing.html

mdemirhan · 2018-03-14T18:54:40Z

@vaikas-google If you don't instrument within the code, you will get a one liner summary view, which is the call from Istio's point of view.

To see the whole view of the trace like:

you need to ensure that the open tracing headers are passed to all other http calls you are making. Telemetry sample shows how to do that using the go client library of zipkin. It can be done manually as well. Or if you are using something like gRPC and enable open tracing, the rpc layer will take care of getting the trace context from the request and copying them to any outgoing calls.

mdemirhan · 2018-03-14T19:02:05Z

That being said, it seems like copying headers manually might actually be a lot more straightforward and I could change the code to just do that. The main disadvantage of that would be the loss of some data (for example, instead of the service name, you will see istio-proxy everywhere).

I will give that a shot and see if that looks better. Thanks for bringing this up.

mdemirhan · 2018-03-14T21:12:52Z

@vaikas-google Digging deeper into this, we will need this type of instrumentation for any service that is not behind istio proxy and that is not local within our cluster. In a service mesh, if you are calling only local services behind an istio-proxy, this instrumentation is not needed. If you are going to call an external service, or if you are going to call an internal service that is not behind istio-proxy, then this instrumentation is needed. I will add a note to the beginning of the file and to the documentation stating that.

Let me know if that sounds reasonable.

@mattmoor Let me know if the updated documentation is clear and if I have your approval to check this in.

# Conflicts: # Gopkg.lock

vaikas · 2018-03-14T21:26:51Z

That's unfortunate especially in the case of functions that are lightweight in code and having them require to manually instrument the code. Thanks for doing the investigation!

mdemirhan · 2018-03-14T22:06:35Z

@vaikas-google Agreed. I will look into this further and see if anything can be done to make the situation better. I will capture this investigation as a Github issue and hopefully get to it within the next couple of days.

mdemirhan · 2018-03-14T22:06:43Z

/retest

mattmoor

Thanks, these comments are super helpful to understand what's going on. thanks for so much detail.

I have a handful more comments.

mattmoor · 2018-03-14T23:24:22Z

sample/telemetrysample/telemetrysample.go

@@ -62,6 +78,18 @@ func main() {
 	glog.Info("Telemetry sample started.")

 	// Zipkin setup
+	// If your service only calls services in the local cluster and if those services are
+	// behind istio-proxy (all revisions are behind istio-proxy), then Zipkin setup below


nit: "... Elafros Revisions ..." may be clearer

mattmoor · 2018-03-14T23:24:52Z

sample/telemetrysample/telemetrysample.go

+	// requests and copy them to your outgoing requests.
+	// x-request-id, x-b3-traceid, x-b3-spanid, x-b3-parentspanid, x-b3-sampled, x-b3-flags, x-ot-span-context
+	//
+	// For richer instrumentation, or to instrument calls that are mad to services not behind


s/mad/made/g

Still need this.

mattmoor · 2018-03-14T23:25:18Z

sample/telemetrysample/telemetrysample.go

+	// x-request-id, x-b3-traceid, x-b3-spanid, x-b3-parentspanid, x-b3-sampled, x-b3-flags, x-ot-span-context
+	//
+	// For richer instrumentation, or to instrument calls that are mad to services not behind
+	// istio-proxy, you can instrument the code using Zipkin's Go client library:


The trailing colon is weird here.

mattmoor · 2018-03-14T23:29:10Z

sample/telemetrysample/telemetrysample.go

 		defer func(start time.Time) {
+			// Counters only support incrementing. Increment the count by one
+			// to capture the single call.


Are there any built-in metrics that you get by simply registering the middleware?

One of the nice things about our internal systems is well-defined built-in metrics that just work. :(

The go client library adds builtin metrics for golang (memstats, thread, go routines, ...etc) but nothing else. I will create an issue and see if we can simplify this further and add a library that can add some common metrics by default.

Forgot to mention: There are set of metrics around request time, count & result sent by envoy automatically (whether you instrument your code or not). Those should be good for most use cases. This example here was to show more of a custom metric and how to emit it.

mattmoor · 2018-03-14T23:30:29Z

sample/telemetrysample/telemetrysample.go

@@ -147,15 +191,23 @@ func rootHandler(client *zipkinhttp.Client) http.HandlerFunc {
 	}
 }

+// Makes an http call with distributed tracing enabled.


I think it would be a more idiomatic example if this was done by creating an http.Roundtripper that instruments outbound requests.

Added #374 to track this.

mdemirhan · 2018-03-15T17:02:58Z

/retest

mattmoor · 2018-03-15T17:15:20Z

sample/telemetrysample/telemetrysample.go

+	// requests and copy them to your outgoing requests.
+	// x-request-id, x-b3-traceid, x-b3-spanid, x-b3-parentspanid, x-b3-sampled, x-b3-flags, x-ot-span-context
+	//
+	// For richer instrumentation, or to instrument calls that are mad to services not behind


Still need this.

mdemirhan · 2018-03-15T17:31:09Z

/retest

mdemirhan added 2 commits March 13, 2018 14:19

Add distributed tracing via Zipkin.

ed6c835

Remove unused sleep method.

3a37d76

mdemirhan requested a review from a team March 13, 2018 21:25

google-prow-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Mar 13, 2018

mdemirhan requested review from grantr, mattmoor, akyyy and josephburnett March 13, 2018 21:25

mattmoor reviewed Mar 13, 2018

View reviewed changes

Extra documentation.

56ea9ba

mdemirhan added 2 commits March 13, 2018 16:22

Documentation updates.

8080cb7

More documentation.

0fc2d1d

mdemirhan added 2 commits March 14, 2018 14:19

Further documentation changes.

e2d8c26

Merge branch 'master-upstream' into mdemirhan/zipkin2

abbbe17

# Conflicts: # Gopkg.lock

mattmoor reviewed Mar 14, 2018

View reviewed changes

Documentation fixes.

83ccdbb

mattmoor approved these changes Mar 15, 2018

View reviewed changes

Spelling fix.

fa567ca

mdemirhan merged commit b06988f into knative:master Mar 15, 2018

mdemirhan deleted the mdemirhan/zipkin2 branch March 15, 2018 18:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add distributed tracing via Zipkin. #354

Add distributed tracing via Zipkin. #354

mdemirhan commented Mar 13, 2018

mdemirhan commented Mar 13, 2018

grantr commented Mar 13, 2018

mdemirhan commented Mar 13, 2018

mattmoor Mar 13, 2018

mdemirhan Mar 13, 2018

mattmoor Mar 13, 2018

mdemirhan Mar 13, 2018

grantr commented Mar 13, 2018

mdemirhan commented Mar 13, 2018

grantr commented Mar 13, 2018

mdemirhan commented Mar 13, 2018

vaikas commented Mar 14, 2018

mdemirhan commented Mar 14, 2018

mdemirhan commented Mar 14, 2018

mdemirhan commented Mar 14, 2018

vaikas commented Mar 14, 2018

mdemirhan commented Mar 14, 2018

mdemirhan commented Mar 14, 2018

mattmoor left a comment

mattmoor Mar 14, 2018

mattmoor Mar 14, 2018

mattmoor Mar 15, 2018

mattmoor Mar 14, 2018

mattmoor Mar 14, 2018

mdemirhan Mar 15, 2018

mdemirhan Mar 15, 2018

mattmoor Mar 14, 2018

mdemirhan Mar 15, 2018

mdemirhan commented Mar 15, 2018

mattmoor Mar 15, 2018

mdemirhan commented Mar 15, 2018

Add distributed tracing via Zipkin. #354

Add distributed tracing via Zipkin. #354

Conversation

mdemirhan commented Mar 13, 2018

mdemirhan commented Mar 13, 2018

grantr commented Mar 13, 2018

mdemirhan commented Mar 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grantr commented Mar 13, 2018

mdemirhan commented Mar 13, 2018

grantr commented Mar 13, 2018

mdemirhan commented Mar 13, 2018

vaikas commented Mar 14, 2018

mdemirhan commented Mar 14, 2018

mdemirhan commented Mar 14, 2018

mdemirhan commented Mar 14, 2018

vaikas commented Mar 14, 2018

mdemirhan commented Mar 14, 2018

mdemirhan commented Mar 14, 2018

mattmoor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdemirhan commented Mar 15, 2018

Choose a reason for hiding this comment

mdemirhan commented Mar 15, 2018