Welcome to Dronutz! This is a sample application and OpenTracing walkthrough, written in Go.
OpenTracing is a vendor-neutral, open standard for distributed tracing. To learn more, check out opentracing.io, and try the walkthrough below!
In addition to doing the walkthrough yourself, the results of each step are saved in their own git branch.
git checkout step-1
to see dronutz with turnkey/network tracing.git checkout step-2
to see an example of application-level client tracing.
Dronutz can be installed into your GOPATH via go get
.
go get github.com/tedsuo/ot-walkthrough-go/dronutz/cmd/api
Dronutz has two server components, API
and Kitchen
, which can be run in two
separate terminal windows. Both are configured with the same config file.
./scripts/run_kitchen.sh -config config_example.yaml
./scripts/run_api.sh -config config_example.yaml
In your web broswer, navigate to http://127.0.0.1:10001 and get yourself some donuts.
The configuration file currently supports three tracers: zipkin
, lightstep
,
and log
.
The easiest way to run zipkin locally is with docker.
docker run -d -p 9411:9411 openzipkin/zipkin
tracer: zipkin
tracer_host: localhost
tracer_port: 9411
If you're using lightstep, you will need your access token.
tracer: lightstep
tracer_host: LS_HOST
tracer_port: LS_PORT
tracer_access_token: LS_ACCESS_TOKEN
Log simply logs the traces, showing off how you can create structured logging without being tied to a particular logging framework.
tracer: log
When you go to add tracing to a system, the best place to start is installing OpenTracing plugins for the OSS components you are using. Instrumenting your networking libraries, web frameworks, and service clients quickly gives you a lot of information about your distributed system, without requiring you to change a lot of code.
To do this, let's change the startup of dronutz/cmd/api
and
dronutz/cmd/kitchen
to include tracing.
In OpenTracing, there is a concept of a global tracer for everyone to access.
As a convenience, we already have a function dronutz.ConfigureGlobalTracer
that works with the dronutz configuration file. In the main
of both API and
Kitchen, add it before initializing any other components.
err = dronutz.ConfigureGlobalTracer(cfg, "api") // <-- be sure to name api and kitchen differently
if err != nil {
panic(err)
}
There's a trick in this first block of code. Make sure to change the second
parameter to kitchen
when you add it to the Kitchen component.
The first thing to always intrument is the inbound RPC server. For API,
this is an http server. Use the nethttp
package from opentracing-contrib
to wrap the top-level service http.Handler
with opentracing middleware.
https://godoc.org/github.com/opentracing-contrib/go-stdlib/nethttp
err = http.ListenAndServe(
cfg.APIAddress(),
nethttp.Middleware(
opentracing.GlobalTracer(),
service.ServeMux(),
nethttp.OperationNameFunc(func(req *http.Request) string {
return "/dronutz.API" + req.URL.Path // <-- ensures we use a standard naming convention
}),
),
)
Now that we have inbound RPC taken care of, let's look at our outbound service clients. In API, we talk to the Kitchen service over gRPC.
https://godoc.org/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc
conn, err := grpc.Dial(
cfg.KitchenAddress(),
grpc.WithInsecure(),
grpc.WithUnaryInterceptor(
otgrpc.OpenTracingClientInterceptor(opentracing.GlobalTracer()),
),
)
At this point, the API service is successfully wrapped.
Though we already have the client side wrapped, let's now wrap the server side of Kitchen's gRPC service.
server := grpc.NewServer(
grpc.UnaryInterceptor(
otgrpc.OpenTracingServerInterceptor(
opentracing.GlobalTracer(),
otgrpc.LogPayloads()),
),
)
This will allow us to automatically have traces bound to rpc endpoints, so we can start tracing internally, and will let us know if something other than the API server were to start talking to this backend service.
Try including the otgrpc.LogPayloads()
option to have more detailed
information show up in your traces as logs.
Now that we're all hooked up, try buying some donuts in the browser. You should
see the traces appear in your tracer, or in the terminal if you are using the
log
tracer.
Search for traces starting with /dronutz.API
or /dronutz.Kitchen
to see the
patterns of requests that occur when you click the order button.
Now that the components in our system are linked up at the networking level, we can start adding application level tracing by tying multiple network calls together into a single trace.
In dronutz, we'd like to know the time and resources involved with buying a donut, from the moment it is ordered to when it is delivered. Let's add OpenTracing to the client so that we can bundle all of these network calls into the same trace.
NOTE: Unfortunately, zipkin does not have opentracing support for javascript at this time, so you must use the LightStep tracer for this part of the walkthrough. Hopefully this will be changing soon.
Just like on the server, we start be configuring the global tracer. Do so at
the begining of document ready in client/client.js
.
Tracer.initGlobalTracer(LightStep.tracer({
component_name : 'client',
access_token : Config.tracer_access_token,
collector_host : Config.tracer_host,
collector_port : Config.tracer_port,
collector_encryption: "none",
xhr_instrumentation : false,
}));
On the client, we don't want to trace an individual request, but record a logical collection of actions under a single span. The Order object in dronutz encapsulates the logic of buying donuts. To start tracing the order process, store a span on the order, starting and finishing it when the order activates and completes.
Order.prototype.activate = function(){
this._span = Tracer.startSpan('/dronutz.Client/BuyDonuts');
this._state = "active";
}
Order.prototype.complete = function(){
this._span.finish();
return this.reset();
}
Because we are now doing manual tracing, we have to inject the span context into
the http requests ourselves, so that the endpoints on the server can extract and
bind their child spans to the same trace. To do this, add a function to Order
that serializes the span's context into a set of http headers.
Order.prototype.headers = function(){
var otHeaders = {};
Tracer.inject(this._span, Tracer.FORMAT_TEXT_MAP, otHeaders);
return otHeaders;
}
Now that we can generate headers, use them as the headers for the /order
and
/status
requests in client.js
:
$.ajax('/order', {
headers: order.headers(),
method:'POST',
// yadda yadda
}
And we're done! Buy some donuts, check out /dronutz.Client/BuyDonuts
, and
notice how the order and polling requests are now grouped under a single span,
with timing information for the entire operation.
If you still have time, try to trace other things! For example, maybe we would like to extend the trace on the client to record when an order is opened, and log when each donut is added to the cart.
Also, that kitchen component is pretty strange. What's going on in there? Think of how you
might trace it to find the answers. Alternatively, add a DroneDispatch
service
to the system and try to create a more complicated trace.
NOTE: you must install protoc
and run ./scripts/build_proto.sh
if you
change the protobuf definitions.
Thanks for joining us in this walkthrough! Hope you enjoyed it. If you did, let us know, and consider spreading the love!
A great way to get the feel for OpenTracing is to try your hand at instrumenting the OSS servers, frameworks, and client libraries that we all share. If you make one, consider adding it to the growing ecosystem at http://github.com/opentracing-contrib. If you maintain a library yourself, plase consider adding built-in OT support.
We also need walkthroughs for languages other than Golang. Feel free to reuse the client, protobufs, and other assets from here if you'd like to make one.
For a more detailed explanation of OSS Instrumentation, check out the TurnKey Tracing proposal at http://bit.ly/turnkey-tracing.
Aloha!