Working with Distributed Tracing using Pedestal's Trace Interceptor
This sample walks through how to use the distributed tracing capabilities built into Pedestal's Trace Interceptor.
Out of the box, Pedestal's
io.pedestal.log supports the OpenTracing API,
but additional support is possible by extending Pedestal's tracing protocols.
Distributed tracing works by having a centralized server collect portions of traces called "spans" or "segments" and proving a comprehensive view of requests and data moving through your service. A single span can append tags, data, and log messages during execution, which all get captured during the trace.
If your service forwards calls to additional services, you're able to encode the trace ID into those requests and continue tracking the trace across boundaries.
There are some conventions around naming schemes in tracing elements. You can read about those in the OpenTracing Docs.
Running a distributed trace server
We're going to use Docker to run an instance of the X-Ray Daemon, which provides distributed tracing for us in AWS.
$ docker run --name aws_xray --rm -p 2000:2000/udp \ --attach STDOUT \ -e AWS_ACCESS_KEY_ID=aws_access_key \ -e AWS_SECRET_ACCESS_KEY=aws_secret_key \ -e AWS_REGION=aws_region \ namshi/aws-xray --local-mode
- Start the application:
PEDESTAL_TRACER=io.pedestal.log.aws.xray/tracer lein run
- Go to localhost:8080 to see:
Hello Tracing World!
- Now go to localhost:8080/trace, which will call another service endpoint, tracing each step.
- Read your app's source code at src/tracing/service.clj. Explore the routes, functions, and tracing interceptor setup.
- Go to your X-Ray UI and explore the traces
- Stop your service and kill your X-Ray Container with Ctrl-c
- Learn more! See the Links section below.
To configure logging see config/logback.xml. By default, the app logs to stdout and logs/. To learn more about configuring Logback, read its documentation.
How we set our service up to use X-Ray
With our additional dependency,
we can configure our access to our local X-Ray Daemon.
The default tracer in Pedestal can be set with a JVM property setting or an environment variable.
We used the environment variable to set the Tracer to X-Ray.
It is also possible to register a tracer with the
-register protocol function at the main
entry point of your service (for example, in
although this is tricky for non-OpenTracing systems.
Developing your service
- Start a new REPL:
- Start your service in dev-mode:
(def dev-serv (run-dev))
- Connect your editor to the running REPL session. Re-evaluated code will be seen immediately in the service.
Docker container support
- Build an uberjar of your service:
- Build a Docker image:
sudo docker build -t tracing .
- Run your Docker image:
docker run -p 8080:8080 tracing
- Build and run your image:
capstan run -f "8080:8080"
Once the image it built, it's cached. To delete the image and build a new one:
capstan rmi tracing; capstan build