instrumentation

brave-instrumentation

This module a redo of all major instrumentation libraries since Brave 3. Artifacts have the naming convention "brave-instrumentation-XXX": for example, the directory "servlet" includes the artifact "brave-instrumentation-servlet".

Here's a brief overview of what's packaged here:

• dubbo - Tracing filter for RPC providers and consumers in Apache Dubbo
• grpc - Tracing client and server interceptors for grpc
• httpasyncclient - Tracing decorator for Apache HttpClient 4.0+
• httpclient - Tracing decorator for Apache HttpClient 4.3+
• jaxrs2 - Client tracing filter and span customizing resource filter for JAX-RS 2.x
• jersey-server - Tracing and span customizing application event listeners for Jersey Server.
• jms - Tracing decorators for JMS 1.1-2.01 producers, consumers and listeners.
• kafka-clients - Tracing decorators for Kafka 0.11+ producers and consumers.
• kafka-streams - Tracing decorator for Kafka Streams 2.0+ clients.
• mongodb - Tracing MongoDB command listener
• mysql - Tracing MySQL statement interceptor
• mysql6 - Tracing MySQL v6 statement interceptor
• mysql8 - Tracing MySQL v8 statement interceptor
• netty-codec-http - Tracing handler for Netty 4.x http servers
• okhttp3 - Tracing decorators for OkHttp 3.x
• servlet - Tracing filter for Servlet 2.5+ (including Async)
• spring-rabbit - Tracing MessagePostProcessor and ListenerAdvice for Spring Rabbit
• spring-web - Tracing interceptor for Spring RestTemplate
• spring-webmvc - Tracing filter and span customizing interceptors for Spring WebMVC
• vertx-web - Tracing routing context handler for Vert.x Web

Here are other tools we provide for configuring or testing instrumentation:

• http - HttpTracing that allows portable configuration of HTTP instrumentation
• http-tests - Interop test suit that all http client and server instrumentation must pass
• messaging - MessagingTracing that allows portable configuration of messaging instrumentation
• rpc - RpcTracing that allows portable configuration of RPC instrumentation
• spring-beans - This allows you to setup tracing with XML instead of custom code.
• benchmarks - JMH microbenchmarks that measure instrumentation overhead

Configuration

Log integration

You may want to put trace IDs into your log files, or change thread local behavior. Look at our context libraries, for integration with tools such as SLF4J.

XML Configuration

If you are trying to trace legacy applications, you may be interested in Spring XML Configuration. This allows you to setup tracing without any custom code.

Custom configuration

When re-using trace instrumentation, you typically do not need to write any code. However, you can customize data and sampling policy through common types. The HttpTracing type configures all libraries the same way.

Ex.

apache = TracingHttpClientBuilder.create(httpTracing.clientOf("s3"));
okhttp = TracingCallFactory.create(httpTracing.clientOf("sqs"), new OkHttpClient());

Below introduces common configuration. See the http instrumentation docs for more.

Span Data

Naming and tags are configurable in a library-agnostic way. For example, to add a non-default tag for HTTP clients, you can do this:

httpTracing = httpTracing.toBuilder()
    .clientRequestParser((req, context, span) -> {
      HttpClientRequestParser.DEFAULT.parse(req, context, span);
      span.tag("http.url", req.url()); // add the url in addition to defaults
    })
    .build();

Request-based Sampling

Which requests to start traces for is configurable in a library-agnostic way. You can change the sampling policy by specifying it in the HttpTracing component. Here's an example which doesn't start new traces for requests to favicon (which many browsers automatically fetch).

httpTracing = httpTracing.toBuilder()
    .serverSampler(HttpRuleSampler.newBuilder()
      .putRule(pathStartsWith("/favicon"), Sampler.NEVER_SAMPLE)
      .build())
    .build();

Writing new instrumentation

We worked very hard to make writing new instrumentation easy and efficient. Most of our built-in instrumentation are 50-100 lines of code, yet allow flexible configuration of tags and sampling policy.

Brave includes two fundamental types to support tracing interprocess communication brave.Request and brave.Response. These types represent the following communication patterns, defined by the Zipkin Api and returned by spanKind():

• CLIENT
• SERVER
• PRODUCER
• CONSUMER

Brave includes abstractions for the above that handle aspects including sampling, header processing and data parsing:

• HTTP is CLIENT/SERVER
• Messaging is PRODUCER/CONSUMER
• RPC is CLIENT/SERVER

You should use these abstractions instead of modeling your own. Not only are they well thought through, but they also include tests to prevent common mistakes. For example, HTTP tests makes sure spans are modeled consistently and uniform configuration works.

Using abstractions also helps avoid modeling gotchas that might not be intuitive at first.

For example, a common mistake is using CLIENT kind for long lived connections such as database pools or chat sessions. Doing so, however, can result in confusing data, such as incorrect service diagrams or server calls being attached to the wrong trace.

It is also a common misconception that a grouping of CLIENT spans should itself be a CLIENT span. A logical span that serves only to group others, is a local span (Span.kind() is unset). CLIENT spans represent a single remote request. It is relatively common to use a local span as the parent of multiple CLIENT requests, or to represent a single request that was served from a local cache.

Some features that already exist are subtle, yet proven with tests. It is best to look for tests in the "features" package before deciding something doesn't work.

• Core feature tests
• HTTP feature tests
• Messaging feature tests
• RPC feature tests

If you end up in a position where you still believe you need a custom model, please contact us before committing to it. There may be an alternative not yet documented, you might prefer. Moreover, folks usually want support and we cannot support every idea. It is better to understand this before you commit to something bespoke. In short, please reach out on gitter, as there are usually others around to help.