Call tracing and log correlation in distributed systems
Clone or download
whiskeysierra Merge pull request #185 from zalando/dependabot/maven/org.springframe…
…work-spring-framework-bom-5.1.3.RELEASE

Bump spring-framework-bom from 5.1.2.RELEASE to 5.1.3.RELEASE
Latest commit f9f9471 Dec 2, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github Added code owners and cleaned up maintainers Jul 12, 2018
.mvn/wrapper Updated to newer maven wrapper version without binary Apr 26, 2018
docs Updated banner to fit new github design Dec 3, 2015
tracer-aspectj Development 1.0.0-SNAPSHOT Nov 19, 2018
tracer-benchmark Bump dependency-check-maven from 3.3.4 to 4.0.0 Nov 22, 2018
tracer-bom Development 1.0.0-SNAPSHOT Nov 19, 2018
tracer-core Development 1.0.0-SNAPSHOT Nov 19, 2018
tracer-httpclient Development 1.0.0-SNAPSHOT Nov 19, 2018
tracer-hystrix Development 1.0.0-SNAPSHOT Nov 19, 2018
tracer-junit Development 1.0.0-SNAPSHOT Nov 19, 2018
tracer-okhttp Development 1.0.0-SNAPSHOT Nov 19, 2018
tracer-servlet Development 1.0.0-SNAPSHOT Nov 19, 2018
tracer-spring-boot-starter Bump mockito-core from 2.23.0 to 2.23.4 Nov 21, 2018
.gitignore Excluded jar files from repo Apr 26, 2018
.mailmap Replaced MAINTAINERS with AUTHORS Mar 18, 2016
.travis.yml Made Spring 5 the default Jul 4, 2018
.zappr.yaml Update and rename .zappr.yml to .zappr.yaml May 24, 2016
LICENSE Switched to MIT license Aug 30, 2016
MAINTAINERS Renamed AUTHORS to MAINTAINERS May 9, 2018
README.md Addedn Hystrix timeout as known issue Oct 28, 2018
SECURITY.md Added SECURITY file May 3, 2018
mvnw Updated to newer maven wrapper version without binary Apr 26, 2018
mvnw.cmd Updated to newer maven wrapper version without binary Apr 26, 2018
pom.xml Bumped spring 4.x version Nov 28, 2018
release.sh Updated release script Jun 19, 2018

README.md

Tracer: Distributed system tracing

Highway at Night

Stability: Active Build Status Coverage Status Code Quality Javadoc Release Maven Central License

Tracer noun, /ˈtɹeɪsɚ/: A round of ammunition that contains a flammable substance that produces a visible trail when fired in the dark.

Tracer is a library that manages custom trace identifiers and carries them through distributed systems. Traditionally traces are transported as custom HTTP headers, usually generated by clients on the very first request. Traces are added to any subsequent requests and responses, especially to transitive dependencies. Having a consistent trace across different services in a system allows to correlate requests and responses beyond the traditional single client-server communication.

This library historically originates from a closed-source implementation called Flow-ID. The goal was to create a clean open source version in which we could get rid of all the drawbacks of the old implementation, e.g. strong-coupling to internal libraries, single hard-coded header and limited testability.

  • Status: Under development and used in production

Features

  • Tracing of HTTP requests and responses
  • Customization by having a pluggable trace format and lifecycle listeners for easy integration
  • Support for Servlet containers, Apache’s HTTP client, Square's OkHttp, Hystrix, JUnit, AspectJ and (via its elegant API) several other frameworks
  • Convenient Spring Boot Auto Configuration
  • Sensible defaults

Dependencies

  • Java 8
  • Any build tool using Maven Central, or direct download
  • Servlet Container (optional)
  • Apache HTTP Client (optional)
  • OkHttp (optional)
  • Hystrix (optional)
  • AspectJ (optional)
  • JUnit (optional)
  • Spring 4.x or 5.x (optional)
  • Spring Boot 1.x or 2.x (optional)

Installation

Add the following dependency to your project:

<dependency>
    <groupId>org.zalando</groupId>
    <artifactId>tracer-core</artifactId>
    <version>${tracer.version}</version>
</dependency>

Additional modules/artifacts of Tracer always share the same version number.

Alternatively, you can import our bill of materials...

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.zalando</groupId>
      <artifactId>tracer-bom</artifactId>
      <version>${tracer.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

... which allows you to omit versions and scopes:

<dependency>
    <groupId>org.zalando</groupId>
    <artifactId>tracer-core</artifactId>
</dependency>
<dependency>
    <groupId>org.zalando</groupId>
    <artifactId>tracer-servlet</artifactId>
</dependency>
<dependency>
    <groupId>org.zalando</groupId>
    <artifactId>tracer-httpclient</artifactId>
</dependency>
<dependency>
    <groupId>org.zalando</groupId>
    <artifactId>tracer-okhttp</artifactId>
</dependency>
<dependency>
    <groupId>org.zalando</groupId>
    <artifactId>tracer-hystrix</artifactId>
</dependency>
<dependency>
    <groupId>org.zalando</groupId>
    <artifactId>tracer-aspectj</artifactId>
</dependency>
<dependency>
    <groupId>org.zalando</groupId>
    <artifactId>tracer-junit</artifactId>
</dependency>
<dependency>
    <groupId>org.zalando</groupId>
    <artifactId>tracer-spring-boot-starter</artifactId>
</dependency>

Usage

After adding the dependency, create a Tracer and specify the name of the traces you want to manage:

Tracer tracer = Tracer.create("X-Trace-ID");

If you need access to the current trace's value, call getValue() on it:

Trace trace = tracer.get("X-Trace-ID"); // this is a live-view that can be a shared as a singleton
entity.setLastModifiedBy(trace.getValue());

By default there can only be one trace active at a time. Sometimes it can be useful to stack traces:

Tracer tracer = Tracer.builder()
    .stacked(true)
    .trace("X-Trace-ID")
    .build();

Generators

When starting a new trace, Tracer will create trace value by means of pre-configured generator. You can override this on a per-trace level by adding the following to your setup:

Tracer tracer = Tracer.builder()
        .trace("X-Trace-ID", new CustomGenerator())
        .build();

There are several generator implementations included.

UUID

The uuid generator creates random-based UUID values.

5cdc0690-536b-11e8-8b88-6b0631b44b96

The generated values have a length of 36 characters.

Flow ID

The flow id generator creates 128-bit random integer values encoded in base64:

REcCvlqMSReeo7adheiYFA

The generated values have a length of 22 characters.

Phrase

The phrase generator creates over 10^9 different phrases like:

tender_goodall_likes_evil_panini
nostalgic_boyd_helps_agitated_noyce
pensive_allen_tells_fervent_einstein

The generates values have a length between 22 and 61 characters.

Random64 and Random128

The random generators create random hexadecimal integers of length 64- and 128-bit, respectively.

c94af1541b3e4f3a
1ff222514858965f0960bdbb78fe550e

The generated values have a length of 16 or 32 characters.

Listeners

For some use cases, e.g. integration with other frameworks and libraries, it might be useful to register a listener that gets notified every time a trace is either started or stopped.

Tracer tracer = Tracer.builder()
        .trace("X-Trace-ID")
        .listener(new CustomTraceListener())
        .build();

Tracer comes with a very useful listener by default, the MDCTraceListener:

Tracer tracer = Tracer.builder()
        .trace("X-Trace-ID")
        .listener(new MDCTraceListener())
        .build();

It allows you to add the trace id to every log line:

<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} [%X{X-Trace-ID}] - %msg%n"/>

Another built-in listener is the LoggingTraceListener which logs the start and end of every trace.

Servlet

On the server side is a single filter that you must be register in your filter chain. Make sure it runs very early — otherwise you might miss some crucial information when debugging.

You have to register the TracerFilter as a Filter in your filter chain:

context.addFilter("TracerFilter", new TracerFilter(tracer))
    .addMappingForUrlPatterns(EnumSet.of(REQUEST, ASYNC, ERROR), true, "/*");

Apache HTTP Client

Many client-side HTTP libraries on the JVM use the Apache HTTPClient, which is why Tracer comes with a request interceptor:

DefaultHttpClient client = new DefaultHttpClient();
client.addRequestInterceptor(new TracerHttpRequestInterceptor(tracer));

OkHttp

The tracer-okhttp module contains an Interceptor to use with the OkHttpClient:

OkHttpClient client = new OkHttpClient.Builder()
        .addNetworkInterceptor(new TracerInterceptor(tracer))
        .build();

Hystrix

Tracer comes with built-in Hystrix support in form of a custom HystrixConcurrencyStrategy:

final HystrixPlugins plugins = HystrixPlugins.getInstance();
final HystrixConcurrencyStrategy delegate = HystrixConcurrencyStrategyDefault.getInstance(); // or another
plugins.registerConcurrencyStrategy(new TracerConcurrencyStrategy(tracer, delegate));

AspectJ

For background jobs and tests, you can use the built-in aspect:

@Traced
public void performBackgroundJob() {
    // do work
}

or you can manage the lifecycle yourself:

tracer.start();

try {
    // do work
} finally {
    tracer.stop();
}

JUnit

Tracer comes with a special TestRule that manages traces for every test run for you:

@Rule
public final TracerRule tracing = new TracerRule(tracer);

It also supports JSR-330-compliant dependency injection frameworks:

@Inject
@Rule
public final TracerRule tracing;

Spring Boot Starter

Tracer comes with a convenient auto configuration for Spring Boot users that sets up aspect, servlet filter and MDC support automatically with sensible defaults:

Configuration Description Default
tracer.stacked Enables stacking of traces false
tracer.aspect.enabled Enables the TracedAspect true
tracer.async.enabled Enables for asynchronous tasks, i.e. @Async true
tracer.filter.enabled Enables the TracerFilter true
tracer.logging.enabled Enables the LoggingTraceListener false
tracer.logging.category Changes the category of the LoggingTraceListener org.zalando.tracer.Tracer
tracer.mdc.enabled Enables the MdcTraceListener true
tracer.scheduling.enabled Enables support for Task Scheduling, i.e. @Scheduled true
tracer.scheduling.pool-size Configures the thread pool size, i.e. the number of scheduled tasks # of CPUs
tracer.traces Configures actual traces, mapping from name to generator type (uuid, flow-id, phrase, random64, random128)
tracer:
    aspect.enabled: true
    async.enabled: true
    filter.enabled: true
    logging:
        enabled: false
        category: org.zalando.tracer.Tracer
    mdc.enabled: true
    scheduling.enabled: true
    traces:
        X-Trace-ID: uuid
        X-Flow-ID: flow-id

The TracerAutoConfiguration will automatically pick up any TraceListener bound in the application context.

Known Issues

Hystrix timeouts don't preserve the thread-local traces and applications get stuck. Please follow Netflix/Hystrix#1767 for more details. The only known viable workaround for now is to downgrade Hystrix at least to version 1.5.11.

Getting Help with Tracer

If you have questions, concerns, bug reports, etc., please file an issue in this repository's Issue Tracker.

Getting Involved/Contributing

To contribute, simply make a pull request and add a brief description (1-2 sentences) of your addition or change. For more details, check the contribution guidelines.

Alternatives

Tracer, by design, does not provide sampling, metrics or annotations. Neither does it use the semantics of spans as most of the following projects do. If you require any of these, you're highly encouraged to try them.