Java Agent for OpenTracing,
with auto-instrumentation

Automatically instruments 3rd-party libraries in Java applications

Overview

This file contains the operational instructions for the use and development of SpecialAgent.

Definitions

SpecialAgent

A Java agent that automatically instruments distinct 3rd-party libraries in Java applications via the OpenTracing API.

Tracer

Vendor implementation of the io.opentracing.Tracer interface.

Examples:

• Jaeger Tracer
• LightStep Tracer
• DataDog Tracer
• MockTracer

Tracers ARE NOT coupled to the SpecialAgent.

Tracer Plugin

A bridge providing automatic discovery of Tracers in a runtime instrumented with the OpenTracing API. This bridge implements the TracerFactory interface of TracerResolver, and is distributed as a single "fat JAR" that can be conveniently added to the classpath of a Java process.

Tracer Plugins ARE NOT coupled to the SpecialAgent.

Instrumentation Plugin

An OpenTracing Instrumentation project that exist as individual repositories under opentracing-contrib.

Examples:

• opentracing-contrib/java-okhttp
• opentracing-contrib/java-jdbc
• opentracing-contrib/java-jms

Instrumentation Plugins ARE NOT coupled to the SpecialAgent.

Instrumentation Rules

A submodule of the SpecialAgent that implements the auto-instrumentation rules for Instrumentation Plugins via the opentracing-specialagent-api.

Examples:

• rules/opentracing-specialagent-okhttp
• rules/opentracing-specialagent-jdbc
• rules/opentracing-specialagent-jms-1
• rules/opentracing-specialagent-jms-2

Instrumentation Rules ARE coupled to the SpecialAgent.

Supported Instrumentation Plugins

These plugins have Instrumentation Rules implemented.

1. OkHttp3
2. JDBC API (java.sql)
3. Concurrent API (java.util.concurrent)
4. Java Web Servlet API (javax.servlet)
5. Mongo Driver
6. Apache Camel
7. AWS SDK
8. Cassandra Driver
9. JMS API (javax.jms v1 & v2)
10. Elasticsearch6 Client
11. RxJava 2
12. Kafka Client
13. AsyncHttpClient
14. RabbitMQ Client

Goals

1. The SpecialAgent must allow any Instrumentation Plugin available in opentracing-contrib to be automatically installable in applications that utilize a 3rd-party library for which an Instrumentation Plugin exists.
2. The SpecialAgent must automatically install the Instrumentation Plugin for each 3rd-party library for which a module exists, regardless in which class loader the 3rd-party library is loaded.
3. The SpecialAgent must not adversely affect the runtime stability of the application on which it is intended to be used. This goal applies only to the code in the SpecialAgent, and cannot apply to the code of the Instrumentation Plugins made available in opentracing-contrib.
4. The SpecialAgent must support static and dynamic attach to applications running on JVM versions 1.7, 1.8, 9, and 11.
5. The SpecialAgent must implement a lightweight test methodology that can be easily applied to a module that implements instrumentation for a 3rd-party library. This test must simulate:
   1. Launch the test in a process simulating the -javaagent vm argument that points to the SpecialAgent (in order to test auto-instrumentation functionality).
   2. Elevate the test code to be executed from a custom class loader that is disconnected from the system class loader (in order to test bytecode injection into an isolated class loader that cannot resolve classes on the system classpath).
   3. Allow tests to specify their own Tracer instances via GlobalTracer, or initialize a MockTracer if no instance is specified. The test must provide a reference to the Tracer instance in the test method for assertions with JUnit.
6. The SpecialAgent must provide a means by which Instrumentation Plugins can be configured before use on a target application.

Non-Goals

1. The SpecialAgent is not designed to modify application code, beyond the installation of Instrumentation Plugins. For example, there is no facility for dynamically tracing arbitrary code.

Operation

When SpecialAgent attaches to an application, either statically or dynamically, it will automatically load the Instrumentation Plugins explicitly specified as dependencies in its POM.

Any exception that occurs during the execution of the bootstrap process will not adversely affect the stability of the target application. It is, however, possible that the Instrumentation Plugin code may result in exceptions that are not properly handled, and could destabilize the target application.

Installation

The SpecialAgent has 2 artifacts: main and test. These artifacts are built by Maven, and can be obtained by cloning this repository and following the Building instructions, or downloading directly from Maven's Central Repository.

1.     RELEASE: opentracing-specialagent-1.0.2.jar
   SNAPSHOT: opentracing-specialagent-1.0.3-SNAPSHOT.jar

   This is the main artifact that contains within it the Instrumentation Plugins from the opentracing-contrib organization for which Instrumentation Rules have been implemented. This JAR can be specified as the -javaagent target for static attach to an application. This JAR can also be executed, standalone, with an argument representing the PID of a target process to which it should dynamically attach. Please refer to Usage section for usage instructions.

2.     RELEASE: opentracing-specialagent-1.0.2-tests.jar
   SNAPSHOT: opentracing-specialagent-1.0.3-SNAPSHOT-tests.jar

   This is the test artifact that contains within it the AgentRunner, which is a JUnit runner class provided for testing of the ByteBuddy auto-instrumentation rules. This JAR does not contain within it any Instrumentation Plugins themselves, and is only intended to be applied to the test phase of the build lifecycle of a single plugin for an Instrumentation Plugin implementation. For direction with the AgentRunner, please refer to the opentracing-specialagent-api module.

Usage

The SpecialAgent uses Java’s Instrumentation mechanism to transform the behavior of a target application. The entrypoint into the target application is performed via Java’s Agent convention. SpecialAgent supports both static and dynamic attach.

Static Attach

Statically attaching to a Java application involves the use of the -javaagent vm argument at the time of startup of the target Java application. The following command can be used as an example:

java -javaagent:opentracing-specialagent-1.0.2.jar -jar myapp.jar

This command statically attaches SpecialAgent into the application in myapp.jar.

Dynamic Attach

Dynamically attaching to a Java application involves the use of a running application’s PID, after the application’s startup. The following commands can be used as an example:

1. Call this to obtain the PID of the target application:

   jps

2. For jdk1.8

   java -Xbootclasspath/a:$JAVA_HOME/lib/tools.jar -jar opentracing-specialagent-1.0.2.jar <PID>

3. For jdk9+

   java -jar opentracing-specialagent-1.0.2.jar <PID>

Configuration

The SpecialAgent exposes a simple pattern for configuration of SpecialAgent, the Instrumentation Plugins, as well as Tracer Plugins. The configuration pattern is based on system properties, which can be defined on the command-line, in a properties file, or in @AgentRunner.Config:

1. Properties passed on the command-line via -D${PROPERTY}=... override same-named properties defined in layers below.

2. The @AgentRunner.Config annotation allows one to define log level and transformation event logging settings. Properties defined in the @Config annotation override same-named properties defined in layers below.

3. The -Dconfig=${PROPERTIES_FILE} command-line argument can be specified to have the SpecialAgent load property names from a ${PROPERTIES_FILE}. Properties defined in the ${PROPERTIES_FILE} override same-named properties defined in the layer below.

4. The SpecialAgent has a default.properties file that defines default values for properties that need to be defined.

Selecting the Tracer Plugin

The SpecialAgent supports 2 ways to connect to a OpenTracing-compatible tracer:

1. Internal Tracer Plugins

   The SpecialAgent includes the collowing Tracer Plugins:

   1. Jaeger Tracer Plugin
   2. LightStep Tracer Plugin

   The -Dsa.tracer=${TRACER_PLUGIN} property is used on the command-line to specify which Tracer Plugin will be used. The value of ${TRACER_PLUGIN} is the short name of the Tracer Plugin, i.e. jaeger or lightstep.

2. External Tracer Plugins

   The SpecialAgent allows external Tracer Plugins to be attached to the runtime.

   The -Dsa.tracer=${TRACER_JAR} property is used on the command-line to specify the JAR path of the Tracer Plugin to be used. The ${TRACER_JAR} must be a JAR that conforms to the TracerFactory API of the TracerResolver project.

NOTE: If a tracer is not specified with the -Dsa.tracer=... property, the SpecialAgent will present a warning in the log that states: Tracer NOT RESOLVED.

Disabling Instrumentation Plugins

The SpecialAgent allows Instrumentation Plugins to be disabled. To disable a plugin, specify a system property, either on the command-line or in the properties file referenced by -Dconfig=${PROPERTIES_FILE}.

${PLUGIN_NAME}.enable=false

The value of ${PLUGIN_NAME} is the artifactId of the plugin as it is included in the SpecialAgent, such as opentracing-specialagent-okhttp or opentracing-specialagent-web-servlet-filter.

Building

Prerequisite: The SpecialAgent requires Oracle Java to build successfully.

The SpecialAgent is built in 2 passes that utilize different profiles:

1. The default profile is used for development of Instrumentation Rules. It builds and runs tests for each rule, but does not bundle the rules into opentracing-specialagent-1.0.2.jar

   To run this profile:

   mvn clean install

2. The assemble profile is used to bundle the Instrumentation Rules into opentracing-specialagent-1.0.2.jar. It builds each rule, but does not run tests. Once the build with the assemble profile is finished, the opentracing-specialagent-1.0.2.jar will contain the built rules inside it.

   Note: If you do not run this step, the opentracing-specialagent-1.0.2.jar from the previous step will not contain any Instrumentation Plugins within it!

   Note: It is important to not run Maven's clean lifecycle when executing the assemble profile.

   To run this profile:

   mvn -Dassemble package

• For a one-line build command to build SpecialAgent, its rules, run all tests, and create the assemble package:

  mvn clean install && mvn -Dassemble package

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

License

This project is licensed under the Apache 2 License - see the LICENSE.txt file for details.