OpenTelemetry Java static instrumentation

Project structure

Project consists of the following modules:

• app-instrumenter - Maven plugin, which uses instrumented OpenTelemetry agent to statically instrument specified project's artifacts. Responsible for repackaging and injecting needed agent code to final jar.
• agent-instrumenter - logic for instrumenting OpenTelemetry agent. Automatically injects instrumented agent to app-instrumenter module resources.
• instrumentation-tests - tests, which verify whether instrumentation is actually applied. Configured with Github Actions.

Parameters

• artifactName - name of the artifact that should be instrumented. Plugin will search for artifact with this exact name. If such artifact won't be found, plugin will default to instrumenting all project artifacts.

• outputFolder - path to output folder for instrumented JAR. If no name for the folder is provided, the instrumented JAR will be stored in the default folder for Maven project.

• suffix - suffix will be added to your file name after the instrumentation. The default suffix is '-instrumented' (e.g 'example-2.1.jar' will become 'example-2.1-instrumented.jar').

• noSuffix - if set to true, no suffix will be added to file name.

Execution

Add plugin to app

1. Add following config to project's pom.xml file

<plugin>
    <groupId>agh.edu.pl</groupId>
    <artifactId>app-instrumenter</artifactId>
    <version>1.0-SNAPSHOT</version>
    <executions>
        <execution>
            <goals>
                <goal>instrument</goal>
            </goals>
        </execution>
    </executions>
</plugin>

2. Run mvn package in project's folder.

3. Run instrumented app with -Dio.opentelemetry.javaagent.shaded.io.opentelemetry.context.contextStorageProvider=default system property.

Example: java -jar -Dio.opentelemetry.javaagent.shaded.io.opentelemetry.context.contextStorageProvider=default target/your-app-instrumented.jar

Build plugin

Make sure that opentelemetry-javaagent.jar file is present in this repo root folder.

1. If any changes to agent-instrumenter are made, AgentInstrumenter class must be run. It takes OpenTelemetry agent from resources, instruments it and injects to app-instrumenter resources.

NOTE: If this step is omitted, pre-commit hook will fail.

2. Run mvn --projects app-instrumenter package

3. Run mvn --projects app-instrumenter install

Code quality

Project uses Spotless. To apply Spotless

• to all modules run mvn spotless:apply
• to a specific module run mvn --projects <module-name> spotless:apply

Tests results

Spring PetClinic - microservices, JAR

App GitHub: https://github.com/spring-petclinic/spring-petclinic-microservices

OpenTelemetry javaagent version: 1.9.0

Tests were run on Docker

Summary report:

Average initialization time overhead for dynamic instrumentation: 40%

Average initialization time overhead for static instrumentation: 5%

Spans genereration:

Compared with dynamic instrumentation, static instrumentation was missing spans related to Docker environment (because at the time the static instrumentation takes place there is no information that app will run on Docker, and related modules cannot be instrumented). However, those spans were a tiny fraction of all spans generated by particular microservices (2-3 spans of over 100).

Full report: https://docs.google.com/document/d/1z8wkFOvVERpbNYcXzsSyLEgn6dajHAILymLHGjX7agU/edit?usp=sharing

Spring PetClinic - WAR

App GitHub: https://github.com/spring-petclinic/spring-framework-petclinic

OpenTelemetry javaagent version: 1.9.0

Tests were run on Tomcat 9.0.55

Summary report:

Spans and metrics generation was insufficient, mainly because of the fact that the majority of dependecies for WAR file was delivered with the environment, in cause of those tests - Tomcat. Instrumentation of those dependencies was impossible during the mvn package phase.

The main problem was that the runtime environment was not defined when the static instrumentation took place. For apps that fall insto that category, the dynamic instrumentation should be recommended.