About Replacing SpanNameExtractor and AttributeExtractor for Specific Instrumentation When Building an Extension JAR

[Rebwon]

Hello, I am configuring an extension based on the following OpenTelemetry example project:

https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/examples/extension

Currently, in the Spring Scheduling instrumentation, the instrumented span name is displayed as a class that wraps our internal Runnable implementation.

For explanation purposes, I will call this wrapper CustomRunnable.

What I want to do is:

• unwrap CustomRunnable
• store the actual target Runnable information into span attributes such as code.function and code.namespace
• change the span name to the format:
  TargetRunnableClassName.MethodName

However, after trying to implement this by referencing the example project, it seems I have no choice but to use reflection. Also, because the agent JAR class loader and the actual application class loader are different, I am having difficulties implementing this properly.

I would like to ask whether there is a simpler or more recommended approach than the method above.

[Rebwon]

For example, if we define a CodeAttributesGetter like below and register an InstrumenterCustomizerProvider via SPI to use it in SpanNameExtractor and AttributesExtractor,

the problem is that it cannot find the application class loader, so it does not work as intended.

public class SpringSchedulingCodeAttributesGetter implements CodeAttributesGetter<Runnable> {
    @Nullable
    private static final Class<?> OUTCOME_TRACKING_RUNNABLE_CLASS = getRunnableClass(
            "org.springframework.scheduling.config.Task$OutcomeTrackingRunnable"
    );

    @Nullable
    private static final Field OUTCOME_TRACKING_RUNNABLE_FIELD = getRunnableField(OUTCOME_TRACKING_RUNNABLE_CLASS);

    @Nullable
    private static final Class<?> SAMPLE_CONTEXT_RUNNABLE_CLASS = getRunnableClass(
            "com.sample.SampleContextRunnable"
    );

    @Nullable
    private static final Field SAMPLE_CONTEXT_RUNNABLE_FIELD = getRunnableField(SAMPLE_CONTEXT_RUNNABLE_CLASS);

    @Nullable
    private static final Class<?> SCHEDULED_METHOD_RUNNABLE_CLASS = getRunnableClass(
            "org.springframework.scheduling.support.ScheduledMethodRunnable"
    );

    @Nullable
    private static final Method SCHEDULED_METHOD_RUNNABLE_GET_METHOD =
            getNoArgMethod(SCHEDULED_METHOD_RUNNABLE_CLASS, "getMethod");

    @Nullable
    private static Method getNoArgMethod(@Nullable Class<?> clazz, String methodName) {
        if (clazz == null) {
            return null;
        }
        try {
            Method method = clazz.getMethod(methodName);
            method.setAccessible(true);
            return method;
        } catch (Exception ignored) {
            return null;
        }
    }

    @Nullable
    private static Method getScheduledMethod(Runnable runnable) {
        if (SCHEDULED_METHOD_RUNNABLE_CLASS == null
                || SCHEDULED_METHOD_RUNNABLE_GET_METHOD == null
                || !SCHEDULED_METHOD_RUNNABLE_CLASS.isAssignableFrom(runnable.getClass())) {
            return null;
        }

        try {
            return (Method) SCHEDULED_METHOD_RUNNABLE_GET_METHOD.invoke(runnable);
        } catch (Exception ignored) {
            return null;
        }
    }

    @Nullable
    private static Class<?> getRunnableClass(String className) {
        try {
            return Class.forName(className);
        } catch (ClassNotFoundException ignored) {
            return null;
        }
    }

    @Nullable
    private static Field getRunnableField(@Nullable Class<?> clazz) {
        if (clazz == null) {
            return null;
        }
        try {
            Field field = clazz.getDeclaredField("runnable");
            field.setAccessible(true);
            return field;
        } catch (Exception ignored) {
            return null;
        }
    }

    private static Runnable unwrap(Runnable runnable) {
        if (OUTCOME_TRACKING_RUNNABLE_CLASS != null
                && OUTCOME_TRACKING_RUNNABLE_FIELD != null
                && OUTCOME_TRACKING_RUNNABLE_CLASS.isAssignableFrom(runnable.getClass())) {
            try {
                return unwrap((Runnable) OUTCOME_TRACKING_RUNNABLE_FIELD.get(runnable));
            } catch (IllegalAccessException ignored) {
                // should not happen because setAccessible was called
            }
        }

        if (SAMPLE_CONTEXT_RUNNABLE_CLASS != null
                && SAMPLE_CONTEXT_RUNNABLE_FIELD != null
                && SAMPLE_CONTEXT_RUNNABLE_CLASS.isAssignableFrom(runnable.getClass())) {
            try {
                return unwrap((Runnable) SAMPLE_CONTEXT_RUNNABLE_FIELD.get(runnable));
            } catch (IllegalAccessException ignored) {
                // should not happen because setAccessible was called
            }
        }

        return runnable;
    }

    @Override
    public Class<?> getCodeClass(Runnable runnable) {
        runnable = unwrap(runnable);
        Method method = getScheduledMethod(runnable);
        if (method != null) {
            return method.getDeclaringClass();
        }
        return runnable.getClass();
    }

    @Override
    public String getMethodName(Runnable runnable) {
        runnable = unwrap(runnable);
        Method method = getScheduledMethod(runnable);
        if (method != null) {
            return method.getName();
        }
        return "run";
    }
}

@AutoService(InstrumenterCustomizerProvider.class)
public class SampleInstrumenterCustomizerProvider implements InstrumenterCustomizerProvider {
    private static final CodeAttributesGetter<Runnable> SPRING_SCHEDULING_CODE_ATTRIBUTES_GETTER = new SpringSchedulingCodeAttributesGetter();

    @Override
    public void customize(InstrumenterCustomizer customizer) {
        String instrumentationName = customizer.getInstrumentationName();
        if (instrumentationName.contains("spring-scheduling")) {
            customizer.setSpanNameExtractorCustomizer(
                    it -> CodeSpanNameExtractor.create(SPRING_SCHEDULING_CODE_ATTRIBUTES_GETTER)
            );
            customizer.addAttributesExtractor(
                    CodeAttributesExtractor.create(SPRING_SCHEDULING_CODE_ATTRIBUTES_GETTER)
            );
        }
    }
}

[laurit]

it seems I have no choice but to use reflection

in agent extensions this is true, as you have noted code is in different class loaders

    private static Class<?> getRunnableClass(String className) {
        try {
            return Class.forName(className);
        } catch (ClassNotFoundException ignored) {
            return null;
        }
    }

since SpringSchedulingCodeAttributesGetter is in agent class loader it won't find org.springframework.scheduling.config.Task$OutcomeTrackingRunnable and other classes from you application.

[Rebwon]

Additionally, when it comes to spans instrumented by Tomcat or Kafka, I haven’t really found a good way to make the span names follow something like {ClassName}.{MethodName} unless we create entirely new custom instrumentation ourselves.

For Tomcat, the span names are usually something like {HTTP_METHOD} {URL}, and for Kafka they are typically {TOPIC_NAME} {OPERATION} (e.g. process, publish). From the perspective of actually analyzing traces, it’s difficult to clearly understand which class and method were executed just by looking at those names.

Do you happen to have any better guidance or recommended approaches for customizing span names?

It looks like the frameworks provide various extension points, but in practice, it often feels like achieving meaningful customization requires overriding or rebuilding almost the entire instrumentation implementation.

[laurit]

Additionally, when it comes to spans instrumented by Tomcat or Kafka, I haven’t really found a good way to make the span names follow something like {ClassName}.{MethodName} unless we create entirely new custom instrumentation ourselves.

Where should class and method name come from? If it is something that is not recorded by our instrumentation you may need to create a custom instrumentation that replaces our instrumentation or augments it in desired way e.g. you could have an instrumentation that calls Span.current().updateName(...) to overwrite the name set by us.

For Tomcat, the span names are usually something like {HTTP_METHOD} {URL}, and for Kafka they are typically {TOPIC_NAME} {OPERATION} (e.g. process, publish).

these names come from opentelemetry semantic conventions

It looks like the frameworks provide various extension points, but in practice, it often feels like achieving meaningful customization requires overriding or rebuilding almost the entire instrumentation implementation.

I think it largely depends on what you need to do. If you have ideas on how this could be improved then pull requests are welcome in this project.

[Rebwon]

Thank you for the detailed explanation.

So in the end, if the information is not recorded by the existing instrumentation, it seems there is no choice but to create a custom instrumentation that replaces the existing one.

However, custom instrumentation would ultimately need to be an extension of the existing instrumentation, and maintaining that does not seem very easy. In practice, it often means copying and pasting most of the existing implementation and modifying only the necessary parts, which also means that whenever the original instrumentation changes, those changes need to be tracked and reflected manually as well.

[laurit]

I think it really depends on what you need to do. Sometimes it could be easy, for example if you need to add an attribute to the server span created by tomcat (or some other) instrumentation you could just call Span.current().setAttribute(...) or LocalRootSpan.current().setAttribute(...). Other times it could be complicated.

In practice, it often means copying and pasting most of the existing implementation and modifying only the necessary parts, which also means that whenever the original instrumentation changes, those changes need to be tracked and reflected manually as well.

You could consider whether the functionality you are implemented is suitable for upstream contribution. Or you could help us figure out what we could do to make whatever it is you need to get done easier.