Integration with Spring Sleuth

[meong1234]

it nice if axon provide integration with distrubuted tracing use spring sleuth
https://cloud.spring.io/spring-cloud-sleuth/

[meong1234]

`@Aspect
public class TraceEventHandler {

private static final String EVENT_HANDLER_COMPONENT = "eventhandler";

private final Tracer tracer;
private final TraceKeys traceKeys;
private final Pattern skipPattern;

public TraceEventHandler(Tracer tracer, TraceKeys traceKeys, Pattern skipPattern) {
    this.tracer = tracer;
    this.traceKeys = traceKeys;
    this.skipPattern = skipPattern;
}

@Around("execution (@org.axonframework.eventhandling.annotation.EventHandler  * *.*(..))")
public Object traceBackgroundThread(final ProceedingJoinPoint pjp) throws Throwable {
    if (this.skipPattern.matcher(pjp.getTarget().getClass().getName()).matches()) {
        return pjp.proceed();
    }
    String spanName = SpanNameUtil.toLowerHyphen(pjp.getSignature().getName());
    Span span = this.tracer.createSpan(spanName);
    this.tracer.addTag(Span.SPAN_LOCAL_COMPONENT_TAG_NAME, EVENT_HANDLER_COMPONENT);
    this.tracer.addTag(this.traceKeys.getAsync().getPrefix() +
        this.traceKeys.getAsync().getClassNameKey(), pjp.getTarget().getClass().getSimpleName());
    this.tracer.addTag(this.traceKeys.getAsync().getPrefix() +
        this.traceKeys.getAsync().getMethodNameKey(), pjp.getSignature().getName());
    try {
        return pjp.proceed();
    }
    finally {
        this.tracer.close(span);
    }
}

}`

example by use anotation @eventhandler

[jorgheymans]

just a thought, IIRC axon3 supports dropwizard, so you might be able to get something working combining https://github.com/opentracing-contrib/java-dropwizard , https://github.com/openzipkin/brave and the zipkin connector of spring cloud sleuth https://cloud.spring.io/spring-cloud-sleuth/spring-cloud-sleuth.html#_spring_cloud_sleuth_stream_zipkin_collector

[dzonekl]

Hey Guys,

We want to have a go at this. The impl. I am thinking of:

On a: QueryGateway / CommandGateway:

A MessageDispatchInterceptor, which gets Tracing injected, from which we get the current span, and add it as a MetaData to the Message.

Handler interceptors

A MessageHandlerInterceptor which gets Tracing injected , from which we spin a new child span.
This could potentially be a CorrelationDataProvider, so it maps the MetaData to other messages.

I notice CorrelationDataProvider is registered on the commandbus in AxonAutoConfiguration, and in EventHandlingConfiguration. Is it correct these are the only two places, where MHI can be registered?

Netty/Grpc: (Axonhub specific or distributed in general).

The MetaData could potentially also be written/read has b3 headers in Grpc or HTTP/1 communication. Not sure if happens by default somehow, or an Trace injector/extractor should be used.

Enabling the headers, could let other tools use the tracing information. (In our case we will use Spring Cloud Sleuth to send the tracing data to Zipkin).

Finally, some auto configuration could be created, to enable the Axon instrumentation with a good set of defaults.

I hope, for your feedback on this approach...

[smcvb]

@dzonekl on face value that doesn't sound to off for me to be honest.
Mainly posting a response here to notify you that we'll look in to your suggestion, but time-frame-wise I am hard pressed to tell you when that response will come.
We have to focus on our 4.0 deadline for next week Friday first, after that, I assume things like this will be fair game again.
So, that's my heads up for you. Thanks for showing interest of course!

[dzonekl]

Hi Steven, Thanks for the head-up.
I am actually working on it, made some good progress ... At current stage I implemented the
Dispatcher and Handler and an Injector/Extractor which puts traceId, spanId and parentSpanId into MetaData. This is propagated between VM's using AxonHub, so it already kind of works.
It's a rough cut, have time to continue on it now, and I will likely offer this as a PR.

Below you see two span's between two axon apps, separated by Axonhub.

Cheers Christophe

screenshot

[dzonekl]

@smcvb question, where does the tracing code fit best? As an extension or as part of the framework? There will be deps. on opentracing.io

[smcvb]

We'd very much like this to be part of our new 'extensions' idea, so yes, making it a separate repository to contain the code would be very nice. Might be worthwhile for you to start it of as a personal repository, for which we can discuss the migration (if you want to migrate ownership to us that is) at a latter stage.
Would that work out?

[dzonekl]

Sure it will, it's actually easier, are API's for extensions? What I would otherwise do, is to do an Autoconfigure, conditional on opentracing or sleuth dependency, and then register the various handlers. BTW, for the event processors, registering a handler in a generic way is tricky. The processor group names are not available at application configuration time. (They are resolved later) .

What I would like to do is this method in a @configuration class.

    @Autowired
    public void registerHandlers(EventProcessingConfiguration config, Tracer tracer){

        OpenTraceHandlerInterceptor openTraceHandlerInterceptor = new OpenTraceHandlerInterceptor(tracer);
        Map<String, EventProcessor> stringEventProcessorMap = config.eventProcessors();
        
        // map is empty! 
        stringEventProcessorMap.keySet().forEach( processorName ->
                config.usingTrackingProcessors().registerHandlerInterceptor(processorName, configuration -> openTraceHandlerInterceptor)
        );
    }

[jorgheymans]

@dzonekl just curious, is this an integration with sleuth or opentracing. They are not the same. Sleuth uses brave under the hood and is able to bridge traces to an opentracing backend.

[dzonekl]

@jorgheymans yeah, I am aware they are different and I have a few tested.
There are actually more combo's (as far as I know).

• sleuth -> zipkin/brave (tested, working).
• sleuth -> Jaeger (Not tested)
• opentracing -> jaeger (Testing as we speak).
• opentracing -> zipkin (Tested working, Which uses the brave bridge as you mentioned).

The focus is on opentracing/jaeger combination.

Rgds.