How to link traces between HTTP/SQLClient/Messaging in same service

[FanneYang]

Hi Team,

We are running a micro-service with three major functions - receive HTTP Client call, interact with Database (read/write) and also publish a Kafka message if write action to database is successful.

We add below code to add HttpClientInstrumentation, SqlClientInsstrumentation and ConfluentKafkaInstrumentation. We are able to see traces however they are separated without parent id linkage (so for one http call and the db connection/kafka message triggered by that http call, we have 3 traces instead of 1). Wonder if there is a way to link those traces as a whole instead of break apart?

            Sdk.CreateTracerProviderBuilder()
                    .SetResourceBuilder(thisService)
                    .AddXRayTraceId()
                    .AddAWSInstrumentation()
                    .AddAspNetCoreInstrumentation()
                    .AddHttpClientInstrumentation()
                    .AddSqlClientInstrumentation(options => options.EnableConnectionLevelAttributes = true)
                    .AddConfluentKafkaInstrumentation()
                    .AddOtlpExporter(options =>
                    {
                        options.Protocol = OtlpExportProtocol.Grpc;
                        options.Endpoint = new Uri(otelCollectorSettings.Endpoint);
                    })
                    .Build();

Thanks a lot for helping!

[vishweshbankwar]

@FanneYang - You should see 3 spans (same traceId) (one for each, httpClient, SqlClient and Kafka(assuming it creates one)). All three of them should be linked to a parent span which will be created during incoming request (AddAspNetCoreInstrumentation()).

Would recommend to use just these 4 instrumentations first and confirm if you are seeing the behavior noted above.

[FanneYang]

Hi @vishweshbankwar - I did the instrumentation already like the code above, however I see 3 separate trace ids instead of 3 spans with the same trace id. Do you know is there any configuration I might missing to link them together?

[vishweshbankwar]

@FanneYang Have you tried without .AddXRayTraceId() , .AddAWSInstrumentation()? I am not familiar with how those instrumentations work.

If you still see the behavior, share a minimal repro.

[FanneYang]

Let me try that - will keep you updated! Thanks!

[FanneYang]

Another question though - if each instrumentation has a separate ActivitySource created (as they are different libraries), will that cause the trace id separate?

[vishweshbankwar]

if each instrumentation has a separate ActivitySource created (as they are different libraries), will that cause the trace id separate

No