Distributed Tracing between different Azure Function Instances

[plamber]

Hi @brettsam,

I want to express my gratitude for your outstanding work on the ApplicationInsights integration.

Currently, we are utilizing the latest versions of Microsoft.Azure.Functions.Worker 1.13.0 and Microsoft.Azure.Functions.Worker.ApplicationInsights 1.0.0-preview4.

Our team is investigating different possibilities for implementing distributed tracing between various external components that we own. These external components may consist of other dedicated Azure Functions that share the same Application Insights instance.

One of our use cases involves writing to a Queue to trigger specific processes. We were wondering if you have any recommendations on how we can enhance the queue message to include trace contexts of the parent?

Alternatively, if this isn't feasible, do you have any suggestions for linking operations carried out from the client or backend with the Azure Function operations, including queue messages?

Thank you for your insights,

Best regards,

Patrick

[brettsam]

Most distributed tracing should "just work" today. Functions will pass the parent id along to the worker, which will then set it on the Activity.Current and App Insights will use that for correlation.

However, Azure Storage Queues do not have any built-in mechanism for tracking parent ids today.

I chatted with @lmolkova about this yesterday -- she's said she can take up the task of putting together a sample that shows how this can be done.

[plamber]

Thank you @brettsam and @lmolkova.

An example would help a lot

[RobARichardson]

Most distributed tracing should "just work" today. Functions will pass the parent id along to the worker, which will then set it on the Activity.Current and App Insights will use that for correlation.

Is this true for Durable Functions because I can't seem to get it working in the same way that it works for In-Process Durable Functions. For example, a ServiceBusTrigger Function is not correlated with the Orchestrator instance scheduled by the ServiceBusTrigger Function or any of the Activities that are called from the Orchestrator.

[joproulx]

I have the same behavior as described @RobARichardson. Could you please confirm if distributed tracing should work for Durable Functions?

@brettsam I am trying to step in the code and I cannot see where the Activity.Current is set as you mentioned in your previous comment. Could you direct me to the place where this should be accomplished please?

[jviau]

Durable functions distributed tracing is a different solution than functions. @RobARichardson I see you already have an issue on the DF extension repo: Azure/azure-functions-durable-extension#2662 - please continue to use that issue there.

@plamber, as @brettsam said we should have Activity.Current hydrated in the worker today. Is it not working for you?

We are also looking to improve this experience with our OTel efforts: Azure/azure-functions-host#9273

[jviau]

Re-reading this issue, your specific ask is regarding Azure Storage queues. Unfortunately, the C# Azure Storage queues SDK does not flow trace context. So, we are unable to correlate the action which enqueued the message back to the function that processed it.

#1126

[microsoft-github-policy-service]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

If you are not the original author (plamber) and believe this issue is not stale, please comment with /bot not-stale and I will not close it.

[microsoft-github-policy-service]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.