[FEATURE REQ] More detailed logging on SDK internal flow

[XiaofeiCao]

Is your feature request related to a problem? Please describe.
VMWare is migrating to track2, and they met occasional thread stuck. When it happens, there's no HttpClient/Netty log indicating the request is sent or not.
Since the request was never reached backend service, the stuck is most likely on SDK side. We are unsure where, whether it's in HttpPipeline, or Reactor chain, or HttpClient/Netty.

I'm aware that there are two types of logs in current SDK: Request/response log in HttpLoggingPolicy, and HttpPipeline tracer. Though they both track the final HTTP request, not our internal pipeline flow.

Describe the solution you'd like
We need more detailed log to track the client-side request lifecycle, e.g. which HttpPipelinePolicy has the request reached, optimally both inbound and outbound.

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Description Added
• Expected solution specified

/cc @alzimmermsft @weidongxu-microsoft

[lmolkova]

@XiaofeiCao have you looked into distributed tracing?
We create a span per public API call (well, ideally, in Java we create one in Rest-proxy and it's not great) and per each HTTP request. Some details on what we put are here https://github.com/Azure/azure-sdk/blob/main/docs/tracing/distributed-tracing-conventions.md#public-api-calls

They are industry-wide practice on how to expose the details to client applications in a structured and visualization/analysis-friendly way. More verbose data can come, but would be expensive to collect and store, and should be off by default. For example per-policy logs or traces would be overwhelming.

We can definitely add ad-hoc logs into policies to record key events, configs, branches, etc if we see a log is missing. We should also enrich our public spans with more details and are actively looking for customer feedback.

Let me know what you think.

[lmolkova]

Also, one more thing to explore when diagnosing threading and async issues is Reactor metrics - https://projectreactor.io/docs/core/release/reference/#metrics and JVM metrics supported by OpenTelemetry or Micrometer. They would be much more frugal way to find reactor-related, threading, locking, etc issues.
Reactor metrics documentation needs some love, but maybe VMWare can pull some internal strings to make it better?

[XiaofeiCao]

Thanks @lmolkova ! The distributed tracing best fits our goal(glad to know that my previous perception of out Tracer is wrong). Though there may be more work to do for VMWare to integrate with it(or not?).

We'll definitely consider Reactor metrics as well. Appreciate your help!

[lmolkova]

Thanks @XiaofeiCao , please let me know if you need any help or have examples where we can improve things - there is a lot we can do and should improve and I'm sure we can find ways!

[weidongxu-microsoft]

Trying some Reactor log/metric in https://github.com/weidongxu-microsoft/az-java-sdk-aad/blob/main/src/main/java/com/microsoft/azure/aad/Main.java