Trouble interpretting Jetbrains DotTrace backtraces involving Pipelines.Sockets.Unofficial via StackExchange.Redis

[kieranbenton]

Hi,
Firstly, StackExchange.Redis is an amazing library and I'm loving the improvements in stability to it since you've switched over to using piplines.

I think I may be doing something wrong, but I can't find a way forward myself and this seems to be originating in this library so thought I would ask some advice.

Have been playing with a toy project to get into .NET core development and pick up tools I haven't used in while (like dottrace) - and ran into trouble interpretting its output to try and find the slow/hot path in the app when SE.Redis is involved:

Specifically I'm trying to backtrace out to which parts of my code are initiating the calls in the first place - but dottrace can't see any 'higher' than:
System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) which immediately calls into Pipelines.Sockets.Unofficial.DedicatedThreadPoolPipeScheduler.RunWorkLoop().

Given that I never used to have this problem I am making the assumption that through the DedicatedThreadPoolPipeScheduler tasks are being scheduled in a way that means dottrace can't piece together the stack and so my trail runs cold.

So really a few questions:

1. Is my analysis right? Or way off the mark.
2. Is there anything I can do about it? Or is this a bug with dottrace? Or am I being an idiot and I'm just missing something basic.
3. If dottrace is blocked by this, are there other tools (perfview?) that might be worth learning to help diagnose issues?

[mgravell]

The fundamental issue here is that there is a fundamental disconnect here between requests and responses - the loop that dequeues network traffic, processes frames, and matches each response to the originating request - is not, and cannot be, directly tied into the same context that is sending requests, because the core of the lib is not "request, wait/await, response" - it heavily reuses minimal connections, sending lots of messages from separate threads down the same socket, and just keeping track of the request order.

So; if what you're asking is "how can I trace this from end to end" - I don't have a mechanism to make that work; we're not passing flow-context here - rather, we're simply signalling completion via TaskCompletionSource<T> as we identify responses.

---

Unrelated: I have some other work in the pipe to make protobuf-net work much more efficiently with pipelines, and this has already absorbed changes to make it work with the kinds of buffers that RedisValue makes available, so: there might be some other gains up ahead if you're using binary, although it looks like you're using base-64. I wonder whether there might be an alternative scheme possible there in the future where you lease an oversized array from ArrayPool, decode into that array, then pass the oversized array to protobuf (and finally release the array back to the pool)

[kieranbenton]

Thanks @mgravell - that does make sense, and at least confirms roughly what I thought, and thanks for the heads up on the protobuf support for pipelines as well. This isnt a super performant demo yet and I'm just tweaking as I go to get back into some skills (perf analysis) that I haven't used for a while.

Am I completely SOA here? Given its deeply entrenched position I'm assuming turning pipelining 'off' is a non-starter. Can I ask how you would approach finding the upstream hot path under these circumstances?

[mgravell]

did you mean SOL? yes, the codebase is written fully with that intent

[kieranbenton]

Ha, yes I meant SOL. Obviously having flashbacks today to other projects!