[APM] Improvements for profiler-inferred spans in waterfall view

[felixbarny]

Let's discuss how we can improve the experience of profiler-inferred spans.

Indicate that inferred spans are not exact measurements but rather estimations

Currently, inferred spans look just like regular ones. Rendering them with a gradient to fade them in and out could be used to convey that inferred spans are not exact measurements.

Favor transactions with inferred spans when showing a trace sample

Depending on the configuration, not all transactions will have inferred spans. If a latency bucket has both transactions with and without inferred spans, the UI should favor those with inferred spans.

Update: the logic changed a little in the Java agent. Now almost every sampled transaction should contain inferred spans, as the profiler is running continuously. There can still be some exceptions to that but this should happen much more infrequently.

Show stack trace above the span

Inferred spans can also have a stack trace attached to them. But they have slightly different semantics. They don't contain a stack trace that goes down to the very bottom of the stack. Instead, it's just the sequence of methods that took the same amount of time than the current span.

To illustrate this:

Stack traces

    fff  <- top of stack
    eee    
    ddd 
  cccccc   
  bbbbbb  
aaaaaaaaaa <- bottom of stack
1234567890 <- time

Waterfall

           stack trace
[a       ]
  [c   ]   [b]
    [f]    [d, e]
1234567890 <- time

We don't create a span for b, d, and e because they just lead up to c and f, respectively, and thus don't add any more information besides explaining the sequence of method calls that lead up to the spans. Hence, these methods are part of the span.stacktrace instead of being additional inferred spans.

Because the stack trace does not go to the very bottom of the stack, but rather to the next inferred span, we could display the stack trace a little bit different:

[a       ]                [a       ]   
  ^1 frame                  ^1 frame   
  [c   ]                    [c   ]     
    ^2 frames <-*click*->     ^
    [f]                       d
                              e
                              v
                              [f]

Impl note: inferred spans can be identified by their subtype: span.subtype: inferred.

Expand/collapse

We have an issue about this already (#20660) but in the context of inferred spans, it becomes more important. That is because there are more spans overall and it gets quite complex to keep track of which method caused which DB call, for example.

Documentation

To educate users on what inferred spans are and how they're different from regular ones, we could either have inline documentation (?) or link to external documentation. The advantage of the former is that there are fewer context switches but it makes updating the docs depend on the Kibana release cadence.

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[formgeist]

@cauemarcondes Does your PR #63695 also close this issue for the most part, at least as a first iteration?

[cauemarcondes]

@cauemarcondes Does your PR #63695 also close this issue for the most part, at least as a first iteration?

@formgeist it doesn't, it only handles the reparenting part.

[formgeist]

@felixbarny @nehaduggal It's not immediately apparent, after our discussions, what in this design issue needs actual design. There's an open issue for the expand/collapse rows and if we need to improve documentation, we probably need to open a separate issue. Based on @cauemarcondes PR #63695, what's left to do on this for now?

[felixbarny]

Then the only thing left would be Show stack trace above the span. But that is definitely lower in priority than the other items.

[formgeist]

I suggest we create a new issue for that and close this meta one. Agree?

[felixbarny]

I think we forgot about Indicate that inferred spans are not exact measurements but rather estimations. What's your opinion on that? Did you already create designs for that? IIRC, you mentioned concerns about increasing opacity for profiler-inferred spans, right?.

[formgeist]

I haven't done any design for an optimized experience of inferred spans yet, but if the indication that these are estimates are of high priority, we should open an issue so we can track it and get it done in time for the next release.

[felixbarny]

I've created #66149