Enhance APM UI Tracing Features for RAG Workflows

[neptunian]

Summary

This issue is a list of enhancements aimed at improving tracing in the APM UI, particularly for RAG applications. The need for these improvements has emerged as the Security team has been integrating tracing functionalities within the Security AI assistant.

Issues

• [APM] Enable embedding of Waterfall view in other plugins #172411 Create a Waterfall Embeddable to improve in-app UX
• [APM] Enable ability to show additional metadata for a given span within the waterfall view #172413 Enhance the span details to include additional metadata, such as token counts, which will provide valuable insights directly within the traces.
• [APM] Enable customization to collapse specific spans within the waterfall view #172412 Introduce the ability to collapse certain spans to clean up the UI and focus on the most relevant data, reducing clutter from common but less critical information like ES authentication spans.
• [APM] Enable ability to provide custom renderers for span labels shown in span details flyout #172415 Although less critical once the embeddable waterfall component is in place, adding custom renderers could highlight specific documents retrieved within the RAG pipeline, enhancing the detail available in span explorations.
• [APM] When creating Trace Explorer links transaction.id is replaced when page is loaded #171368 Trace explorer bug

Related

• should events on an outgoing HTTP request (ClientRequest) object be bound to the Span context for that request? apm-agent-nodejs#3734 Needed to capture the outgoing request from Kibana to the LLM provider, otherwise we can't see what the LLM is actually being sent

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[smith]

@roshan-elastic

[roshan-elastic]

@smith - Do you think this set of issues should be 'batched' like we're doing with OTel and Profiling or do you think we should 'trickle' them through (i.e. do them in parallel with batched stuff like 'OTel')?

This seems like something we could prioritise

[smith]

@roshan-elastic

@smith - Do you think this set of issues should be 'batched' like we're doing with OTel and Profiling or do you think we should 'trickle' them through (i.e. do them in parallel with batched stuff like 'OTel')?

Yes I think usually if we batch things up into epics for maintenance it's easier to work on and easier to communicate the value of the changes we're making.

[roshan-elastic]

Cheers @smith - I've added the label APM AI RAG so we can trickle these through once we've done the OTel stuff.

[smith]

@roshan-elastic let's remove that label and convert these to sub-issues, then we can track this as an internal epic.