[APM] Enable ability to show additional metadata for a given span within the waterfall view

[spong]

In #171153 we instrumented the Security Solution Assistant with APM so we could trace and debug our calls to LLM's.

This is an enhancement request to enable the ability to show additional metadata for a given span within the waterfall view. For instance, it would be really beneficial if we could surface the token count (which exists as a label on the span) on relevant spans. Extra awesomeness here if we could also set the icon (like ES spans have) and provide a renderer for showing detailed information in a popover/tooltip on hover like is done in the LangSmith tracing image below:

Note: In discussion with @smith, looks like this lives around here in the codebase.

[elasticmachine]

Pinging @elastic/apm-ui (Team:APM)

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.