Profiling Integration on APM services uses incorrect data source

[JonasKunz]

While playing around with the Universal Profiling <-> APM integration on staging, I uncovered a bug in the Universal Profiling tab on the APM service UI.

I'm talking about this page:

When an APM-agent is running with the universal profiling correlation enabled, this page is supposed to show a flamegraph for all samples recorded for the process of the service.

However, what it actually shows is only the samples which have occurred within transactions of said service. This is visible by following the Go to Universal Profiling Flamegraph link:

Here the flamegraph shows much more data, consisting of samples which occurred within transaction AND samples of background tasks which are not covered by transactions.

Note that I manually had to add the service.name: "$serviceName" filter after following the link, this also seems like a bug to me.

I assume the root cause for this problem is that the Universal Profiling tab on the APM-service page uses the wrong data set for rendering the flamegraph:

• It appears to be using the transaction.profiler_stack_trace_ids field from APM-transaction documents
• It should be using the profiling indices instead filtered on the service.name (like shown in the second screenshot)

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)