Add query-tracer to profilecli

[simonswine]

Query tracer uses distributed tracing to analyse the query statistics of Pyroscope queries.

For a start this only looks at a few aspects, but this can be extended fairly easily, while we don't have to expose further information in the connect API.

 go run ./cmd/profilecli query-tracer --tempo-url="https://1:${LOKI_PASSWORD}@tempo-ops-us-east4.grafana.net/tempo" --interval 10m --filter-sensitive
level=info msg="search for matching traces" start=2024-01-25T18:08:16.486648Z end=2024-01-25T18:18:16.486648Z
level=info msg="found 2 matching traces"
level=info msg="query successfully traced" cluster=ops-us-east-0 duration=5.073891s grafana_plugin=grafana-pyroscope-app grafana_tenant_id=*** grafana_user=*** namespace=profiles-ops-001 query_duration=2h59m59.999s query_end_time=2024-01-24T14:59:59.999Z query_method=/pyroscope/render query_profile_type=process_cpu:cpu:nanoseconds:cpu:nanoseconds query_selector=*** query_start_time=2024-01-24T12:00:00Z request_end_time=2024-01-25T18:13:11.159987Z request_start_time=2024-01-25T18:13:06.086096Z response_size=197435 stats_profiles=700063 stats_profiles_fetched=1.525843e+06 status=200 trace_id=x
level=info msg="query successfully traced" cluster=ops-us-east-0 duration=4.945028s grafana_plugin=grafana-pyroscope-app grafana_tenant_id=*** grafana_user=*** namespace=profiles-ops-001 query_duration=2h59m59.999s query_end_time=2024-01-24T14:59:59.999Z query_method=/pyroscope/render query_profile_type=process_cpu:cpu:nanoseconds:cpu:nanoseconds query_selector=*** query_start_time=2024-01-24T12:00:00Z request_end_time=2024-01-25T18:13:29.474718Z request_start_time=2024-01-25T18:13:24.52969Z response_size=197531 stats_profiles=700063 stats_profiles_fetched=1.525843e+06 status=200 trace_id=x

[cyriltovena]

LGTM

I think we should also fix our tracing integrations that breaks with stream. Not sure if that has affected you though