use spanmetricsprocessor alike to generate metrics from span

[futurist]

Is your feature request related to a problem? Please describe.

Currently the metrics is hard coded into middleware, in below section:

obs-opentelemetry/tracing/middleware.go

Line 88 in f719a52

metricsAttributes := extractMetricsAttributesFromSpan(span)

It's not good for extension, and a open-telemetry collector processor can be a good way for the job.

Describe the solution you'd like

Use spanmetricsprocessor or create a new processor for hertz to extract metrics.

[CoderPoet]

@futurist Very good question, here are some of my considerations

There are some problems with the community's spanmetrics processor:

1. The generated metrics cannot be used to draw topology, only convert metrics based on a single span, rather than correlating upstream and downstream spans resource.service_name semantics
2. overhead is too concentrated in collector, the performance of high QPS scenarios is slightly poor
3. the dimension cache mechanism is easy to cause span loss, which ultimately leads to inaccurate metrics

[CoderPoet]

@futurist Very good question, here are some of my considerations

There are some problems with the community's spanmetrics processor:

1. The generated metrics cannot be used to draw topology, only convert metrics based on a single span, rather than correlating upstream and downstream spans resource.service_name semantics
2. overhead is too concentrated in collector, the performance of high QPS scenarios is slightly poor
3. the dimension cache mechanism is easy to cause span loss, which ultimately leads to inaccurate metrics

Of course, on the other hand, the metrics instrumentation on the SDK side can be considered to make a pluggable capability

[CoderPoet]

@futurist Very good question, here are some of my considerations

There are some problems with the community's spanmetrics processor:

1. The generated metrics cannot be used to draw topology, only convert metrics based on a single span, rather than correlating upstream and downstream spans resource.service_name semantics
2. overhead is too concentrated in collector, the performance of high QPS scenarios is slightly poor
3. the dimension cache mechanism is easy to cause span loss, which ultimately leads to inaccurate metrics

4. It also limits the possibility of SDK-side sampling, and full sampling must be required to accurately calculate the metrics