Reading metrics out of Registry as structured data

[krisztianfekete]

Hey everyone, I wanted to discuss this gap before writing any code, because it would be impossible to get concensus on a PR at this point.

Today, the only way to get data out of a Registry is to encode it: either via encoding::text::encode / encoding::protobuf::encode, or by implementing Collector.

DescriptorEncoder and MetricEncoder are concrete types, so I can't "bring my own encoder" to achieve what I want. AFAIKT, there's no way to "get me a structured view of every registered metric" in a clean way.

Why I'm bringing this up: I'm considering building a Prometheus -> OpenTelemetry metrics bridge in Rust, equivalent to what is already available in opentelemetry-go-contrib. That bridge works because client_golang exposes prometheus.Gatherer with a structured snapshot of the registry. The bridge calls Gather() on every collection cycle and translates the families into OTel data points. In Rust the equivalent would need to re-parse protobuf bytes on every scrape, which doesn't seem like a good idea.

This also seems to be relevant relevant for #131, and maybe other use-cases as well that could benefit from the same hook.

What do you all think about:

1. A gather()-like method on Registry that returns structured metric data (basically Gather() in client_golang, or even
   Registry::gather() in tikv/rust-prometheus). This seems quite simple, but introduces a new public data model that has to stay in sync with the protobuf schema (especially once native histograms land see protobuf: move from OpenMetrics to Prometheus proto #298).
2. Let users implement DescriptorEncoder / MetricEncoder themselves so consumers can write their own "capture" encoder. This adds no new data type and the protobuf types stay an implementation detail, but feels weirder to use.

I don't have a strong opinion and would happily defer to whatever the maintainers think fits the existing design better. There might be an even better way other than these two. What do you think?

cc @mxinden @brancz, please add other maintainers if you can, thank you!