Automate the Collector list of internal metrics using mdatagen docs

[tiffany76]

Desired feature or idea: The Collector's internal metrics were recently added to the metadata generating tool mdatagen in open-telemetry/opentelemetry-collector#10170. Basic documentation is now automatically generated in repo markdown files for each component, for example:

• Batch processor metrics
• Exporter metrics

A manually compiled list of metrics is included on the internal telemetry page of the website, but concerns have been raised about maintaining it.

Can we incorporate the mdatagen automated docs into the OTel website?

Additional context:

• Note that the metadata-generated docs do not list the metric level, which was the primary reason we added the list to the website docs. Could mdatagen's schema be modified to include an optional field for level?
• This issue may relate to [Proposal] Automate reference documentation as YAML files community#1610 and other prior discussions on the topic of automation that I am not currently aware of.

[svrnm]

Thanks for raising this issue.

We can (and should) incorporate that data into the otel website. Ideally this data flows into the registry and then we reuse it from there in the documentation.

To do so we need to collaborate with @open-telemetry/collector-approvers / @open-telemetry/collector-maintainers to automate that (and maybe enrich the metadata file even further)

[theletterf]

We leverage mdatagen files in the Splunk distro's docs using client side JS. For example, this table...

https://docs.splunk.com/observability/en/gdi/opentelemetry/components/kubelet-stats-receiver.html#metrics

Is built using this file...

https://raw.githubusercontent.com/splunk/collector-config-tools/main/metric-metadata/kubeletstatsreceiver.yaml

...through this JS logic:

https://github.com/splunk/public-o11y-docs/blob/main/_static/jsonpull-splunk.js