feature: Add support to scrape /metrics endpoints in both local and remote clusters #70
Description
What would you like to be added:
Currently the metrics operator is more like a metrics-per-krm (k8s resource model) operator. To add support for standard metrics we can try to scrape /metrics endpoints that resources may have.
This should be too hard to add and since we are sending metrics via OTEL this should be easy to transport them then.
Why is this needed:
It is a common enough use case. While we initially intended this as 'business or insight' metrics operator, there are indeed some metrics in a metrics endpoint that can also provide insights apart from telemetry.
If the user wants to use that for additional data to make other analysis then they can.
Proposal
Potentially add it as a separate controller that scrapes the /metrics endpoints, then sends the data as needed via OTEL (existing logic). That way we can control it behind a feature flag.
Long term this should be a separate scraper with cache which the controller reads from. The scraper itself uses access cluster and access remote cluster types to get the data that is ready to be sent via OTEL when the next reconcile from the operator executes.