Improve auto-detection when multiple Prometheus instances are installed

[kurthaegeman]

What would you like to be added:
I would like for an improved or assisted way to be implemented for Lens to detect the correct Prometheus instance when multiple instances are deployed on the cluster. The current k get svc -A approach to auto-detection appears to pick the first Prometheus service it finds, but that might not be the main one. Maybe with the help of some labels or annotations, we can tag the correct service for Lens to find and avoid all developers having to hardcode the Prometheus path?

Why is this needed:
Applications like Kubecost come with their own Prometheus install, optimised for their purpose.

Kubecost strongly recommends installing Kubecost with the bundled Prometheus in most environments.

However, since its installation, it is the Kubecost Prometheus service that gets auto-detected by Lens and not the main install.

• Kubernetes distribution: RKE-deployed on-prem clusters, Kube v1.26.6
• Desktop OS: Windows 11 and Mac OS X Sonoma

[IanMoroney]

Adding to this,
prometheus installs managed / deployed by argocd also can't be detected, as they're not direct helm chart installations.
Being able to have an additional prometheus provider maybe for "other" or "custom install" and being able to specify the required details would be preferred.

[RobCannon]

I would like to know the criteria Lens uses for auto-detecting so we can make sure our Prometheus endpoint (a proxy to Grafana Cloud) is auto-detected.