[BUG] prometheus datasource mongodb and other exporter metrics are missing

[rmarops]

Describe the bug
Unable to create visualizations based on mongodb* metrics

To Reproduce
Steps to reproduce the behavior:

1. add prometheus datasource as per https://opensearch.org/docs/latest/observing-your-data/prometheusmetrics/
2. navigate to Observability > Metrics analytics > Available Metrics
3. note that there are several prometheus*, go* metrics available
4. search for mongodb but no metrics appear

Expected behavior
mongodb metrics are available

Plugins
opensearch-master opensearch-alerting 2.10.0.0
opensearch-master opensearch-anomaly-detection 2.10.0.0
opensearch-master opensearch-asynchronous-search 2.10.0.0
opensearch-master opensearch-cross-cluster-replication 2.10.0.0
opensearch-master opensearch-custom-codecs 2.10.0.0
opensearch-master opensearch-geospatial 2.10.0.0
opensearch-master opensearch-index-management 2.10.0.0
opensearch-master opensearch-job-scheduler 2.10.0.0
opensearch-master opensearch-knn 2.10.0.0
opensearch-master opensearch-ml 2.10.0.0
opensearch-master opensearch-neural-search 2.10.0.0
opensearch-master opensearch-notifications 2.10.0.0
opensearch-master opensearch-notifications-core 2.10.0.0
opensearch-master opensearch-observability 2.10.0.0
opensearch-master opensearch-performance-analyzer 2.10.0.0
opensearch-master opensearch-reports-scheduler 2.10.0.0
opensearch-master opensearch-security 2.10.0.0
opensearch-master opensearch-security-analytics 2.10.0.0
opensearch-master opensearch-sql 2.10.0.0

Screenshots

Host/Environment (please complete the following information):
Amazon Linux 2

Additional context
running PPL query from the query workbench works fine
example: source=prometheus.mongodb_up

[rmarops]

adding the screenshot of query workbench working with mongodb_up metric

[rmarops]

just confirmed that all node_exporter (node*) metrics are not available as well

[rmarops]

I believe the question and answer here is related to the issue https://stackoverflow.com/questions/68081062/prometheus-not-expose-all-collected-metrics

Excerpt screenshots

I can confirm the metrics URL only lists prometheus* and go* metrics whereas using the prometheus query UI or any other query tool against the prometheus service exposes all available metrics as expected.

[rmarops]

This looks to return all metrics http://localhost:9090/api/v1/label/__name__/values
mentioned here: https://stackoverflow.com/questions/58319911/prometheus-endpoint-of-all-available-metrics

[rmarops]

metric list implementation looks to be here

https://github.com/opensearch-project/sql/blob/main/prometheus/src/main/java/org/opensearch/sql/prometheus/client/PrometheusClientImpl.java

  @Override
  public Map<String, List<MetricMetadata>> getAllMetrics() throws IOException {
    String queryUrl = String.format("%s/api/v1/metadata", uri.toString().replaceAll("/$", ""));
    logger.debug("queryUrl: " + queryUrl);
    Request request = new Request.Builder().url(queryUrl).build();
    Response response = this.okHttpClient.newCall(request).execute();
    JSONObject jsonObject = readResponse(response);
    TypeReference<HashMap<String, List<MetricMetadata>>> typeRef = new TypeReference<>() {};
    return new ObjectMapper().readValue(jsonObject.getJSONObject("data").toString(), typeRef);
  }