E2E Integration with Prometheus

[pazbardanl]

Prize category

• Best Model
• Best Content
• Beyond Carbon
• Best contribution to the framework

User Story

• As an IF user, I want to to use IF to pull input metrics from and push carbon emission data into Prometheus so that I can view and analyze Carbon emissions as a standard metric

Rationale

Carbon emissions are a metric. It's data we produce using the Impact Framework. As such, we would want it to be stored and ready to use just like any other metric that is being monitored by Devops, Site Reliability Engineers an others.
On the same note: we should be able to fetch (import) metric data from the same storage to use as input to IF.

Impact

Looking forward to stuff being built on top of IF (Beta?), successfully implementing this idea will put it in a great starting point for being used to enhance observability.
A ready-to-use manifest file, backed by proper Prometheus importer and exported models, will:
(1) Encourage people to look at carbon emission as a metric to monitor, alert and make decisions based-of.
(2) Show how IF caters to the common use case of Observability.

Implementation guidelines

• models built for this task MUST conform to the ModelPlugin interface

Difficulty

4/5

Scope Of Work

• 2 new models: Prometheus importer and exporter

• A local instance of Prometheus with CPU data (could be random/fake static data)

• A manifest file where the pipeline starts with the Prometheus importer and closes with the Prometheus exporter

• IDEALLY a demo, visualizing inputs (like CPU util) and output (carbon) on the same dashboard, emphasizing that data for both was extracted from the same Prometheus instance

• A new model plugin

• A new feature in an existing model plugin

• A manifest file will have to be written

• Implement test cases

• A new feature to the framework itself (the cli tool)

• It will require changes to the documentation

Examples and resources

getting started with Prometheus

Manifest File

name: prometheus-e2e-demo
description:
tags:
  kind: web
  complexity: moderate
  category: on-premise
initialize:
  models:
    - name: prometheus-importer
      model: PrometheusImporter
      path: "@grnsft/if-unofficial-models"
   ...
   ...
   ...
    - name: prometheus-exporter
      model: PrometheusExporter
      path: "@grnsft/if-unofficial-models"
graph:
  children:
    child: # an advanced grouping node
      pipeline:
        - prometheus-importer
       ...
       ...
       ...
        - prometheus-exporter
      config:
        prometheus-importer:
          {prometheus connection info, data source / table from which to fetch input data}
       ...
       ...
       ...
        prometheus-exporter:
          {prometheus connection info, data source / table to which to push output data}
        
      children:
        child-1:
          inputs:

[jawache]

This is an excellent idea @pazbardanl!

I wonder if the Prometheus protocol is open, as in maybe Prometheus is so popular that other non-Prometheus databases will allow you to query using the Prometheus protocol. If so this would be even more useful.

The approach we took in the azure-importer required us to store certain sensitive data in env variables (for making the API calls etc...) I think we'd need something like that here also.

The approach we took in azure-importer was to put the query params as the first input in the child component rather than the config. Not sure if that was the right approach but the reasoning was then you could have a shared pipeline at a higher level node and then customise the query at the component level, like so:

graph:
  children:
    child: # an advanced grouping node
      pipeline:
        - prometheus-importer
       ...
       ...
       ...
        - prometheus-exporter
      config:
        prometheus-importer:
          {prometheus connection info}
       ...
       ...
       ...
        prometheus-exporter:
          {prometheus connection info}
      children:
        child-1:
          inputs: {data source / table from which to fetch input data}
        child-2:
          inputs: {data source / table from which to fetch input data}
        child-2:
          inputs: {data source / table from which to fetch input data}

[pazbardanl]

ok so as far as i could gather there is no way to directly query a different / non-Prometheus database using PromQL (Prometheus query language). There is a way to do that indirectly by "hooking" Prometheus to a DB and then reading/writing to that DB via Prometheus using PromQL.

What i can propose here, similar to our thread on Wrapper / Template for non-Typescript models
is to define the design as requirement for this hackathon idea: hackers would need to implement the exporter/importer with as much reusable code as possible, so that Prometheus-specific components (queries, connection handling etc) would be decoupled for the rest of the logic.

Also, I've filled in some of the blanks in the impl / manifest example you provided above.
Notice that writing into Prometheus is slightly more complicated than a standard DB - you need to define a Pushgateway to which your code writes the data to, and have your Prometheus configured in the background to scrape this Pushgateway as if it were a target (monitored) machine from which it scrapes metrics.
Reading/querying data from Prometheus is more straightforward.

graph:
  children:
    child: # an advanced grouping node
      pipeline:
        - prometheus-importer
       ...
       ...
       ...
        - prometheus-exporter
      config:
        prometheus-importer:
           targets: ['prometheus:9090']
       ...
       ...
       ...
        prometheus-exporter:
           - job_name: 'pushgateway_metrics'
             honor_labels: true
            static_configs:
             - targets: ['pushgateway:9091']
      children:
        child-1:
          inputs:
             metric: cpu_util
             instance: some-instance-1
             start: 2023-11-02T10:35:31
             end: 2023-11-02T10:42:01
        child-2:
          inputs:
             metric: cpu_util
             instance: some-instance-2
             start: 2023-11-02T10:00:31
             end: 2023-11-02T10:05:01

[jawache]

Ahh, that's interesting. Importers might be a lot easier than exporters. This makes sense and actually, importers would be just more useful as well (in general).

In the IF refactor, I'm proposing a special function called exhaust where you could write a more generalized Prometheus exporter. Esp. if you have to do something with a gateway. It will be run only once the whole graph has been computed so that might be an easier solution to exporting. You could maybe just launch the gateway, parse the output graph, extract metrics, and then push to gateway and then end process.

[pazbardanl]

yeah that makes sense - do it once for the entire load of data instead of paying the overhead several times.

So if the plan is to separate read (multiple times per manifest file execution) and write (once at exhaust) then E2E integration with Prometheus would be a bit too much.. so maybe separate it into 2 project ideas?