[Input Package] Create new Prometheus Input package

[gizas]

Input package specification is defined here

The new input package tries to address flexible implementation for custom Ingest configurations for our customers based on preexisting modules . Basic advantages until now is the a) specification of custom dataset per input b) custom mappings

Continuing discussion of #4271, Cloudnative plans to create a new Prometheus Input package based on the existing Prometheus Collector Module.

Goal:

• Understand inner details of the new Input Specification
• Define custom mappings in the new input package
• Define custom dataset name per input and verify functionality
• Define custom ingest pipelines and verify functionality
• Promote beta version to production

[gizas]

@tetianakravchenko #4517

[StephanErb]

Has there been a discussion on how the Prometheus input can leverage TSDB?

We would need to do something like this below, but I am not sure how the dynamic and unbounded number of labels will play together with index.mapping.dimension_fields.limit

- name: prometheus
  type: group
  fields:
    - name: labels.*
      type: object
      object_type: keyword
      dimension: true 

I am curious what your thoughts are on this.

[ishleenk17]

The development of the Prometheus Input package is ongoing. We are using the collector metricsets as input for retrieving the metrics.
Testing it out with Influxdb as service.