[prometheus][remote_write] Fix remote_write ingest pipeline to include both metric path 'prometheus.*' and 'prometheus.metrics.*' to fingerprint calculation

[tetianakravchenko]

What does this PR do?

Fix remote_write ingest pipeline to include both metric path 'prometheus.' and 'prometheus.metrics.' to fingerprint calculation

There are 2 use cases:

1. If Use Types is enabled - metrics will be added under prometheus.* in this format:

{ "prometheus":
  "metric1": {
    "counter": x
  },
  "metric2": {
    "value": y
  }
  "labels": { 
    ...
  }
}

2. If Use Types is disabled - metrics will be added under prometheus.metrics.* in this format:

{ "prometheus":
  "metrics": {
    "metric1": x
    "metric2": y
  }
  "labels": { 
    ...
  }
}

This fix is needed to support both cases, otherwise when TSDB is enabled, some documents will be dropped in case Use Types is disabled

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Screenshots

Previous version - 1.12.0

For demonstration purpose I've remove remove processor in the local setup -

integrations/packages/prometheus/data_stream/remote_write/elasticsearch/ingest_pipeline/default.yml

Lines 29 to 31 in ce0f9ec

	- remove:
	field: "prometheus.labels.metrics_names"
	description: Remove field as it contains a redundant information and can impact the documents size

Use_type disabled:

In this case fingerprint based on the [metric, labels] object is not unique that can cause some dropping of documents.

Use_type enabled:

Upgrade prometheus version to 1.12.1

Upgrade caused recreation of the ingest pipeline, I've removed `remove` processor.

Use_type enabled:

Use_type disabled (notice that metrics are stored under prometheus.metric):

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-09-20T09:17:41.088+0000

• Duration: 15 min 36 sec

Test stats 🧪

Test	Results
Failed	0
Passed	9
Skipped	0
Total	9

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (0/0)	💚
Files	100.0% (0/0)	💚
Classes	100.0% (0/0)	💚
Methods	66.667% (8/12)	❕
Lines	100.0% (0/0)	💚
Conditionals	100.0% (0/0)	💚

[ChrsMark]

lgtm. Would be nice if we could cover these through "unit" testing, not sure if we have already or if it's doable though.

[gizas]

I am missing only some evidence for tests that the fix worked. Some screenshots only would have been ok

[tetianakravchenko]

I am missing only some evidence for tests that the fix worked. Some screenshots only would have been ok

@gizas I've updated the PR description, could you please have a look?

[elasticmachine]

Package prometheus - 1.12.1 containing this change is available at https://epr.elastic.co/search?package=prometheus