Skip to content

fix(prometheusreceiver): invalid metric name validation error in scrape start from target allocator #40803

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

krajorama
Copy link
Member

@krajorama krajorama commented Jun 18, 2025

Description

Fix invalid metric name validation error in scrape start from target allocator.

Prometheus made setting metric_name_validation_scheme, metric_name_escaping_scheme mandatory mandatory, use sane defaults ("legacy", "undescores").

Link to tracking issue

Fixes #40788

Testing

Added unit test for target allocator + scrape e2e.
The test fails if:

  • Prometheus logged something above the WARN level
  • Prometheus reports failed scrape pools in metrics
  • No metrics are in the result

Documentation

N/A

This test checks target allocator + scrape e2e.
The test fails if:
- Prometheus logged something above the WARN level
- Prometheus reports failed scrape pools in metrics
- No metrics are in the result

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
@github-actions github-actions bot added the receiver/prometheus Prometheus receiver label Jun 18, 2025
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Set sane defaults if target allocator didn't return value for
metric_name_validation_scheme
metric_name_escaping_scheme

The defaults are "legacy" and "underscores"

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
@krajorama krajorama changed the title test(prometheusreceiver): add unit test for target allocator fix(prometheusreceiver): invalid metric name validation error in scrape start from target allocator Jun 18, 2025
@krajorama krajorama marked this pull request as ready for review June 18, 2025 14:57
@krajorama krajorama requested review from dashpole, ArthurSens and a team as code owners June 18, 2025 14:57
dashpole
dashpole previously approved these changes Jun 18, 2025
Comment on lines 157 to 164
if scrapeConfig.MetricNameValidationScheme == "" {
scrapeConfig.MetricNameValidationScheme = promconfig.LegacyValidationConfig
}

if scrapeConfig.MetricNameEscapingScheme == "" {
scrapeConfig.MetricNameEscapingScheme = model.EscapeUnderscores
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, the default behavior is a little more complex: default validation scheme is actually UTF8. And then the escaping scheme is "allow-utf-8" if the validation scheme is UTF8, and is "underscores" if the validation scheme is "legacy"

https://github.com/prometheus/prometheus/blob/main/config/config.go#L874-L920

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reused the Validation code from Prometheus to set these instead, so that the code is not duplicated here.

@dashpole dashpole dismissed their stale review June 18, 2025 15:56

We should update the default configuration in prometheus instead

@krajorama krajorama marked this pull request as draft June 18, 2025 15:57
@krajorama krajorama marked this pull request as draft June 18, 2025 15:57
@krajorama
Copy link
Member Author

We've discussed this in the OTEL Prometheus WG meeting. The real problem is that Prometheus doesn't set the correct defaults itself. It's ok to make this workaround for now, linked to a new issue in Prometheus, however I need to update the test so that it's creating the base scrape configuration from Prometheus defaults and not manually as in my test code.

in the target allocator when we receive a scrape config. This makes sure
that globals are applied and defaults are set.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
@krajorama
Copy link
Member Author

I think I found a nice solution to not duplicate logic from Prometheus , however I don't have a testbed for the target allocator operator, so it's very hard for me to tell if I'm causing some other problem.

I imagine it's fair to validate the scrape config that the target allocator provides, but I'm not 100% sure. @dashpole @ywwg

@krajorama krajorama marked this pull request as ready for review June 19, 2025 08:21
krajorama and others added 2 commits June 20, 2025 16:37
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
receiver/prometheus Prometheus receiver
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[receiver/prometheus] Invalid metric name validation scheme error in 0.128.0
5 participants