Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problems with prometheus on OTel 1.8 #5613

Closed
dominikjeske opened this issue May 14, 2024 · 3 comments
Closed

Problems with prometheus on OTel 1.8 #5613

dominikjeske opened this issue May 14, 2024 · 3 comments
Labels
bug Something isn't working pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package

Comments

@dominikjeske
Copy link

dominikjeske commented May 14, 2024

Area

area:exporter:prometheus

Package Version

Package Name Version
OpenTelemetry.Api 1.8.0
OpenTelemetry 1.8.0
OpenTelemetry.Exporter.OpenTelemetryProtocol 1.8.0
OpenTelemetry.Exporter.Prometheus.AspNetCore .8.0-rc.1
OpenTelemetry.Exporter.Prometheus.HttpListener 1.8.0-rc.1
OpenTelemetry 1.8.0

Runtime Version

net8.0

Description

After update to new Otel 1.8 we are expirience issues in our prometheus. In one version it says "Error on ingesting samples that are too old or are too far into the future" and after cross check with new version of prometheus it says "Error on ingesting samples with different value but same timestamp". Additionally we observe that on some random queries to /metrics (we are getting otel_scope_info that is missing timestamp like described here prometheus/prometheus#6554). Like below

image

If I'm getting this right this info should be returned only when "Accept: application/openmetrics-text" is send but maybe I'm wrong. Additionally this info is random - one query return metrics with this info and another without.

If this behavior is by design please share some documentation.

ps. After downgrade to 1.7 we don't have those errors

Steps to Reproduce

Just use standard OTel and serve prometheus endpoint

Expected Result

Prometheus agent is grabbing data with success

Actual Result

Errors from description

Additional Context

No response

@dominikjeske dominikjeske added the bug Something isn't working label May 14, 2024
@reyang reyang added pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package and removed area:exporter labels May 14, 2024
@WalissonPires
Copy link

Today I had the same problem after upgrading from version 1.6.0 to 1.8.1. (.NET 8)
I had to use version 1.7.0 as indicated by dominikjeske (thrx).

@reyang
Copy link
Member

reyang commented May 21, 2024

Duplicated with #5515. The bug was introduced by #5107 and fixed by #5623.

@reyang reyang closed this as not planned Won't fix, can't repro, duplicate, stale May 21, 2024
@dominikjeske
Copy link
Author

dominikjeske commented Jun 26, 2024

@reyang Can we reopen this issue? Today I tried to use OTel 1.9 but still I have problems with metrics. Prometheus logs:

prometheus[603702]: ts=2024-06-26T13:18:31.847Z caller=scrape.go:1607 level=warn component="scrape manager" scrape_pool=webapi target=
http://xxxxxxxxx:8080/metrics
msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=1

When rollback to 1.7 everything looks good. Is this a bug or we have to use new prometheus to handle metrics from 1.8+?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package
Projects
None yet
Development

No branches or pull requests

3 participants