[otlp] Fix panic in dropped count (again!) #3538
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gouthamve
force-pushed
the
fix-otel-panics-once-and-forall
branch
from
f3c0849
to
f2fe3f4
Compare
gouthamve
force-pushed
the
fix-otel-panics-once-and-forall
branch
from
f2fe3f4
to
46f0333
Compare
This doesn't accurately count the dropped samples. For example if a single metric with multiple samples is faulty, we get a single error rather than an error per sample. But I believe its the best best-effort measurement. Before we used to do `DatapointCount() - samplesInMap()` The problem is the following: 1. target_info is a synthetic metric added in Prometheus, so the final samples could higher. 2. A single histogram datapoint in OTLP corresponds to many samples in Prometheus. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
gouthamve
force-pushed
the
fix-otel-panics-once-and-forall
branch
from
46f0333
to
03cbd10
Compare
pstibrany
reviewed
Nov 28, 2022
pstibrany
approved these changes
Nov 28, 2022
There was a problem hiding this comment.
Code change looks good to me.
It's little strange to me that tenant check is only done in case of errors. Should we do it before we even try to convert metrics?
Added unit tests don't seem to be checking for dropped metric at all. In other words, they are unrelated to the PR. 🤔
It is on an authenticated route so the check essentially NEVER fails. It only exists to load the
I first wrote the unit tests to reproduce the panic and then proceeded to fix it. Which is why it seems like the unit test is unrelated. |
replay
pushed a commit
that referenced
this pull request
Nov 28, 2022
This doesn't accurately count the dropped samples. For example if a single metric with multiple samples is faulty, we get a single error rather than an error per sample. But I believe its the best best-effort measurement. Before we used to do `DatapointCount() - samplesInMap()` The problem is the following: 1. target_info is a synthetic metric added in Prometheus, so the final samples could higher. 2. A single histogram datapoint in OTLP corresponds to many samples in Prometheus. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
grafanabot
pushed a commit
that referenced
this pull request
Nov 29, 2022
This doesn't accurately count the dropped samples. For example if a single metric with multiple samples is faulty, we get a single error rather than an error per sample. But I believe its the best best-effort measurement. Before we used to do `DatapointCount() - samplesInMap()` The problem is the following: 1. target_info is a synthetic metric added in Prometheus, so the final samples could higher. 2. A single histogram datapoint in OTLP corresponds to many samples in Prometheus. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> (cherry picked from commit 2ad7eb4)
replay
pushed a commit
that referenced
this pull request
Nov 29, 2022
This doesn't accurately count the dropped samples. For example if a single metric with multiple samples is faulty, we get a single error rather than an error per sample. But I believe its the best best-effort measurement. Before we used to do `DatapointCount() - samplesInMap()` The problem is the following: 1. target_info is a synthetic metric added in Prometheus, so the final samples could higher. 2. A single histogram datapoint in OTLP corresponds to many samples in Prometheus. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> (cherry picked from commit 2ad7eb4) Co-authored-by: Goutham Veeramachaneni <gouthamve@gmail.com>
stevesg
pushed a commit
that referenced
this pull request
Nov 30, 2022
This doesn't accurately count the dropped samples. For example if a single metric with multiple samples is faulty, we get a single error rather than an error per sample. But I believe its the best best-effort measurement. Before we used to do `DatapointCount() - samplesInMap()` The problem is the following: 1. target_info is a synthetic metric added in Prometheus, so the final samples could higher. 2. A single histogram datapoint in OTLP corresponds to many samples in Prometheus. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> (cherry picked from commit 2ad7eb4)
masonmei
pushed a commit
to udmire/mimir
that referenced
this pull request
Dec 16, 2022
This doesn't accurately count the dropped samples. For example if a single metric with multiple samples is faulty, we get a single error rather than an error per sample. But I believe its the best best-effort measurement. Before we used to do `DatapointCount() - samplesInMap()` The problem is the following: 1. target_info is a synthetic metric added in Prometheus, so the final samples could higher. 2. A single histogram datapoint in OTLP corresponds to many samples in Prometheus. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
masonmei
pushed a commit
to udmire/mimir
that referenced
this pull request
Dec 16, 2022
This reverts commit 3744cb5.
masonmei
pushed a commit
to udmire/mimir
that referenced
this pull request
Dec 16, 2022
This doesn't accurately count the dropped samples. For example if a single metric with multiple samples is faulty, we get a single error rather than an error per sample. But I believe its the best best-effort measurement. Before we used to do `DatapointCount() - samplesInMap()` The problem is the following: 1. target_info is a synthetic metric added in Prometheus, so the final samples could higher. 2. A single histogram datapoint in OTLP corresponds to many samples in Prometheus. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> (cherry picked from commit 2ad7eb4) Co-authored-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This doesn't accurately count the dropped samples. For example if a single metric with multiple samples is faulty, we get a single error rather than an error per sample.
But I believe its the best best-effort measurement.
Before we used to do
DatapointCount() - samplesInMap()The problem is the following:
Checklist
Documentation addedCHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]