compact: relax index label checking #848
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It is possible that Prometheus will produce time series with duplicate
labels. E.g.: a{b="2",b="2"} 1
Where such metric is pushed to Prometheus either wittingly or
unwittingly, the compact process will stall because the check will not
pass. For example:
{"caller":"main.go:181","err":"error executing compaction: compaction failed: compaction: gather index issues for block /data/compact/0@{monitor="monitor",replica="repl"}/01D34EDQMSQ29RHAC47XGKHGC7: out-of-order label set {**name**="foo",exported_job="vv",host="172_16_226_56",host="172_16_226_56",region="lt",subtask_index="5",task_attempt_id="32e4b047bb768583ff57c709be3b1046",task_attempt_num="8",task_id="688c028a219ff3372f3eecb0ee5811f9",task_name="Source:_foo",tenant="abc",tier="cooltier",tm_id="53b2ed987b08f427dec4ee1465df91fa"} for series 2594231","level":"error","msg":"running command failed","ts":"2019-02-11T13:30:33.901722306Z"}
This commit fixes the mentioned issue. Amend the e2e test to test the
case when a series has identical labels.
GiedriusS
force-pushed
the
relax_compactor
branch
from
d98041e
to
bccfdcc
Compare
This was referenced Feb 15, 2019
FUSAKLA
approved these changes
Feb 17, 2019
ykorikk
approved these changes
Feb 26, 2019
bwplotka
approved these changes
Mar 4, 2019
| @@ -347,7 +347,7 @@ func GatherIndexIssueStats(logger log.Logger, fn string, minTime int64, maxTime | |||
| } | |||
| l0 := lset[0] | |||
| for _, l := range lset[1:] { | |||
| if l.Name <= l0.Name { | |||
| if l.Name < l0.Name { | |||
There was a problem hiding this comment.
You mean warn the user if they are equal? What message do you think would fit here?
There was a problem hiding this comment.
Yea, for example "Invalid labelset found; multiple labels with the same name for the same series found"
Wonder how is that even possible? (: I would go even further and put the whole verification step under some flag accept-malformed-index, but not sure. (: If any, we can do it in later PRs.
There was a problem hiding this comment.
Will merge this, let's think about this later on.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It is possible that Prometheus will produce time series with duplicate
labels. E.g.: a{b="2",b="2"} 1
Where such metric is pushed to Prometheus either wittingly or
unwittingly, the compact process will stall because the check will not
pass. For example:
{"caller":"main.go:181","err":"error executing compaction: compaction failed: compaction: gather index issues for block /data/compact/0@{monitor="monitor",replica="repl"}/01D34EDQMSQ29RHAC47XGKHGC7: out-of-order label set {name="foo",exported_job="vv",host="172_16_226_56",host="172_16_226_56",region="lt",subtask_index="5",task_attempt_id="32e4b047bb768583ff57c709be3b1046",task_attempt_num="8",task_id="688c028a219ff3372f3eecb0ee5811f9",task_name="Source:_foo",tenant="abc",tier="cooltier",tm_id="53b2ed987b08f427dec4ee1465df91fa"} for series 2594231","level":"error","msg":"running command failed","ts":"2019-02-11T13:30:33.901722306Z"}
This commit fixes the mentioned issue. Amend the e2e test to test the
case when a series has identical labels.
Brings us closer to accepting data that Prometheus can produce. Maybe we are still missing something as evident by other issues?! :)
Changes
Relaxed the labels check in the GatherIndexIssueStats function so that it would be permitted to have identical labels.
Verification
Tests pass.