Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

store: out-of-order label set #1255

Closed
miekg opened this issue Jun 14, 2019 · 5 comments
Closed

store: out-of-order label set #1255

miekg opened this issue Jun 14, 2019 · 5 comments

Comments

@miekg
Copy link
Contributor

miekg commented Jun 14, 2019

We've ended up with data that can't be verified nor fixed in our data store. I'm currently in the process of delving deeper in this problem with a more up to date Thanos (v0.5). The problem happened while we where running 0.4.

The symptom is:

01D9S32REZD25B65AN54FVX44F: out-of-order label set
{
  <omitted json - available on request>
} for series 813726

We think it may have to do with receiving a metrics that had an_identical_ label with a value only differing in case.

We still have a copy of the data, and, as said, I'm in the process of narrowing down what actually happened.

This looks like #711, which is fixed, so I wonder why we've hit it.

@GiedriusS
Copy link
Member

Are you running Prometheus which has this fix prometheus/prometheus#5372 included?

@miekg
Copy link
Contributor Author

miekg commented Jun 14, 2019

No, this was def. on an older version. From the looks of it, this is included in 2.9.x ?

With that fix in Prometheus, does there still need to be something in Thanos? I.e. I want to re-inject the metrics we have into our metrics store. Is that possible?

@bwplotka
Copy link
Member

Well new Thanos gives ability to ignore this error, so it just warns. It is slight inefficiency, but that's it. Are you blocked with this?

Also it looks like compactor warns / errors out if debug.accept-malformed-index is not set to true, not store right?

@miekg
Copy link
Contributor Author

miekg commented Jun 21, 2019

Thanks. No not blocked. Good to know we can ignore the error, which gives us more defence in depth

Think this can be closed with WAI

@GiedriusS
Copy link
Member

@miekg closing as per your request but what's the meaning of WAl in this context? 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants