Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compact: relax sorted label names check under accept-malformed-index flag #888

Closed
bwplotka opened this issue Mar 5, 2019 · 3 comments
Closed

Comments

@bwplotka
Copy link
Member

bwplotka commented Mar 5, 2019

Also in "relax" mode, we should produce metrics.

Relevant discussion: #848 (comment)

Acceptance Criteria:

  • Add debug.accept-malformed-index flag.
  • When this flag is true, relax the compaction index verification to not check if label names are sorted and produce metrics when this issue occurs so that it'd be known.
@GiedriusS
Copy link
Member

Fixed an issue upstream which makes Prometheus produce invalid metrics with upper-case letters in label names: prometheus/prometheus#5372
The debug.accept-malformed-index flag should make Thanos Compact ignore the fact that the invariant is not being upheld i.e. it should turn that check off mentioned in #848.

@GiedriusS GiedriusS changed the title compact: Relax some index check details under accept-malformed-index flag? compact: relax sorted label names check under accept-malformed-index flag Mar 17, 2019
@GiedriusS
Copy link
Member

Also, perhaps this should be called --ignore-nonsorted-labels.

@GiedriusS
Copy link
Member

GiedriusS commented Mar 25, 2019

A hidden --debug.accept-malformed-index flag has been added to Thanos Compactor recently, courtesy of @jjneely. Thanks a lot! So now this has been fixed in Prometheus >2.8.0, and if users are running older Prometheus versions they can enable this (which is mentioned in the error message) to ignore this error since it is inevitable if any of the metrics have upper-case label names.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants