Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow empty matcher at store gateway to fetch all postings #7260

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

yeya24
Copy link
Contributor

@yeya24 yeya24 commented Apr 5, 2024

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

If a query only contains external labels, no matcher is left after matching external labels at store gateway and in this case store gateway fetches nothing. https://github.com/thanos-io/thanos/blob/main/pkg/store/bucket.go#L2471

This PR changes Store Gateway to add a matcher ""="" to fetch all postings instead. This should match Prometheus behavior better.

This changes existing behavior. Current behavior all Stores will return nil. After that, Store Gateway will try to fetch all postings (series). This change is more compatible to Prometheus from what I understand. For example, cluster is just a normal label in Prometheus and users can do count ({cluster="A"}) without issues. However, after they make cluster an external label, then there is no way to run count ({cluster="A"}) anymore.

If there is concern about the behavior change, I can put this behavior under a FF and make it hidden to Thanos users.

Verification

Updated E2E test case.

Signed-off-by: Ben Ye <benye@amazon.com>
@yeya24 yeya24 force-pushed the allow-empty-matchers-after-external-label branch from a115092 to b3fc96d Compare April 5, 2024 05:30
Signed-off-by: Ben Ye <benye@amazon.com>
Copy link
Member

@GiedriusS GiedriusS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have this check right now to forbid this kind of situations: https://github.com/search?q=repo%3Athanos-io%2Fthanos+%22no+matchers+specified%22&type=code. What's the relationship between those checks and your change? Is this something you need in Cortex?

@yeya24
Copy link
Contributor Author

yeya24 commented Apr 10, 2024

@GiedriusS Yes, we are evaluating external label based sharding in Cortex where blocks can have external labels. Similar to #7256.
Without this PR, a query with a single external label matcher only will break and returns no data. Imagine the migration from no external label blocks to external label blocks.

I understand that this changes the behavior of Store Gateway. Does it sound ok if we put this under a FF?

@MichaHoffmann
Copy link
Contributor

Could you add an acceptance test please?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants