Store: store responses should always be sorted #6706

MichaHoffmann · 2023-09-06T15:09:11Z

I added CHANGELOG entry for this change.
Change is not relevant to the end user.

Changes

always resort store responses
compare labelsets without dropping replica labels in proxy heap

Verification

Not verified yet

vCra · 2023-09-06T18:07:38Z

Tested this out in our test environment:

After deployment number of 422s and 503s disappear completely

And we no longer see any of the Error executing query: vector cannot contain metrics with the same labelset issues seen in #6677

tekicode · 2023-09-06T22:58:00Z

Thanks for your effort on this PR @MichaHoffmann I'm running this in our testing environment as well and the labelset error I reported in #6677 is resolved

saswatamcode

LGTM, seems like this also preserves the optimization

@fpetkovski @GiedriusS maybe we can merge and cut a patch?

pkg/store/bucket.go

GiedriusS · 2023-09-07T06:24:44Z

Thanks for working on this but I think that this is a wrong approach to take because it just papers over the problem namely that StoreAPIs don't uphold the invariant of sending a sorted response set. I think that for project's long term health we should focus our efforts toward fixing that.

I think this is related: prometheus/prometheus#12605. This is because we actually implement a very similar logic regards external labels.

Essentially, I think that external labels functionality breaks this invariant when the internal TSDB has overlapping labels with different values. Everywhere we use this logic:

Select() or, in other words, get a sorted series set
Overwrite labels with the external labels

But if the 1st operation has a mix of time series with and without labels that are part of external labels, problems might ensue. Because time series with less labels come before those with more labels, it means that the end result might be unsorted if the first series in the result do not have external labels in the TSDB and the latter ones do.

I'm a bit busy but I could try writing a test for this case in all components to show what I'm talking about. I'll try to do that over the next few working days.

Also, I think it's correct to only compare raw labels and the previous change regarding comparing external labels first tries to workaround the same problem that I mentioned here

MichaHoffmann · 2023-09-07T06:30:43Z

Just to state: I didnt intend this as the solution to the problem; more as a short term ad-hoc fix so we can uphold the invariant and solve the problem properly in subsequent PRs.

GiedriusS · 2023-09-07T06:57:50Z

It's all good, no worries. I was thinking about this and perhaps this problem could be temporarily fixed by forcing resorting if there's an intersection between the external labelset and the label names filter? This would allow us to check if such a possibility is even possible before starting to read the series.

MichaHoffmann · 2023-09-07T07:00:46Z

It's all good, no worries. I was thinking about this and perhaps this problem could be temporarily fixed by forcing resorting if there's an intersection between the external labelset and the label names filter? This would allow us to check if such a possibility is even possible before starting to read the series.

If we get respSet A and B we would need to check if labelnames of store B intersect extlabels of A and vice versa; in which case we would need to resort i think. But that seems way more complicated then forcing sort on store APIs for first iteration

douglascamata · 2023-09-07T09:20:50Z

Could we commit to merging this as a hotfix and work on improvements or a "proper fix" in follow ups? This dedup issue is plaguing Thanos releases since 6 months, blocking many people out there from upgrading to take advantage of new features because of uncertainty to whether there will be issues or not with queries. Nobody will care if the memory improvements introduced by the big query sorting change are completely negated for them to be able to always have correct query results and the ability to upgrade their Thanos. Query dedup logic is clearly broken and it has to be fixed. Query accuracy is a pilar of any observability software.

@MichaHoffmann is putting a lot of hard effort and time into this working on fixes, automated tests, testing things in general, etc. I'm sure he has many plans or ideas to improve this further down the line. Others could and are very welcome to help too.

But let's get this working first, shall we? We gotta move forward and, most importantly, move over this bug.

pkg/store/flushable.go

GiedriusS · 2023-09-08T10:51:59Z

We've moved this discussion a bit into #prometheus-dev. It seems like in most cases it is not safe to extend a sorted series set with external labels because it might lead an unsorted series set. Maybe someone will have some ideas there. In general it looks like external labels is a broken functionality. Anyway... if nothing comes up, we will probably be forced to revert back buffering and resorting in the querier :'/

fpetkovski · 2023-09-08T11:46:38Z

Isn't it better to resort in the store instead of centralizing a compute intensive operation in a single component?

MichaHoffmann · 2023-09-08T11:47:51Z

Isn't it better to resort in the store instead of centralizing a compute intensive operation in a single component?

The worry is that sidecar gets expensive i think. We could introduce a slight expansion of the protocol where we generally assume that store responses are sorted except if we get a hint that its not ( and sidecar would send that hint maybe )

fpetkovski · 2023-09-08T12:11:03Z

It would be great if we can leave this out to the user to decide. Our performance will drop significantly if we had to sort in the querier again.

MichaHoffmann · 2023-09-09T17:16:37Z

It would be great if we can leave this out to the user to decide. Our performance will drop significantly if we had to sort in the querier again.

I added a sortingStrategy thats currently hard coded in the different stores. I would like to move making it configurable ( to opt-out or to sort in the querier if wanted ) into a followup PR though since strictly speaking its an optimziation and this is already getting kinda big

GiedriusS · 2023-09-10T09:41:39Z

Adding test on Prometheus side to show the issue: prometheus/prometheus#12822.

fpetkovski · 2023-09-12T10:44:58Z

pkg/store/flushable.go

+
+const (
+	sortingStrategyStore sortingStrategy = iota + 1
+	sortingStrategyNone


Do we want a sortingStrategyQuery?

yeah i wanted to add it in followup pr

fpetkovski · 2023-09-12T10:45:49Z

pkg/store/proxy_heap.go

-		}
+		// NOTE. Client is not guaranteed to give a sorted response when extLset is added
+		// Generally we need to resort here.
+		sortWithoutLabels(l.bufferedResponses, l.removeLabels)


Hm, does it mean we will resort again in the querier? Maybe we want the strategy to be set in the querier and propagated to the store. This way we can decide in both places whether to resort or not.

Querier should use lazyresponse set always right?

As I understand it, if sorting strategy querier, then we need to use eager in the querier and stream from the store. Not sure if that gets added as a follow up though.

Ah yeah but if we use querier sorting strategy we should resort here i think; otherwise we could use sorting strategy none right?

Yes, so:

sorting strategy querier = eager in querier, lazy in stores.

sorting strategy store = eager in store, lazy in querier.

yeah precisely ( but i would like to add querier later since its more complicated change )

Agreed, we will need this but maybe for a separate PR.

MichaHoffmann · 2023-09-14T12:23:02Z

@GiedriusS is this PR acceptable?

GiedriusS

Let's also remove the paragraph https://thanos.io/tip/components/query.md/#deduplication-on-non-external-labels from here regarding cuckoo filter

GiedriusS · 2023-09-14T13:35:12Z

pkg/store/bucket.go

@@ -1334,7 +1329,8 @@ func debugFoundBlockSetOverview(logger log.Logger, mint, maxt, maxResolutionMill

 // Series implements the storepb.StoreServer interface.
 func (s *BucketStore) Series(req *storepb.SeriesRequest, seriesSrv storepb.Store_SeriesServer) (err error) {
-	srv := newFlushableServer(seriesSrv, s.LabelNamesSet(), req.WithoutReplicaLabels)
+	srv := newFlushableServer(seriesSrv, sortingStrategyNone)


Why doesn't this need resorting? 🤔

block responses are sorted because of eager retrieval there; basically we put sorted stuff in dedup proxy heap and get sorted series set back out and dont need to sort it again before returning it

all the dropping and extlabel juggling is done on block level already so after we joined it we can just pass it along

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

This is the wrong approach to detect if we need to resort. It cannot detect if we might end up with an unsorted series set if we add extLabels. Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

fpetkovski

Given all the edge cases we have ran into so far, I don't see other options than to eventually resort the dataset.

saswatamcode

LGTM!

* Store: always sort, just compare labelset in proxy heap Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Store: add escape hatch to skip store resorting Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Store: remove stringset This is the wrong approach to detect if we need to resort. It cannot detect if we might end up with an unsorted series set if we add extLabels. Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Docs: drop paragraph about deduplication on inner labels Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> --------- Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> Co-authored-by: Michael Hoffmann <michael.hoffmann@aiven.io>

* Update thanos engine and Prometheus dependencies (#6664) * Update thanos engine and Prometheus dependencies This commit bumps thanos/promql-engine to latest main and resolves breaking changes from the prometheus/prometheus dependency. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add changelog entry Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Avoid closing head more than once Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Remove call to t.TempDir() Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * build(deps): bump github.com/prometheus/alertmanager (#6671) Bumps [github.com/prometheus/alertmanager](https://github.com/prometheus/alertmanager) from 0.25.0 to 0.25.1. - [Release notes](https://github.com/prometheus/alertmanager/releases) - [Changelog](https://github.com/prometheus/alertmanager/blob/v0.25.1/CHANGELOG.md) - [Commits](prometheus/alertmanager@v0.25.0...v0.25.1) --- updated-dependencies: - dependency-name: github.com/prometheus/alertmanager dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * store: add acceptance tests for label methods to bucket store (#6668) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Ruler: Add update label names routine for stateful ruler (#6689) Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Store: add some acceptance tests for label matching (#6691) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Store: fix regex matching with set that matches empty (#6692) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Store: add failing test for potential dedup issue (#6693) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Store: fix block dedup (#6697) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Add Shipper bytes uploaded metric #6438 (#6544) * [FEAT] Add uploaded bytes metric Signed-off-by: rita.canavarro <rita.canavarro@farfetch.com> * [FEAT] Add PR number to log Signed-off-by: rita.canavarro <rita.canavarro@farfetch.com> * [FIX] Log msg Signed-off-by: rita.canavarro <rita.canavarro@farfetch.com> * [FEAT] Clean code Signed-off-by: rita.canavarro <rita.canavarro@farfetch.com> * [FIX] Remove shadow code Signed-off-by: rita.canavarro <rita.canavarro@farfetch.com> * [FIX] Go format Signed-off-by: rita.canavarro <rita.canavarro@farfetch.com> * [FEAT] Update objstore Signed-off-by: rita.canavarro <rita.canavarro@farfetch.com> * [FEAT] Update objstore package Signed-off-by: rita.canavarro <rita.canavarro@farfetch.com> * [FEAT] Update storage.md Signed-off-by: rita.canavarro <rita.canavarro@farfetch.com> * [FEAT] Update erroring bucket Signed-off-by: rita.canavarro <rita.canavarro@farfetch.com> * [FEAT] Update erroring bucket Signed-off-by: rita.canavarro <rita.canavarro@farfetch.com> --------- Signed-off-by: rita.canavarro <rita.canavarro@farfetch.com> * Update objstore library to latest main (#6722) This commit updates the obstore library to the latest main version which optimizes the Iter operation to only request object names. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Store: store responses should always be sorted (#6706) * Store: always sort, just compare labelset in proxy heap Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Store: add escape hatch to skip store resorting Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Store: remove stringset This is the wrong approach to detect if we need to resort. It cannot detect if we might end up with an unsorted series set if we add extLabels. Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Docs: drop paragraph about deduplication on inner labels Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> --------- Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> Co-authored-by: Michael Hoffmann <michael.hoffmann@aiven.io> * Updates busybox SHA (#6724) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Cut patch release v0.32.3 Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: rita.canavarro <rita.canavarro@farfetch.com> Signed-off-by: GitHub <noreply@github.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Michael Hoffmann <mhoffm@posteo.de> Co-authored-by: Rita Canavarro <98762287+ritaCanavarro@users.noreply.github.com> Co-authored-by: Michael Hoffmann <michael.hoffmann@aiven.io> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>

pull-request-size bot added the size/M label Sep 6, 2023

MichaHoffmann force-pushed the mhoffm-proxy-heap-sorting-issue-2 branch 4 times, most recently from c285a43 to 882e417 Compare September 6, 2023 15:58

pull-request-size bot added size/L and removed size/M labels Sep 6, 2023

MichaHoffmann mentioned this pull request Sep 6, 2023

Non deterministic deduplication on receive and sidecar #6705

Closed

saswatamcode previously approved these changes Sep 7, 2023

View reviewed changes

saswatamcode requested review from fpetkovski and GiedriusS September 7, 2023 05:00

fpetkovski reviewed Sep 7, 2023

View reviewed changes

pkg/store/bucket.go Outdated Show resolved Hide resolved

mhoffm-aiven mentioned this pull request Sep 7, 2023

Store: add failing test for proxy heap sorting issue #6702

Closed

MichaHoffmann mentioned this pull request Sep 7, 2023

Sketch: how to stream sorted store responses with dropped replica labels #6710

Closed

yeya24 reviewed Sep 7, 2023

View reviewed changes

pkg/store/flushable.go Outdated Show resolved Hide resolved

MichaHoffmann dismissed saswatamcode’s stale review via 6bd932a September 9, 2023 11:44

MichaHoffmann force-pushed the mhoffm-proxy-heap-sorting-issue-2 branch 2 times, most recently from 9628c04 to 053ca63 Compare September 9, 2023 12:13

pull-request-size bot removed the size/L label Sep 9, 2023

pull-request-size bot added the size/XL label Sep 9, 2023

MichaHoffmann force-pushed the mhoffm-proxy-heap-sorting-issue-2 branch from d4bc4bd to e8d5a69 Compare September 9, 2023 15:57

MichaHoffmann force-pushed the mhoffm-proxy-heap-sorting-issue-2 branch from e8d5a69 to d54de93 Compare September 11, 2023 13:34

MichaHoffmann requested review from saswatamcode, fpetkovski and yeya24 September 11, 2023 13:35

MichaHoffmann force-pushed the mhoffm-proxy-heap-sorting-issue-2 branch from d54de93 to 832ac03 Compare September 11, 2023 14:19

fpetkovski reviewed Sep 12, 2023

View reviewed changes

MichaHoffmann requested a review from fpetkovski September 14, 2023 12:22

GiedriusS reviewed Sep 14, 2023

View reviewed changes

mhoffm-aiven and others added 3 commits September 14, 2023 15:55

Store: always sort, just compare labelset in proxy heap

aab24f1

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

Store: add escape hatch to skip store resorting

ae8b083

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

Store: remove stringset

2a10d70

This is the wrong approach to detect if we need to resort. It cannot detect if we might end up with an unsorted series set if we add extLabels. Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

MichaHoffmann force-pushed the mhoffm-proxy-heap-sorting-issue-2 branch from 16439df to 2a10d70 Compare September 14, 2023 13:55

Docs: drop paragraph about deduplication on inner labels

b928701

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

GiedriusS approved these changes Sep 14, 2023

View reviewed changes

fpetkovski approved these changes Sep 14, 2023

View reviewed changes

saswatamcode approved these changes Sep 15, 2023

View reviewed changes

saswatamcode merged commit 931439b into thanos-io:main Sep 15, 2023
16 checks passed

chenrui333 mentioned this pull request Sep 20, 2023

thanos 0.32.3 Homebrew/homebrew-core#144260

Merged

jtb-sre mentioned this pull request Apr 6, 2024

0.32.0 caused spike in network traffic #7213

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store: store responses should always be sorted #6706

Store: store responses should always be sorted #6706

MichaHoffmann commented Sep 6, 2023 •

edited

vCra commented Sep 6, 2023 •

edited

tekicode commented Sep 6, 2023

saswatamcode left a comment •

edited

GiedriusS commented Sep 7, 2023 •

edited

MichaHoffmann commented Sep 7, 2023

GiedriusS commented Sep 7, 2023

MichaHoffmann commented Sep 7, 2023 •

edited

douglascamata commented Sep 7, 2023 •

edited

GiedriusS commented Sep 8, 2023

fpetkovski commented Sep 8, 2023

MichaHoffmann commented Sep 8, 2023

fpetkovski commented Sep 8, 2023

MichaHoffmann commented Sep 9, 2023

GiedriusS commented Sep 10, 2023

fpetkovski Sep 12, 2023

MichaHoffmann Sep 12, 2023

fpetkovski Sep 12, 2023

MichaHoffmann Sep 12, 2023

fpetkovski Sep 12, 2023 •

edited

MichaHoffmann Sep 12, 2023

fpetkovski Sep 12, 2023

mhoffm-aiven Sep 12, 2023

GiedriusS Sep 14, 2023

MichaHoffmann commented Sep 14, 2023

GiedriusS left a comment

GiedriusS Sep 14, 2023

MichaHoffmann Sep 14, 2023

MichaHoffmann Sep 14, 2023

fpetkovski left a comment

saswatamcode left a comment

Store: store responses should always be sorted #6706

Store: store responses should always be sorted #6706

Conversation

MichaHoffmann commented Sep 6, 2023 • edited

Changes

Verification

vCra commented Sep 6, 2023 • edited

tekicode commented Sep 6, 2023

saswatamcode left a comment • edited

Choose a reason for hiding this comment

GiedriusS commented Sep 7, 2023 • edited

MichaHoffmann commented Sep 7, 2023

GiedriusS commented Sep 7, 2023

MichaHoffmann commented Sep 7, 2023 • edited

douglascamata commented Sep 7, 2023 • edited

GiedriusS commented Sep 8, 2023

fpetkovski commented Sep 8, 2023

MichaHoffmann commented Sep 8, 2023

fpetkovski commented Sep 8, 2023

MichaHoffmann commented Sep 9, 2023

GiedriusS commented Sep 10, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fpetkovski Sep 12, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichaHoffmann commented Sep 14, 2023

GiedriusS left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fpetkovski left a comment

Choose a reason for hiding this comment

saswatamcode left a comment

Choose a reason for hiding this comment

MichaHoffmann commented Sep 6, 2023 •

edited

vCra commented Sep 6, 2023 •

edited

saswatamcode left a comment •

edited

GiedriusS commented Sep 7, 2023 •

edited

MichaHoffmann commented Sep 7, 2023 •

edited

douglascamata commented Sep 7, 2023 •

edited

fpetkovski Sep 12, 2023 •

edited