feat(blooms): compute chunks once #12664

owen-d · 2024-04-17T19:51:54Z

Computes chunks once during planning and passes them to the queriers, reducing a lot of calls to the index.

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

…ks-once Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

…ks-once Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

…ks-once Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

pkg/logproto/logproto.proto

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

…ns + new override Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

…ks-once Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

chaudum · 2024-04-25T06:50:30Z

pkg/logql/shards.go

@@ -176,6 +203,16 @@ func (s Shard) Ptr() *Shard {
 	return &s
 }

+func (s Shard) Bind(chunks *logproto.ChunkRefGroup) *ShardWithChunkRefs {


This function replaces Prt(), right? Can Ptr() be removed then?

It can be, but is still used in a bunch of tests I'm not inclined to update here

chaudum · 2024-04-25T06:52:35Z

pkg/logql/shards.go

+	res := &ShardWithChunkRefs{
+		Shard: s,
+	}
+	if chunks != nil {
+		res.chunks = chunks
+	}
+	return res


Would be the same as

Suggested change

res := &ShardWithChunkRefs{

Shard: s,

}

if chunks != nil {

res.chunks = chunks

}

return res

return &ShardWithChunkRefs{

Shard: s,

chunks: chunks,

}

Good call 👍

salvacorts · 2024-04-29T10:01:21Z

pkg/indexgateway/client.go

@@ -355,48 +352,11 @@ func (s *GatewayClient) GetShards(
 			return errCt <= maxErrs
 		},
 	); err != nil {
-		if isUnimplementedCallError(err) {


IIUC we are removing this since we already rolled out index-gws that support the get shards method, right?

No (the OSS repo is independent from our internal deployments at Grafana Labs). This is because the new bounded shards implementation is an intentional choice that needs to be turned on (default=power_of_two), so I removed the fallback for simplicity's sake.

salvacorts · 2024-04-29T10:08:33Z

pkg/indexgateway/gateway.go

+	filtered := refs
+
+	// 2) filter via blooms if enabled
+	if g.bloomQuerier != nil {


I think we should check here if the AST has any filtering expression:

if g.bloomQuerier != nil && syntax.ExtractLineFilters(p.Plan().AST)) != 0 {

salvacorts · 2024-04-29T10:46:06Z

pkg/logql/shards.go

+	res := &ShardWithChunkRefs{
+		Shard: s,
+	}
+	if chunks != nil {


nit: maybe it's not necessary but since elsewhere we are checking res.Chunks != nil to check if we are passing chunks in the shards, I think we should only set this if len(chunks.refs) > 0

You're right that there's a distinction -- nil indicates there is no override, but a zero-length chunk override is still valid and can happen when we lookup labels which don't map to chunks.

salvacorts · 2024-04-29T10:55:17Z

pkg/querier/multi_tenant_querier.go

+	// in case of multiple tenants, we need to filter the store chunks by tenant if they are provided
+	storeOverridesByTenant := make(map[string][]*logproto.ChunkRef)
+	if overrides := params.GetStoreChunks(); overrides != nil {
+		storeOverridesByTenant = partitionChunkRefsByTenant(params.GetStoreChunks().Refs)


nit:

storeOverridesByTenant = partitionChunkRefsByTenant(overrides.Refs)

salvacorts · 2024-04-29T10:58:07Z

pkg/querier/multi_tenant_querier.go

+func partitionChunkRefsByTenant(refs []*logproto.ChunkRef) map[string][]*logproto.ChunkRef {
+	filtered := make(map[string][]*logproto.ChunkRef)
+	for _, ref := range refs {
+		filtered[ref.UserID] = append(filtered[ref.UserID], ref)


nit: should we preallocate the user refs slice with some capacity?

I'll probably skip this until I see it being a problem (or happy to see it in a followup PR)

salvacorts · 2024-04-29T11:00:49Z

pkg/querier/queryrange/codec.go

@@ -2125,7 +2086,81 @@ func mergeLokiResponse(responses ...queryrangebase.Response) *LokiResponse {
 	}
 }

-// In some other world LabelRequest could implement queryrangebase.Request.
+func parseRangeQueryWithStoreChunksExtension(r *http.Request) (*LokiRequest, error) {


I don't see this method being called conditionally of having chunks. I think this as well as the one for instant queries should be renamed to parseRangeQuery and parseInstantQuery

salvacorts · 2024-04-29T11:04:24Z

pkg/querier/queryrange/shard_resolver.go

@@ -265,37 +269,43 @@ func (r *dynamicShardResolver) ShardingRanges(expr syntax.Expr, targetBytesPerSh
 		if resp, ok := httpgrpc.HTTPResponseFromError(err); ok && (resp.Code == http.StatusNotFound) {


We still need this fallback for ingesters, right?

I should probably remove this code for consistency with removing the same in gatewayclient

salvacorts · 2024-04-29T11:10:34Z

pkg/storage/stores/composite_store.go

+		from,
+		through model.Time,
+		predicate chunk.Predicate,
+		storeChunksOverride *logproto.ChunkRefGroup,


nit: I understand GetChunks does some extra filtering to the storeChunksOverride (validates and filters by the time range and fetcher). Still I think we should avoid calling GetChunk refs at all instead of passing the storeChunksOverride around. What about having a GetFetcherForChunks that returns grouped chunks by fetchers.

salvacorts · 2024-04-29T11:18:38Z

pkg/storage/store.go

@@ -411,7 +417,7 @@ func (s *LokiStore) lazyChunks(ctx context.Context, from, through model.Time, pr
 	stats := stats.FromContext(ctx)

 	start := time.Now()
-	chks, fetchers, err := s.GetChunks(ctx, userID, from, through, predicate)
+	chks, fetchers, err := s.GetChunks(ctx, userID, from, through, predicate, storeChunksOverride)


nit: Here's where I think we should do something like:

var chunks [][]chunkRefs var fetchers []fetchers if storeChunksOverride > 0 { chunks, fetchers := s.GetFetcherForChunks(from, through, storeChunksOverride) } else { chunks, fetchers := s.GetChunks(ctx, userID, from, through, predicate) }

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

owen-d added 9 commits March 27, 2024 16:52

remove unnecessary fallback

4807f48

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

[wip] wiring up support to pass store chunks to queriers

48402d2

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

Merge remote-tracking branch 'upstream/main' into blooms/compute-chun…

bb283f1

…ks-once Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

[wip] threading through store overrides for chunkrefs

7a05032

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

multi-tenant querier partitions store overrides by tenant id

97bc139

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

metrics & ifc alignment

3858893

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

Merge remote-tracking branch 'upstream/main' into blooms/compute-chun…

ea68788

…ks-once Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

remove unused fn

46d85f9

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

send chunks in shards resp

db62dd8

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

owen-d requested a review from a team as a code owner April 17, 2024 19:51

pull-request-size bot added the size/XL label Apr 17, 2024

owen-d added 8 commits April 17, 2024 12:54

type alignment

ad855e6

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

Merge remote-tracking branch 'upstream/main' into blooms/compute-chun…

93a9ce4

…ks-once Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

type alignment

6ce86b4

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

ShardsResponse.Merge extension

294261d

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

fix unrelated codec test err msg

d7f2af9

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

tidy

e8f58f5

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

binding shard to chunk refs

4639cfd

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

simplify+pointer for shard chunks

a764c11

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

owen-d force-pushed the blooms/compute-chunks-once branch from c88b9f4 to a764c11 Compare April 18, 2024 23:20

fix signature

3ba9330

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

salvacorts reviewed Apr 19, 2024

View reviewed changes

pkg/logproto/logproto.proto Show resolved Hide resolved

owen-d added 6 commits April 19, 2024 13:27

precomputed chunk logging

b2990bf

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

log matchers & always use mutex while accumulating chunks to shards

141c4f7

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

more logging

8db855d

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

better logging for gateway.go

8bdb823

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

independent handling for precomputed chunks vs bloom enablement optio…

56eabcc

…ns + new override Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

Merge remote-tracking branch 'upstream/main' into blooms/compute-chun…

33c8e82

…ks-once Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

pull-request-size bot added size/XXL and removed size/XL labels Apr 23, 2024

github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Apr 23, 2024

make doc

a3bd99c

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

owen-d force-pushed the blooms/compute-chunks-once branch from 53d4a46 to a3bd99c Compare April 24, 2024 16:23

chaudum approved these changes Apr 25, 2024

View reviewed changes

salvacorts reviewed Apr 29, 2024

View reviewed changes

owen-d added 2 commits April 29, 2024 13:32

pr feedback

82923e7

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

pr feedback: only dispatch to bloom querier when line filters exist

7b2f72e

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

owen-d merged commit bc78d13 into grafana:main Apr 29, 2024
60 checks passed

shantanualsi pushed a commit that referenced this pull request May 6, 2024

feat(blooms): compute chunks once (#12664)

04b3349

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

This was referenced May 6, 2024

chore(k201): release 3.1.0 #12894

Open

chore(k202): release 3.1.0 #12945

Open

loki-gh-app bot mentioned this pull request May 20, 2024

chore(k203): release 3.1.0 #12988

Open

This was referenced May 27, 2024

chore(k204): release 3.1.0 #13037

Open

chore(k205): release 3.1.0 #13102

Open

This was referenced Jun 10, 2024

chore(k206): release 3.1.0 #13184

Open

chore(k207): release 3.1.0 #13225

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(blooms): compute chunks once #12664

feat(blooms): compute chunks once #12664

owen-d commented Apr 17, 2024 •

edited

Loading

chaudum Apr 25, 2024

owen-d Apr 29, 2024

chaudum Apr 25, 2024

owen-d Apr 29, 2024

salvacorts Apr 29, 2024 •

edited

Loading

owen-d Apr 29, 2024

salvacorts Apr 29, 2024

salvacorts Apr 29, 2024

owen-d Apr 29, 2024

salvacorts Apr 29, 2024

salvacorts Apr 29, 2024 •

edited

Loading

owen-d Apr 29, 2024 •

edited

Loading

salvacorts Apr 29, 2024 •

edited

Loading

salvacorts Apr 29, 2024

owen-d Apr 29, 2024

salvacorts Apr 29, 2024 •

edited

Loading

salvacorts Apr 29, 2024

		@@ -265,37 +269,43 @@ func (r *dynamicShardResolver) ShardingRanges(expr syntax.Expr, targetBytesPerSh
		if resp, ok := httpgrpc.HTTPResponseFromError(err); ok && (resp.Code == http.StatusNotFound) {

feat(blooms): compute chunks once #12664

feat(blooms): compute chunks once #12664

Conversation

owen-d commented Apr 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

salvacorts Apr 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

salvacorts Apr 29, 2024 • edited Loading

Choose a reason for hiding this comment

owen-d Apr 29, 2024 • edited Loading

Choose a reason for hiding this comment

salvacorts Apr 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

salvacorts Apr 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

owen-d commented Apr 17, 2024 •

edited

Loading

salvacorts Apr 29, 2024 •

edited

Loading

salvacorts Apr 29, 2024 •

edited

Loading

owen-d Apr 29, 2024 •

edited

Loading

salvacorts Apr 29, 2024 •

edited

Loading

salvacorts Apr 29, 2024 •

edited

Loading