Fix sharding behavior for vector matches #5799

sdufel · 2022-10-18T21:51:32Z

When analyzing a query composed of boolean expression with a non-"on" vector match, the current vertical sharding logic is incorrectly scoping to labels in the vector match and changing the semantics of the query.

Example:

foo and without (lbl) bar

Changes

Prevents the vertical sharding query analyzer from using labels in a vector match to shard on if the match is done with the ignoring keyword (match On == false)

Verification

Manual verification of queries with and without the change; after this modification, queries worked as expected.

I am not clear on if it's actually possible to shard without labels without breaking queries; it's possible that a more appropriate change would be to the shard matcher code in the querier.

yeya24 · 2022-10-19T05:07:26Z

pkg/querysharding/analyzer.go

@@ -97,7 +97,7 @@ func (a *QueryAnalyzer) Analyze(query string) (QueryAnalysis, error) {
 				return fmt.Errorf("expressions with %s are not shardable", n.Func.Name)
 			}
 		case *parser.BinaryExpr:
-			if n.VectorMatching != nil {
+			if n.VectorMatching != nil && n.VectorMatching.On {


@sdufel Can you add an unit test case please?

After discussing with @fpetkovski, I think we pinned down the issue - a recent change that removed the __name__ label exclusion from sharding caused this regression.

I adjusted the patch to ensure that __name__ isn't used as a shard label for the case of vector matches using the ignoring keyword and added tests to the analyzer.

Sharding by __name__ appears to be fine for aggregations, but it needs special treatment for vector matching.

When analyzing a query composed of boolean expression with a non-"on" vector match, the current vertical sharding logic is incorrectly scoping to labels in the vector match and changing the semantics of the query. Example: ``` foo and without (lbl) bar ``` Signed-off-by: Samuel Dufel <samuel.dufel@shopify.com>

sharadgaur · 2022-10-19T16:47:05Z

pkg/querysharding/analyzer.go

@@ -99,6 +99,9 @@ func (a *QueryAnalyzer) Analyze(query string) (QueryAnalysis, error) {
 		case *parser.BinaryExpr:
 			if n.VectorMatching != nil {
 				shardingLabels := without(n.VectorMatching.MatchingLabels, []string{"le"})
+				if !n.VectorMatching.On && len(shardingLabels) > 0 {
+					shardingLabels = append(shardingLabels, "__name__")


This looks good.

yeya24

LGTM. cc @fpetkovski for another look

alanprot · 2022-10-20T03:10:39Z

I tested this patch on cortex and it indeed fixed the issue.

Do we also need to add this on the analyzeRootExpression?

thanos/pkg/querysharding/analyzer.go

Lines 130 to 132 in 35a3e3d

    
           if n.VectorMatching != nil && n.VectorMatching.On { 
        
           	shardingLabels := without(n.VectorMatching.MatchingLabels, []string{"le"}) 
        
           	return newShardableByLabels(shardingLabels, n.VectorMatching.On)

As a side note, the https://github.com/prometheus/compliance/tree/main/promql was not able to catch this issue.

I added locally the following query to cover this problem:

QUERY: process_virtual_memory_bytes and ignoring(code) promhttp_metric_handler_requests_total
START: 2022-10-20 02:26:10 +0000 UTC, STOP: 2022-10-20 02:36:10 +0000 UTC, STEP: 10s
RESULT: FAILED: Query returned different results:
  model.Matrix{
- 	(
- 		s"""
- 		process_virtual_memory_bytes{instance="localhost:10000", job="demo"} =>
- 		737312768 @[1666233340]
- 		737312768 @[1666233350]
- 		737312768 @[1666233360]
- 		737312768 @[1666233370]
- 		s"""
- 	),
- 	(
- 		s"""
- 		process_virtual_memory_bytes{instance="localhost:10001", job="demo"} =>
- 		737050624 @[1666233340]
- 		737050624 @[1666233350]
- 		737050624 @[1666233360]
- 		737050624 @[1666233370]
- 		s"""
- 	),
- 	(
- 		s"""
- 		process_virtual_memory_bytes{instance="localhost:10002", job="demo"} =>
- 		737050624 @[1666233340]
- 		737050624 @[1666233350]
- 		737050624 @[1666233360]
- 		737050624 @[1666233370]
- 		s"""
- 	),
  }

fpetkovski

Thanks for the fix, this looks good 👍

GiedriusS · 2022-10-20T09:35:14Z

pkg/querysharding/analyzer.go

@@ -99,6 +99,9 @@ func (a *QueryAnalyzer) Analyze(query string) (QueryAnalysis, error) {
 		case *parser.BinaryExpr:
 			if n.VectorMatching != nil {
 				shardingLabels := without(n.VectorMatching.MatchingLabels, []string{"le"})
+				if !n.VectorMatching.On && len(shardingLabels) > 0 {
+					shardingLabels = append(shardingLabels, "__name__")


Minor nit: let's use model.MetricNameLabel.

GiedriusS · 2022-10-20T09:35:19Z

pkg/querysharding/analyzer_test.go

@@ -142,20 +142,20 @@ sum by (container) (
 		{
 			name:           "binary expression with without vector matching and grouping",
 			expression:     `sum without (cluster, pod) (http_requests_total{code="400"}) / ignoring (pod) sum without (cluster, pod) (http_requests_total)`,
-			shardingLabels: []string{"pod", "cluster"},
+			shardingLabels: []string{"pod", "cluster", "__name__"},


Thanks, just updated these

Signed-off-by: Samuel Dufel <samuel.dufel@shopify.com>

sdufel · 2022-10-20T15:05:50Z

The analyzeRootExpression func already has a check for VectorMatching.On == true, so I think it's fine as-is.

pull-request-size bot added the size/XS label Oct 18, 2022

yeya24 reviewed Oct 19, 2022

View reviewed changes

sdufel force-pushed the fix-sharding-vector-matching branch from 7caa8bd to 3e4f24f Compare October 19, 2022 16:45

sharadgaur reviewed Oct 19, 2022

View reviewed changes

yeya24 previously approved these changes Oct 19, 2022

View reviewed changes

alanprot mentioned this pull request Oct 20, 2022

Add test case for vector matches on the promql compliance prometheus/compliance#92

Open

alanprot previously approved these changes Oct 20, 2022

View reviewed changes

fpetkovski previously approved these changes Oct 20, 2022

View reviewed changes

GiedriusS reviewed Oct 20, 2022

View reviewed changes

Replace string literals with constants

a8b39b5

Signed-off-by: Samuel Dufel <samuel.dufel@shopify.com>

sdufel dismissed stale reviews from fpetkovski, alanprot, and yeya24 via a8b39b5 October 20, 2022 15:02

pull-request-size bot added size/S and removed size/XS labels Oct 20, 2022

yeya24 approved these changes Oct 20, 2022

View reviewed changes

yeya24 enabled auto-merge (squash) October 20, 2022 15:10

yeya24 merged commit d5744e7 into thanos-io:main Oct 20, 2022

alanprot mentioned this pull request Oct 20, 2022

chore: Update Thanos cortexproject/cortex#4929

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix sharding behavior for vector matches #5799

Fix sharding behavior for vector matches #5799

sdufel commented Oct 18, 2022

yeya24 Oct 19, 2022

sdufel Oct 19, 2022

sharadgaur Oct 19, 2022

yeya24 left a comment

alanprot commented Oct 20, 2022 •

edited

fpetkovski left a comment

GiedriusS Oct 20, 2022

GiedriusS Oct 20, 2022

sdufel Oct 20, 2022

sdufel commented Oct 20, 2022

Fix sharding behavior for vector matches #5799

Fix sharding behavior for vector matches #5799

Conversation

sdufel commented Oct 18, 2022

Changes

Verification

yeya24 Oct 19, 2022

Choose a reason for hiding this comment

sdufel Oct 19, 2022

Choose a reason for hiding this comment

sharadgaur Oct 19, 2022

Choose a reason for hiding this comment

yeya24 left a comment

Choose a reason for hiding this comment

alanprot commented Oct 20, 2022 • edited

fpetkovski left a comment

Choose a reason for hiding this comment

GiedriusS Oct 20, 2022

Choose a reason for hiding this comment

GiedriusS Oct 20, 2022

Choose a reason for hiding this comment

sdufel Oct 20, 2022

Choose a reason for hiding this comment

sdufel commented Oct 20, 2022

alanprot commented Oct 20, 2022 •

edited