From 7d3550ab03cbac1382f21706ffa7206f9c48bdd8 Mon Sep 17 00:00:00 2001 From: David Percy Date: Tue, 7 May 2024 18:53:06 +0000 Subject: [PATCH 01/25] PERF-5374 Improve comment headers for multiplanner/ workloads Mostly I updated the first sentence about each test's goal. In some cases I also added some more detail. In one case, 'NoResults.yml', I realized that the original description was wrong because I hadn't thought through what would actually happen. The results at [full-results.ipynb](https://github.com/10gen/product-perf-experimentations/blob/master/investigations/PERF-5121-compare-multiplanners/full-results.ipynb) are consistent with the new description. --- .../query/multiplanner/BlockingSort.yml | 11 +++---- .../multiplanner/ClusteredCollection.yml | 12 ++++---- .../query/multiplanner/CompoundIndexes.yml | 17 ++++++----- .../multiplanner/MultiplannerWithGroup.yml | 22 +++++++++----- .../query/multiplanner/NoGoodPlan.yml | 17 ++++++----- .../query/multiplanner/NoResults.yml | 10 ++----- .../query/multiplanner/NoSuchField.yml | 7 +++-- .../multiplanner/NonBlockingVsBlocking.yml | 21 +++++++++----- src/workloads/query/multiplanner/Simple.yml | 4 +-- .../query/multiplanner/UseClusteredIndex.yml | 13 +++++---- .../query/multiplanner/VariedSelectivity.yml | 29 ++++++++++++------- 11 files changed, 92 insertions(+), 71 deletions(-) diff --git a/src/workloads/query/multiplanner/BlockingSort.yml b/src/workloads/query/multiplanner/BlockingSort.yml index 97c0720f4..20acd8486 100644 --- a/src/workloads/query/multiplanner/BlockingSort.yml +++ b/src/workloads/query/multiplanner/BlockingSort.yml @@ -1,11 +1,12 @@ SchemaVersion: 2018-07-01 Owner: "@mongodb/query" Description: | - The goal of this test is to exercise multiplanning. We create as many indexes as possible, and run - a query that makes all of them eligible, so we get as many competing plans as possible. We also - add a sort stage on an unindexed field, ensuring that every plan is a blocking plan. Because all - plans are blocking and return as many documents as possible, multiplanning will hit "max works" - instead of EOF of numToReturn. This maximizes the overhead of multiplanning on both classic and SBE. + The goal of this test is to show how a blocking sort can increase the overhead of multiplanning. + We create as many indexes as possible, and run a query that makes all of them eligible, so we + get as many competing plans as possible. We also add a sort stage on an unindexed field, + ensuring that every plan is a blocking plan. Because all plans are blocking and return as many + documents as possible, multiplanning will hit "max works" instead of EOF of numToReturn. + This maximizes the overhead of multiplanning on both classic and SBE. We expect classic to have better latency and throughput than SBE on this workload, and we expect the combination of classic planner + SBE execution (PM-3591) to perform about diff --git a/src/workloads/query/multiplanner/ClusteredCollection.yml b/src/workloads/query/multiplanner/ClusteredCollection.yml index 681014e28..fd805f08a 100644 --- a/src/workloads/query/multiplanner/ClusteredCollection.yml +++ b/src/workloads/query/multiplanner/ClusteredCollection.yml @@ -1,13 +1,13 @@ SchemaVersion: 2018-07-01 Owner: "@mongodb/query" Description: | - The goal of this test is to exercise multiplanning. We create as many indexes as possible, and run a - query that makes all of them eligible, so we get as many competing plans as possible. Here, we do this on a - clustered collection that has very large strings as _id. + The goal of this test is to exercise multiplanning in the prepence of clustered indexes. We + create as many indexes as possible, and run a query that makes all of them eligible, so we get + as many competing plans as possible. The collection is clustered and has very large strings as + _id. - We expect classic to have better latency and throughput than SBE on this workload, - and we expect the combination of classic planner + SBE execution (PM-3591) to perform about - as well as classic. + This workload is similar to 'Simple.yml' except for the collection being clustered. + Maybe we expect the larger record IDs to make fetching more expensive. GlobalDefaults: dbname: &db test diff --git a/src/workloads/query/multiplanner/CompoundIndexes.yml b/src/workloads/query/multiplanner/CompoundIndexes.yml index 170833bb0..eadbe21dd 100644 --- a/src/workloads/query/multiplanner/CompoundIndexes.yml +++ b/src/workloads/query/multiplanner/CompoundIndexes.yml @@ -1,13 +1,16 @@ SchemaVersion: 2018-07-01 Owner: "@mongodb/query" Description: | - The goal of this test is to exercise multiplanning. We create as many compound indexes as possible, each - comprising of tenantId and another field. We then run a query with an equality predicate on tenantId and - other predicates a small number of fields, so we get all the plans that use relevant indexes. + This test exercises multiplanning in the presence of a common pattern, using "tenant IDs": + you have a single collection that conceptually is partitioned into one collection per user ("tenant"), + so each query has an extra equality predicate on tenantId, and each index is prefixed with 'tenantId: 1'. - We expect classic to have better latency and throughput than SBE on this workload, - and we expect the combination of classic planner + SBE execution (PM-3591) to perform about - as well as classic. + Otherwise this workload is the same as 'Simple.yml', so we expect it to behave the same: + classic should have better latency and throughput than SBE, and the combination of classic + planner + SBE execution (PM-3591) to perform about as well as classic. + + TODO(CR) Storch noticed the selectivities don't make sense here: the data is too small since we carve up the same total size among many tenants. + Make a note or leave a TODO pointing to that fix. GlobalDefaults: dbname: &db test @@ -340,4 +343,4 @@ AutoRun: - standalone-all-feature-flags # At time of writing this will enable PM-3591. - standalone-classic-query-engine branch_name: - $gte: v7.3 \ No newline at end of file + $gte: v7.3 diff --git a/src/workloads/query/multiplanner/MultiplannerWithGroup.yml b/src/workloads/query/multiplanner/MultiplannerWithGroup.yml index 4de202392..408e6e627 100644 --- a/src/workloads/query/multiplanner/MultiplannerWithGroup.yml +++ b/src/workloads/query/multiplanner/MultiplannerWithGroup.yml @@ -1,15 +1,21 @@ SchemaVersion: 2018-07-01 Owner: "@mongodb/query" Description: | - The goal of this test is to exercise multiplanning. We create as many indexes as possible, and run a - query that makes all of them eligible, so we get as many competing plans as possible. We add a - group stage, which is blocking. The SBE multiplanner will multiplan group as it is a part of the - canonical query, but the classic multiplanner will not plan. This means the SBE multiplanner will - have the overhead of trial running blocking plans when compared to the classic multiplanner. + This test was created to show how three different multiplanners handle $group. + The query is essentially the one from 'Simple.yml': we have as many indexed predicates as + possible, to create as many indexed plans as possible, but only one of those predicates is + selective, which means only one of those plans is efficient. Where this test departs from + 'Simple.yml' is by adding a $group stage after the access-path part of the query. - We expect classic to have better latency and throughput than SBE on this workload, - and we expect the combination of classic planner + SBE execution (PM-3591) to perform about - as well as classic. + We expect the Classic multiplanner with Classic execution to reach EOF during planning, and + then feed those documents into the $group stage. By contrast the SBE multiplanner will not + reuse results gathered during multiplanning. + + The Classic multiplanner with SBE execution would normally be able to avoid starting over, + when the query finishes during multiplanning, but in this test it can't because of the $group. + When there are any pipeline stages beyond the access-path part of the query, then when + multiplanning finishes we construct a new SBE plan with both the access path and the other + pipeline stages. GlobalDefaults: dbname: &db test diff --git a/src/workloads/query/multiplanner/NoGoodPlan.yml b/src/workloads/query/multiplanner/NoGoodPlan.yml index 5f6d255ea..247ab5936 100644 --- a/src/workloads/query/multiplanner/NoGoodPlan.yml +++ b/src/workloads/query/multiplanner/NoGoodPlan.yml @@ -1,15 +1,16 @@ SchemaVersion: 2018-07-01 Owner: "@mongodb/query" Description: | - The goal of this test is to exercise multiplanning. We create 64 indexes and run a query that - makes all of them eligible, so we get as many competing plans as possible. The only selective - field is unindexed, however, meaning no index will be effective in planning. By ensuring all plans - are relatively equally bad, we are likely to hit the works limit sooner than the 101 results - limit. + The goal of this test is to exercise the case in multiplanning where all competing plans are bad. + + As in 'Simple.yml' we create 64 indexes and run a query that makes all of them eligible, so we + get as many competing plans as possible. However, unlike 'Simple.yml', in this workload the only + selective field is unindexed, however, meaning no index will be effective in planning. By + ensuring all plans are relatively equally bad, we are likely to hit the works limit sooner than + the 101 results limit. - We expect classic to have better latency and throughput than SBE on this workload, - and we expect the combination of classic planner + SBE execution (PM-3591) to perform about - as well as classic. + This case is intended not so much to show a difference between Classic and SBE, but to show a + case where any multiplanner will struggle. GlobalDefaults: dbname: &db test diff --git a/src/workloads/query/multiplanner/NoResults.yml b/src/workloads/query/multiplanner/NoResults.yml index c5f2989cf..1462ecfdd 100644 --- a/src/workloads/query/multiplanner/NoResults.yml +++ b/src/workloads/query/multiplanner/NoResults.yml @@ -1,14 +1,8 @@ SchemaVersion: 2018-07-01 Owner: "@mongodb/query" Description: | - The goal of this test is to exercise multiplanning. We create as many indexes as possible, and run a - query that makes all of them eligible, so we get as many competing plans as possible. All predicates - are very selective (match 0% of the documents). With zero results, we do no hit the EOF optimization - and all competing plans hit the works limit instead of document limit. - - We expect classic to have better latency and throughput than SBE on this workload, - and we expect the combination of classic planner + SBE execution (PM-3591) to perform about - as well as classic. + This test shows an example where we have many competing plans, but they all are very efficient: + they all have empty index bounds, which means any competing plan will finish immediately. GlobalDefaults: dbname: &db test diff --git a/src/workloads/query/multiplanner/NoSuchField.yml b/src/workloads/query/multiplanner/NoSuchField.yml index b01ab3bc1..eb3ee98f4 100644 --- a/src/workloads/query/multiplanner/NoSuchField.yml +++ b/src/workloads/query/multiplanner/NoSuchField.yml @@ -1,9 +1,10 @@ SchemaVersion: 2018-07-01 Owner: "@mongodb/query" Description: | - The goal of this test is to exercise multiplanning. We create as many indexes as possible, and run a - query that makes all of them eligible, so we get as many competing plans as possible. Here, we add - an additional predicate: {no_such_field: "none"} to guarantee that we hit getTrialPeriodMaxWorks(). + The goal of this test is to exercise the "max works" case of multiplanning. The test is similar + to 'Simple.yml' except we add an additional predicate: {no_such_field: "none"}, which is always + false on this dataset. This guarantees that the query will not be able to finish multiplanning + by producing enough documents, so instead we will hit getTrialPeriodMaxWorks(). We expect classic to have better latency and throughput than SBE on this workload, and we expect the combination of classic planner + SBE execution (PM-3591) to perform about diff --git a/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml b/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml index ea90f2899..66c602846 100644 --- a/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml +++ b/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml @@ -1,13 +1,20 @@ SchemaVersion: 2018-07-01 Owner: "@mongodb/query" Description: | - The goal of this test is to exercise multiplanning. If the selectivity value is small enough (less - than 0.5), the optimal plan is to employ a blocking plan by scanning a segment of empty data and - conducting a blocking-sort operation, whereas the other plans' index provides the right sort - order, but requires a full scan, and every document is rejected after the FETCH stage. Because the - SBE multiplanner can't round-robin, it has a heuristic "try nonblocking plans first". This - scenario is a worst case for that heuristic, because we'll try the best plan last. Otherwise, an - IXSCAN and FETCH non-blocking plan will be used. + The goal of this test is to exercise multiplanning when both blocking and non-blocking plans are + available. + + If the selectivity value is small enough (less than 0.5), the optimal plan is to employ a + blocking plan by scanning a segment of empty data and conducting a blocking-sort operation, + whereas the other plans' index provides the right sort order, but requires a full scan, and + every document is rejected after the FETCH stage. + + Because the SBE multiplanner can't round-robin, it has a heuristic "try nonblocking plans first". + This scenario is a worst case for that heuristic, because we'll try the best plan last. + Otherwise, an IXSCAN and FETCH non-blocking plan will be used. + + Another point of view: this scenario shows that "non-blocking" plans can still do an unbounded + amount of work per getNext(). We expect classic to have better latency and throughput than SBE on this workload, and we expect the combination of classic planner + SBE execution (PM-3591) to perform about as well as classic. diff --git a/src/workloads/query/multiplanner/Simple.yml b/src/workloads/query/multiplanner/Simple.yml index 6e4cd7a4c..ead9541c1 100644 --- a/src/workloads/query/multiplanner/Simple.yml +++ b/src/workloads/query/multiplanner/Simple.yml @@ -4,7 +4,7 @@ Description: | The goal of this test is to exercise multiplanning. We create as many indexes as possible, and run a query that makes all of them eligible, so we get as many competing plans as possible. - The original goal of this test was to demonstrate weaknesses of the SBE multiplanner when compared to + This test was originally created to demonstrate weaknesses of the SBE multiplanner when compared to the classic multiplanner. Mainly, the SBE multiplanner can't round-robin between plans, which means it has to run the list of plans sequentially, which means we can't short-circuit when the shortest-running plan finishes. @@ -402,4 +402,4 @@ AutoRun: - standalone-all-feature-flags # At time of writing this will enable PM-3591. - standalone-classic-query-engine branch_name: - $gte: v7.3 \ No newline at end of file + $gte: v7.3 diff --git a/src/workloads/query/multiplanner/UseClusteredIndex.yml b/src/workloads/query/multiplanner/UseClusteredIndex.yml index 764eb2c00..20cbe9ea7 100644 --- a/src/workloads/query/multiplanner/UseClusteredIndex.yml +++ b/src/workloads/query/multiplanner/UseClusteredIndex.yml @@ -1,13 +1,14 @@ SchemaVersion: 2018-07-01 Owner: "@mongodb/query" Description: | - The goal of this test is to exercise multiplanning. We create as many indexes as possible, and run a - query that makes all of them eligible, so we get as many competing plans as possible. Here, we do this on a - clustered collection and add a selective predicate on _id, so that the clustered index is a viable candidate plan. + The goal of this test is to exercise multiplanning in the prepence of clustered indexes. We + create as many indexes as possible, and run a query that makes all of them eligible, so we get + as many competing plans as possible. The collection is clustered and has very large strings as + _id; also, one of the predicates is on _id which means a clustered collection scan is included + in the competing plans. - We expect classic to have better latency and throughput than SBE on this workload, - and we expect the combination of classic planner + SBE execution (PM-3591) to perform about - as well as classic. + This workload is similar to 'Simple.yml' except for the collection being clustered, and the + extra predicate. GlobalDefaults: dbname: &db test diff --git a/src/workloads/query/multiplanner/VariedSelectivity.yml b/src/workloads/query/multiplanner/VariedSelectivity.yml index 7f8e1a960..5e1c743c6 100644 --- a/src/workloads/query/multiplanner/VariedSelectivity.yml +++ b/src/workloads/query/multiplanner/VariedSelectivity.yml @@ -1,17 +1,24 @@ SchemaVersion: 2018-07-01 Owner: "@mongodb/query" Description: | - The goal of this test is to exercise multiplanning. We run the same query 7 times, each one with a - different selectivity value that we are comparing against x1, calcuated based on the number of - documents we want the query to match. This will help us measure the overhead of throwing out the - result set gathered during multi-planning when the result set exceeds 101 documents. Unlike many - of the other multiplanner/ workloads, we only test with 2 indexes here, because 2 indexes is a - worst case for throwing away results. Having more indexes increases planning time, but not query - execution time, so having more indexes makes the *relative* cost of throwing away results smaller. - - We expect classic to have better latency and throughput than SBE on this workload, - and we expect the combination of classic planner + SBE execution (PM-3591) to perform about - as well as classic. + The goal of this test is to measure the overhead of "throwing out" the initial results returned by + multiplanning. + + When a query runs with Classic multiplanner + Classic execution, then when multiplanning finishes + the query can resume running and reuse the partial results it gathered during multiplanning. By + contrast when running with SBE execution (regardless of choice of multiplanner), the query has + to start over--unless it already finished during multiplanning. This means SBE has a + discontinuity in performance as the size of the result set grows: when it crosses from 100 to + 102 documents, it has to recompute those first ~100 documents. + + To measure this, we run the same query 7 times, each one with a different selectivity value. + For example, in phase 'MultiplannerWith50ExpectedResults' we choose a selectivity of + '50 / collection size' to make the query return (approximately) 50 documents. + + Unlike many of the other multiplanner/ workloads, we only test with 2 indexes here, because + 2 indexes is a worst case for throwing away results. Having more indexes increases planning + time, but not query execution time, so having more indexes makes the *relative* cost of + throwing away results smaller. GlobalDefaults: dbname: &db test From 9ac18f3e3c20337daf45626feb0cf0d0396cc184 Mon Sep 17 00:00:00 2001 From: David Percy Date: Fri, 10 May 2024 13:44:36 -0400 Subject: [PATCH 02/25] typo of -> or Co-authored-by: David Storch --- src/workloads/query/multiplanner/BlockingSort.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/workloads/query/multiplanner/BlockingSort.yml b/src/workloads/query/multiplanner/BlockingSort.yml index 20acd8486..45c2fb052 100644 --- a/src/workloads/query/multiplanner/BlockingSort.yml +++ b/src/workloads/query/multiplanner/BlockingSort.yml @@ -5,7 +5,7 @@ Description: | We create as many indexes as possible, and run a query that makes all of them eligible, so we get as many competing plans as possible. We also add a sort stage on an unindexed field, ensuring that every plan is a blocking plan. Because all plans are blocking and return as many - documents as possible, multiplanning will hit "max works" instead of EOF of numToReturn. + documents as possible, multiplanning will hit "max works" instead of EOF or numToReturn. This maximizes the overhead of multiplanning on both classic and SBE. We expect classic to have better latency and throughput than SBE on this workload, From 727c8f5630f9eede87d45f85b94270437715642a Mon Sep 17 00:00:00 2001 From: David Percy Date: Fri, 10 May 2024 13:45:13 -0400 Subject: [PATCH 03/25] no need to split on "classic and SBE" Co-authored-by: David Storch --- src/workloads/query/multiplanner/BlockingSort.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/workloads/query/multiplanner/BlockingSort.yml b/src/workloads/query/multiplanner/BlockingSort.yml index 45c2fb052..c2b9125db 100644 --- a/src/workloads/query/multiplanner/BlockingSort.yml +++ b/src/workloads/query/multiplanner/BlockingSort.yml @@ -6,7 +6,7 @@ Description: | get as many competing plans as possible. We also add a sort stage on an unindexed field, ensuring that every plan is a blocking plan. Because all plans are blocking and return as many documents as possible, multiplanning will hit "max works" instead of EOF or numToReturn. - This maximizes the overhead of multiplanning on both classic and SBE. + This maximizes the overhead of multiplanning. We expect classic to have better latency and throughput than SBE on this workload, and we expect the combination of classic planner + SBE execution (PM-3591) to perform about From 3966b5549c10852ae92446830abbe0952f972eee Mon Sep 17 00:00:00 2001 From: David Percy Date: Fri, 10 May 2024 13:54:18 -0400 Subject: [PATCH 04/25] typo "prepence" -> presence Co-authored-by: David Storch --- src/workloads/query/multiplanner/UseClusteredIndex.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/workloads/query/multiplanner/UseClusteredIndex.yml b/src/workloads/query/multiplanner/UseClusteredIndex.yml index 20cbe9ea7..2c26fdf53 100644 --- a/src/workloads/query/multiplanner/UseClusteredIndex.yml +++ b/src/workloads/query/multiplanner/UseClusteredIndex.yml @@ -1,7 +1,7 @@ SchemaVersion: 2018-07-01 Owner: "@mongodb/query" Description: | - The goal of this test is to exercise multiplanning in the prepence of clustered indexes. We + The goal of this test is to exercise multiplanning in the presence of clustered indexes. We create as many indexes as possible, and run a query that makes all of them eligible, so we get as many competing plans as possible. The collection is clustered and has very large strings as _id; also, one of the predicates is on _id which means a clustered collection scan is included From 608dbb063495a4e57ae2dbe0067e0ad399486605 Mon Sep 17 00:00:00 2001 From: David Percy Date: Fri, 10 May 2024 13:54:34 -0400 Subject: [PATCH 05/25] don't split on "choice of multiplanner" Co-authored-by: David Storch --- src/workloads/query/multiplanner/VariedSelectivity.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/workloads/query/multiplanner/VariedSelectivity.yml b/src/workloads/query/multiplanner/VariedSelectivity.yml index 5e1c743c6..9ca89a925 100644 --- a/src/workloads/query/multiplanner/VariedSelectivity.yml +++ b/src/workloads/query/multiplanner/VariedSelectivity.yml @@ -6,7 +6,7 @@ Description: | When a query runs with Classic multiplanner + Classic execution, then when multiplanning finishes the query can resume running and reuse the partial results it gathered during multiplanning. By - contrast when running with SBE execution (regardless of choice of multiplanner), the query has + contrast when running with SBE execution, the query has to start over--unless it already finished during multiplanning. This means SBE has a discontinuity in performance as the size of the result set grows: when it crosses from 100 to 102 documents, it has to recompute those first ~100 documents. From 46a82aaafe87baee878d76114e332618164e6bee Mon Sep 17 00:00:00 2001 From: David Percy Date: Fri, 10 May 2024 13:54:48 -0400 Subject: [PATCH 06/25] collectionSize one word Co-authored-by: David Storch --- src/workloads/query/multiplanner/VariedSelectivity.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/workloads/query/multiplanner/VariedSelectivity.yml b/src/workloads/query/multiplanner/VariedSelectivity.yml index 9ca89a925..156e5f7ff 100644 --- a/src/workloads/query/multiplanner/VariedSelectivity.yml +++ b/src/workloads/query/multiplanner/VariedSelectivity.yml @@ -13,7 +13,7 @@ Description: | To measure this, we run the same query 7 times, each one with a different selectivity value. For example, in phase 'MultiplannerWith50ExpectedResults' we choose a selectivity of - '50 / collection size' to make the query return (approximately) 50 documents. + '50 / collectionSize' to make the query return (approximately) 50 documents. Unlike many of the other multiplanner/ workloads, we only test with 2 indexes here, because 2 indexes is a worst case for throwing away results. Having more indexes increases planning From bb2a774bb50f6f0e4bc7e08e41a967bb7128d616 Mon Sep 17 00:00:00 2001 From: David Percy Date: Fri, 10 May 2024 18:22:14 +0000 Subject: [PATCH 07/25] state explicitly ClusteredCollection doesn't do a clustered scan --- src/workloads/query/multiplanner/ClusteredCollection.yml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/workloads/query/multiplanner/ClusteredCollection.yml b/src/workloads/query/multiplanner/ClusteredCollection.yml index 1a265e138..fe988ef8b 100644 --- a/src/workloads/query/multiplanner/ClusteredCollection.yml +++ b/src/workloads/query/multiplanner/ClusteredCollection.yml @@ -6,8 +6,9 @@ Description: | as many competing plans as possible. The collection is clustered and has very large strings as _id. - This workload is similar to 'Simple.yml' except for the collection being clustered. - Maybe we expect the larger record IDs to make fetching more expensive. + This workload is similar to 'Simple.yml' except that the collection is clustered. None of the + competing plans actually take advantage of the clustering (there is no bounded collection scan + plan). Maybe we expect the larger record IDs to make fetch take more wall clock time. GlobalDefaults: dbname: &db test From dc9b3af81b117a8fb484cf23be71821649d6ad16 Mon Sep 17 00:00:00 2001 From: David Percy Date: Fri, 10 May 2024 19:32:52 +0000 Subject: [PATCH 08/25] other -> remaining --- src/workloads/query/multiplanner/MultiplannerWithGroup.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/workloads/query/multiplanner/MultiplannerWithGroup.yml b/src/workloads/query/multiplanner/MultiplannerWithGroup.yml index abd710680..65328c4c7 100644 --- a/src/workloads/query/multiplanner/MultiplannerWithGroup.yml +++ b/src/workloads/query/multiplanner/MultiplannerWithGroup.yml @@ -14,7 +14,7 @@ Description: | The Classic multiplanner with SBE execution would normally be able to avoid starting over, when the query finishes during multiplanning, but in this test it can't because of the $group. When there are any pipeline stages beyond the access-path part of the query, then when - multiplanning finishes we construct a new SBE plan with both the access path and the other + multiplanning finishes we construct a new SBE plan with both the access path and the remaining pipeline stages. GlobalDefaults: From 70fd686bbbc05d955defaf3a769f4872147b3391 Mon Sep 17 00:00:00 2001 From: David Percy Date: Fri, 10 May 2024 19:36:02 +0000 Subject: [PATCH 09/25] however, however however; however. --- src/workloads/query/multiplanner/NoGoodPlan.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/workloads/query/multiplanner/NoGoodPlan.yml b/src/workloads/query/multiplanner/NoGoodPlan.yml index ee73f8563..29ed636ad 100644 --- a/src/workloads/query/multiplanner/NoGoodPlan.yml +++ b/src/workloads/query/multiplanner/NoGoodPlan.yml @@ -4,10 +4,10 @@ Description: | The goal of this test is to exercise the case in multiplanning where all competing plans are bad. As in 'Simple.yml' we create 64 indexes and run a query that makes all of them eligible, so we - get as many competing plans as possible. However, unlike 'Simple.yml', in this workload the only - selective field is unindexed, however, meaning no index will be effective in planning. By - ensuring all plans are relatively equally bad, we are likely to hit the works limit sooner than - the 101 results limit. + get as many competing plans as possible. But unlike 'Simple.yml', in this workload the only + selective field is unindexed, meaning no index will be effective in planning. By ensuring all + plans are relatively equally bad, we are likely to hit the works limit sooner than the 101 + results limit. This case is intended not so much to show a difference between Classic and SBE, but to show a case where any multiplanner will struggle. From 9953a7b3ac3728438b036e37649eb139ad88f1c8 Mon Sep 17 00:00:00 2001 From: David Percy Date: Fri, 10 May 2024 19:36:46 +0000 Subject: [PATCH 10/25] equally bad --- src/workloads/query/multiplanner/NoGoodPlan.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/workloads/query/multiplanner/NoGoodPlan.yml b/src/workloads/query/multiplanner/NoGoodPlan.yml index 29ed636ad..76ddaf1a8 100644 --- a/src/workloads/query/multiplanner/NoGoodPlan.yml +++ b/src/workloads/query/multiplanner/NoGoodPlan.yml @@ -6,8 +6,7 @@ Description: | As in 'Simple.yml' we create 64 indexes and run a query that makes all of them eligible, so we get as many competing plans as possible. But unlike 'Simple.yml', in this workload the only selective field is unindexed, meaning no index will be effective in planning. By ensuring all - plans are relatively equally bad, we are likely to hit the works limit sooner than the 101 - results limit. + plans are equally bad, we are likely to hit the works limit sooner than the 101 results limit. This case is intended not so much to show a difference between Classic and SBE, but to show a case where any multiplanner will struggle. From d691dd89a4c9c80a0f6d063d2c3d271817c31ff5 Mon Sep 17 00:00:00 2001 From: David Percy Date: Mon, 17 Jun 2024 17:33:51 +0000 Subject: [PATCH 11/25] fix bad merge --- .../query/multiplanner/CompoundIndexes.yml | 22 ------------------- 1 file changed, 22 deletions(-) diff --git a/src/workloads/query/multiplanner/CompoundIndexes.yml b/src/workloads/query/multiplanner/CompoundIndexes.yml index 694690c82..956fd82f9 100644 --- a/src/workloads/query/multiplanner/CompoundIndexes.yml +++ b/src/workloads/query/multiplanner/CompoundIndexes.yml @@ -402,27 +402,6 @@ Actors: query: *query AutoRun: -<<<<<<< HEAD -- When: - mongodb_setup: - $eq: - - standalone-sbe - - standalone-80-feature-flags # At time of writing this will enable PM-3591. - - standalone-all-feature-flags # At time of writing this will enable PM-3591. - - standalone-classic-query-engine - branch_name: - $gte: v7.3 -||||||| 9aedad60 -- When: - mongodb_setup: - $eq: - - standalone-sbe - - standalone-80-feature-flags # At time of writing this will enable PM-3591. - - standalone-all-feature-flags # At time of writing this will enable PM-3591. - - standalone-classic-query-engine - branch_name: - $gte: v7.3 -======= - When: mongodb_setup: $eq: @@ -432,4 +411,3 @@ AutoRun: - standalone-classic-query-engine branch_name: $gte: v7.3 ->>>>>>> master From e12b6ebd0e75344cdeb5ed8516f126fdf25baf7d Mon Sep 17 00:00:00 2001 From: David Percy Date: Tue, 4 Jun 2024 17:43:16 +0000 Subject: [PATCH 12/25] remove "We expect ..." comment about SBE multiplanner --- src/workloads/query/multiplanner/BlockingSort.yml | 4 ---- src/workloads/query/multiplanner/ManyIndexSeeks.yml | 4 ---- src/workloads/query/multiplanner/MultikeyIndexes.yml | 4 ---- .../query/multiplanner/MultiplannerWithGroup.yml | 10 ---------- src/workloads/query/multiplanner/NoResults.yml | 6 ++++-- src/workloads/query/multiplanner/NoSuchField.yml | 2 +- .../query/multiplanner/NonBlockingVsBlocking.yml | 3 --- src/workloads/query/multiplanner/Simple.yml | 4 ---- src/workloads/query/multiplanner/Subplanning.yml | 4 ---- 9 files changed, 5 insertions(+), 36 deletions(-) diff --git a/src/workloads/query/multiplanner/BlockingSort.yml b/src/workloads/query/multiplanner/BlockingSort.yml index 1273ac456..67320aa61 100644 --- a/src/workloads/query/multiplanner/BlockingSort.yml +++ b/src/workloads/query/multiplanner/BlockingSort.yml @@ -8,10 +8,6 @@ Description: | documents as possible, multiplanning will hit "max works" instead of EOF or numToReturn. This maximizes the overhead of multiplanning. - We expect classic to have better latency and throughput than SBE on this workload, - and we expect the combination of classic planner + SBE execution (PM-3591) to perform about - as well as classic. - GlobalDefaults: dbname: &db test # Collection name used for queries. diff --git a/src/workloads/query/multiplanner/ManyIndexSeeks.yml b/src/workloads/query/multiplanner/ManyIndexSeeks.yml index f9246c630..b7255598d 100644 --- a/src/workloads/query/multiplanner/ManyIndexSeeks.yml +++ b/src/workloads/query/multiplanner/ManyIndexSeeks.yml @@ -8,10 +8,6 @@ Description: | leads to many index seeks on the less effective indices (..., x1). Because every time we hit a non- matching field we seek again, and the scan ends when we reach a non-matching x1. - We expect classic to have better latency and throughput than SBE on this workload, - and we expect the combination of classic planner + SBE execution (PM-3591) to perform about - as well as classic. - GlobalDefaults: dbname: &db test # Collection name used for queries. diff --git a/src/workloads/query/multiplanner/MultikeyIndexes.yml b/src/workloads/query/multiplanner/MultikeyIndexes.yml index a1c8a42a0..2e0f2be76 100644 --- a/src/workloads/query/multiplanner/MultikeyIndexes.yml +++ b/src/workloads/query/multiplanner/MultikeyIndexes.yml @@ -7,10 +7,6 @@ Description: | multi-planner will behave more optimally than the SBE multiplanner because it will cut off execution when the one good plan reaches the end. - We expect classic to have better latency and throughput than SBE on this workload, - and we expect the combination of classic planner + SBE execution (PM-3591) to perform about - as well as classic. - GlobalDefaults: dbname: &db test # Collection name used for queries. diff --git a/src/workloads/query/multiplanner/MultiplannerWithGroup.yml b/src/workloads/query/multiplanner/MultiplannerWithGroup.yml index 65328c4c7..eb659fb24 100644 --- a/src/workloads/query/multiplanner/MultiplannerWithGroup.yml +++ b/src/workloads/query/multiplanner/MultiplannerWithGroup.yml @@ -7,16 +7,6 @@ Description: | selective, which means only one of those plans is efficient. Where this test departs from 'Simple.yml' is by adding a $group stage after the access-path part of the query. - We expect the Classic multiplanner with Classic execution to reach EOF during planning, and - then feed those documents into the $group stage. By contrast the SBE multiplanner will not - reuse results gathered during multiplanning. - - The Classic multiplanner with SBE execution would normally be able to avoid starting over, - when the query finishes during multiplanning, but in this test it can't because of the $group. - When there are any pipeline stages beyond the access-path part of the query, then when - multiplanning finishes we construct a new SBE plan with both the access path and the remaining - pipeline stages. - GlobalDefaults: dbname: &db test # Collection name used for queries. diff --git a/src/workloads/query/multiplanner/NoResults.yml b/src/workloads/query/multiplanner/NoResults.yml index 03ba81ffb..6e9c26589 100644 --- a/src/workloads/query/multiplanner/NoResults.yml +++ b/src/workloads/query/multiplanner/NoResults.yml @@ -1,8 +1,10 @@ SchemaVersion: 2018-07-01 Owner: "@mongodb/query" Description: | - This test shows an example where we have many competing plans, but they all are very efficient: - they all have empty index bounds, which means any competing plan will finish immediately. + The goal of this test is to exercise multiplanning. We create as many indexes as possible, and run a + query that makes all of them eligible, so we get as many competing plans as possible. All predicates + are very selective (match 0% of the documents). With zero results, we do no hit the EOF optimization + and all competing plans hit the works limit instead of document limit. GlobalDefaults: dbname: &db test diff --git a/src/workloads/query/multiplanner/NoSuchField.yml b/src/workloads/query/multiplanner/NoSuchField.yml index cc19b15dd..0e0b7d999 100644 --- a/src/workloads/query/multiplanner/NoSuchField.yml +++ b/src/workloads/query/multiplanner/NoSuchField.yml @@ -6,7 +6,7 @@ Description: | false on this dataset. All of the other predicates match all the data, meaning none of the indexed predicates are selective. This guarantees that the query will not be able to finish multi-planning by producing enough documents, so instead we will hit getTrialPeriodMaxWorks(). - +@ This also covers the special case in which the trial period hits max works without any of the candidate plans producing any documents. This is known to be a troublesome scenario for the multi-planner for a few reasons: diff --git a/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml b/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml index 5542ec22d..3b5c6ebf8 100644 --- a/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml +++ b/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml @@ -16,9 +16,6 @@ Description: | Another point of view: this scenario shows that "non-blocking" plans can still do an unbounded amount of work per getNext(). - We expect classic to have better latency and throughput than SBE on this workload, and we expect - the combination of classic planner + SBE execution (PM-3591) to perform about as well as classic. - GlobalDefaults: dbname: &db test # Collection name used for queries. diff --git a/src/workloads/query/multiplanner/Simple.yml b/src/workloads/query/multiplanner/Simple.yml index fdf0e4546..3d1be0bfa 100644 --- a/src/workloads/query/multiplanner/Simple.yml +++ b/src/workloads/query/multiplanner/Simple.yml @@ -9,10 +9,6 @@ Description: | has to run the list of plans sequentially, which means we can't short-circuit when the shortest-running plan finishes. - We expect classic to have better latency and throughput than SBE on this workload, - and we expect the combination of classic planner + SBE execution (PM-3591) to perform about - as well as classic. - GlobalDefaults: dbname: &db test # Collection name used for queries. diff --git a/src/workloads/query/multiplanner/Subplanning.yml b/src/workloads/query/multiplanner/Subplanning.yml index 4671b0aea..fb73d3ad3 100644 --- a/src/workloads/query/multiplanner/Subplanning.yml +++ b/src/workloads/query/multiplanner/Subplanning.yml @@ -7,10 +7,6 @@ Description: | The workload uses an $or query with 8 clauses each containing 8 predicates. Each branch have only one selective predicate. - We expect classic to have better latency and throughput than SBE on this workload, - and we expect the combination of classic planner + SBE execution (PM-3591) to perform about - as well as classic. - GlobalDefaults: dbname: &db test # Collection name used for queries. From 2b0b7bb19af8f5ed53269137ee8479a46758e757 Mon Sep 17 00:00:00 2001 From: David Percy Date: Tue, 4 Jun 2024 18:26:01 +0000 Subject: [PATCH 13/25] avoid "empty data" wording --- .../query/multiplanner/NonBlockingVsBlocking.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml b/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml index 3b5c6ebf8..59f9ff012 100644 --- a/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml +++ b/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml @@ -4,10 +4,10 @@ Description: | The goal of this test is to exercise multiplanning when both blocking and non-blocking plans are available. - If the selectivity value is small enough (less than 0.5), the optimal plan is to employ a - blocking plan by scanning a segment of empty data and conducting a blocking-sort operation, - whereas the other plans' index provides the right sort order, but requires a full scan, and - every document is rejected after the FETCH stage. + If the selectivity value is small enough (less than 0.5), the optimal plan is to scan a narrow + range of an index and then blocking sort. An alternative, suboptimal query plan does a full scan + of an index that provides the right sort order, and requires fetching each document before running + a residual predicate. Because the SBE multiplanner can't round-robin, it has a heuristic "try nonblocking plans first". This scenario is a worst case for that heuristic, because we'll try the best plan last. From c9d3933952d6c92be5470d902caa2e7537369769 Mon Sep 17 00:00:00 2001 From: David Percy Date: Mon, 17 Jun 2024 19:22:59 +0000 Subject: [PATCH 14/25] typo --- src/workloads/query/multiplanner/NoSuchField.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/workloads/query/multiplanner/NoSuchField.yml b/src/workloads/query/multiplanner/NoSuchField.yml index 0e0b7d999..cc19b15dd 100644 --- a/src/workloads/query/multiplanner/NoSuchField.yml +++ b/src/workloads/query/multiplanner/NoSuchField.yml @@ -6,7 +6,7 @@ Description: | false on this dataset. All of the other predicates match all the data, meaning none of the indexed predicates are selective. This guarantees that the query will not be able to finish multi-planning by producing enough documents, so instead we will hit getTrialPeriodMaxWorks(). -@ + This also covers the special case in which the trial period hits max works without any of the candidate plans producing any documents. This is known to be a troublesome scenario for the multi-planner for a few reasons: From d0b801dd5a6b2bb73a5c6f9199a796a1e7d6272c Mon Sep 17 00:00:00 2001 From: David Percy Date: Mon, 17 Jun 2024 19:25:55 +0000 Subject: [PATCH 15/25] update docs --- docs/generated/workloads.md | 126 ++++++++++++++++-------------------- 1 file changed, 56 insertions(+), 70 deletions(-) diff --git a/docs/generated/workloads.md b/docs/generated/workloads.md index 2c0e0eba6..eb29568c9 100644 --- a/docs/generated/workloads.md +++ b/docs/generated/workloads.md @@ -4214,15 +4214,12 @@ https://docs.mongodb.com/manual/reference/operator/aggregation/setWindowFields/ ### Description -The goal of this test is to exercise multiplanning. We create as many indexes as possible, and run -a query that makes all of them eligible, so we get as many competing plans as possible. We also -add a sort stage on an unindexed field, ensuring that every plan is a blocking plan. Because all -plans are blocking and return as many documents as possible, multiplanning will hit "max works" -instead of EOF of numToReturn. This maximizes the overhead of multiplanning on both classic and SBE. - -We expect classic to have better latency and throughput than SBE on this workload, -and we expect the combination of classic planner + SBE execution (PM-3591) to perform about -as well as classic. +The goal of this test is to show how a blocking sort can increase the overhead of multiplanning. +We create as many indexes as possible, and run a query that makes all of them eligible, so we +get as many competing plans as possible. We also add a sort stage on an unindexed field, +ensuring that every plan is a blocking plan. Because all plans are blocking and return as many +documents as possible, multiplanning will hit "max works" instead of EOF or numToReturn. +This maximizes the overhead of multiplanning. @@ -4233,13 +4230,14 @@ as well as classic. ### Description -The goal of this test is to exercise multiplanning. We create as many indexes as possible, and run a - query that makes all of them eligible, so we get as many competing plans as possible. Here, we do this on a - clustered collection that has very large strings as _id. +The goal of this test is to exercise multiplanning in the presence of clustered indexes. We +create as many indexes as possible, and run a query that makes all of them eligible, so we get +as many competing plans as possible. The collection is clustered and has very large strings as +_id. - We expect classic to have better latency and throughput than SBE on this workload, - and we expect the combination of classic planner + SBE execution (PM-3591) to perform about - as well as classic. +This workload is similar to 'Simple.yml' except that the collection is clustered. None of the +competing plans actually take advantage of the clustering (there is no bounded collection scan +plan). Maybe we expect the larger record IDs to make fetch take more wall clock time. @@ -4278,10 +4276,6 @@ having predicates on all the fields, while only the predicate on field x1 has se leads to many index seeks on the less effective indices (..., x1). Because every time we hit a non- matching field we seek again, and the scan ends when we reach a non-matching x1. -We expect classic to have better latency and throughput than SBE on this workload, -and we expect the combination of classic planner + SBE execution (PM-3591) to perform about -as well as classic. - ## [MultiPlanningReadsALotOfData](https://www.github.com/mongodb/genny/blob/master/src/workloads/query/multiplanner/MultiPlanningReadsALotOfData.yml) @@ -4327,10 +4321,6 @@ an IXSCAN of a multikey index has to deduplicate RIDs, a lot of space will be us multi-planner will behave more optimally than the SBE multiplanner because it will cut off execution when the one good plan reaches the end. -We expect classic to have better latency and throughput than SBE on this workload, -and we expect the combination of classic planner + SBE execution (PM-3591) to perform about -as well as classic. - ## [MultiplannerWithGroup](https://www.github.com/mongodb/genny/blob/master/src/workloads/query/multiplanner/MultiplannerWithGroup.yml) @@ -4340,15 +4330,11 @@ as well as classic. ### Description -The goal of this test is to exercise multiplanning. We create as many indexes as possible, and run a -query that makes all of them eligible, so we get as many competing plans as possible. We add a -group stage, which is blocking. The SBE multiplanner will multiplan group as it is a part of the -canonical query, but the classic multiplanner will not plan. This means the SBE multiplanner will -have the overhead of trial running blocking plans when compared to the classic multiplanner. - -We expect classic to have better latency and throughput than SBE on this workload, -and we expect the combination of classic planner + SBE execution (PM-3591) to perform about -as well as classic. +This test was created to show how three different multiplanners handle $group. +The query is essentially the one from 'Simple.yml': we have as many indexed predicates as +possible, to create as many indexed plans as possible, but only one of those predicates is +selective, which means only one of those plans is efficient. Where this test departs from +'Simple.yml' is by adding a $group stage after the access-path part of the query. @@ -4364,10 +4350,6 @@ query that makes all of them eligible, so we get as many competing plans as poss are very selective (match 0% of the documents). With zero results, we do no hit the EOF optimization and all competing plans hit the works limit instead of document limit. -We expect classic to have better latency and throughput than SBE on this workload, -and we expect the combination of classic planner + SBE execution (PM-3591) to perform about -as well as classic. - ## [NoSuchField](https://www.github.com/mongodb/genny/blob/master/src/workloads/query/multiplanner/NoSuchField.yml) @@ -4400,16 +4382,20 @@ multi-planner for a few reasons: ### Description -The goal of this test is to exercise multiplanning. If the selectivity value is small enough (less -than 0.5), the optimal plan is to employ a blocking plan by scanning a segment of empty data and -conducting a blocking-sort operation, whereas the other plans' index provides the right sort -order, but requires a full scan, and every document is rejected after the FETCH stage. Because the -SBE multiplanner can't round-robin, it has a heuristic "try nonblocking plans first". This -scenario is a worst case for that heuristic, because we'll try the best plan last. Otherwise, an -IXSCAN and FETCH non-blocking plan will be used. +The goal of this test is to exercise multiplanning when both blocking and non-blocking plans are +available. -We expect classic to have better latency and throughput than SBE on this workload, and we expect -the combination of classic planner + SBE execution (PM-3591) to perform about as well as classic. +If the selectivity value is small enough (less than 0.5), the optimal plan is to scan a narrow +range of an index and then blocking sort. An alternative, suboptimal query plan does a full scan +of an index that provides the right sort order, and requires fetching each document before running +a residual predicate. + +Because the SBE multiplanner can't round-robin, it has a heuristic "try nonblocking plans first". +This scenario is a worst case for that heuristic, because we'll try the best plan last. +Otherwise, an IXSCAN and FETCH non-blocking plan will be used. + +Another point of view: this scenario shows that "non-blocking" plans can still do an unbounded +amount of work per getNext(). @@ -4423,15 +4409,11 @@ the combination of classic planner + SBE execution (PM-3591) to perform about as The goal of this test is to exercise multiplanning. We create as many indexes as possible, and run a query that makes all of them eligible, so we get as many competing plans as possible. -The original goal of this test was to demonstrate weaknesses of the SBE multiplanner when compared to +This test was originally created to demonstrate weaknesses of the SBE multiplanner when compared to the classic multiplanner. Mainly, the SBE multiplanner can't round-robin between plans, which means it has to run the list of plans sequentially, which means we can't short-circuit when the shortest-running plan finishes. -We expect classic to have better latency and throughput than SBE on this workload, -and we expect the combination of classic planner + SBE execution (PM-3591) to perform about -as well as classic. - ## [Subplanning](https://www.github.com/mongodb/genny/blob/master/src/workloads/query/multiplanner/Subplanning.yml) @@ -4447,10 +4429,6 @@ query that makes all of them eligible, so we get as many competing plans as poss The workload uses an $or query with 8 clauses each containing 8 predicates. Each branch have only one selective predicate. -We expect classic to have better latency and throughput than SBE on this workload, -and we expect the combination of classic planner + SBE execution (PM-3591) to perform about -as well as classic. - ## [UseClusteredIndex](https://www.github.com/mongodb/genny/blob/master/src/workloads/query/multiplanner/UseClusteredIndex.yml) @@ -4460,13 +4438,14 @@ as well as classic. ### Description -The goal of this test is to exercise multiplanning. We create as many indexes as possible, and run a - query that makes all of them eligible, so we get as many competing plans as possible. Here, we do this on a - clustered collection and add a selective predicate on _id, so that the clustered index is a viable candidate plan. +The goal of this test is to exercise multiplanning in the presence of clustered indexes. We +create as many indexes as possible, and run a query that makes all of them eligible, so we get +as many competing plans as possible. The collection is clustered and has very large strings as +_id; also, one of the predicates is on _id which means a clustered collection scan is included +in the competing plans. - We expect classic to have better latency and throughput than SBE on this workload, - and we expect the combination of classic planner + SBE execution (PM-3591) to perform about - as well as classic. +This workload is similar to 'Simple.yml' except for the collection being clustered, and the +extra predicate. @@ -4477,17 +4456,24 @@ The goal of this test is to exercise multiplanning. We create as many indexes as ### Description -The goal of this test is to exercise multiplanning. We run the same query 7 times, each one with a -different selectivity value that we are comparing against x1, calcuated based on the number of -documents we want the query to match. This will help us measure the overhead of throwing out the -result set gathered during multi-planning when the result set exceeds 101 documents. Unlike many -of the other multiplanner/ workloads, we only test with 2 indexes here, because 2 indexes is a -worst case for throwing away results. Having more indexes increases planning time, but not query -execution time, so having more indexes makes the *relative* cost of throwing away results smaller. +The goal of this test is to measure the overhead of "throwing out" the initial results returned by +multiplanning. + +When a query runs with Classic multiplanner + Classic execution, then when multiplanning finishes +the query can resume running and reuse the partial results it gathered during multiplanning. By +contrast when running with SBE execution, the query has +to start over--unless it already finished during multiplanning. This means SBE has a +discontinuity in performance as the size of the result set grows: when it crosses from 100 to +102 documents, it has to recompute those first ~100 documents. + +To measure this, we run the same query 7 times, each one with a different selectivity value. +For example, in phase 'MultiplannerWith50ExpectedResults' we choose a selectivity of +'50 / collectionSize' to make the query return (approximately) 50 documents. -We expect classic to have better latency and throughput than SBE on this workload, -and we expect the combination of classic planner + SBE execution (PM-3591) to perform about -as well as classic. +Unlike many of the other multiplanner/ workloads, we only test with 2 indexes here, because +2 indexes is a worst case for throwing away results. Having more indexes increases planning +time, but not query execution time, so having more indexes makes the *relative* cost of +throwing away results smaller. From 69cbdc652d0a9a30286de3fd77746efdb59932c3 Mon Sep 17 00:00:00 2001 From: David Percy Date: Mon, 17 Jun 2024 21:23:25 +0000 Subject: [PATCH 16/25] rephrase SBE multiplanner as "historical" --- .../query/multiplanner/NonBlockingVsBlocking.yml | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml b/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml index 59f9ff012..098e8ae1b 100644 --- a/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml +++ b/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml @@ -9,12 +9,10 @@ Description: | of an index that provides the right sort order, and requires fetching each document before running a residual predicate. - Because the SBE multiplanner can't round-robin, it has a heuristic "try nonblocking plans first". - This scenario is a worst case for that heuristic, because we'll try the best plan last. - Otherwise, an IXSCAN and FETCH non-blocking plan will be used. - - Another point of view: this scenario shows that "non-blocking" plans can still do an unbounded - amount of work per getNext(). + This case shows that it's important for the multiplanner to round-robin the execution of the candidate + plans. Historically (in an alternative multiplanner based on SBE execution) we have tried a simpler + strategy that runs the candidates sequentially, starting with the nonblocking plans--this heuristic + does not work, because a nonblocking plan can still do an unbounded amount of work per getNext(). GlobalDefaults: dbname: &db test From 8524decf20e41fd4105284f2460fe627c686ce73 Mon Sep 17 00:00:00 2001 From: David Percy Date: Mon, 17 Jun 2024 21:34:49 +0000 Subject: [PATCH 17/25] note about residual selectivity --- src/workloads/query/multiplanner/NonBlockingVsBlocking.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml b/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml index 098e8ae1b..22f8fe758 100644 --- a/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml +++ b/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml @@ -7,7 +7,8 @@ Description: | If the selectivity value is small enough (less than 0.5), the optimal plan is to scan a narrow range of an index and then blocking sort. An alternative, suboptimal query plan does a full scan of an index that provides the right sort order, and requires fetching each document before running - a residual predicate. + a very selective residual predicate: this means each getNext() has to scan many index entries + (1/selectivity on average). This case shows that it's important for the multiplanner to round-robin the execution of the candidate plans. Historically (in an alternative multiplanner based on SBE execution) we have tried a simpler From d61a02b371151272e471cfe39d6ca5f7b74f4162 Mon Sep 17 00:00:00 2001 From: David Percy Date: Mon, 17 Jun 2024 21:35:54 +0000 Subject: [PATCH 18/25] update docs --- docs/generated/workloads.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/docs/generated/workloads.md b/docs/generated/workloads.md index eb29568c9..e00977802 100644 --- a/docs/generated/workloads.md +++ b/docs/generated/workloads.md @@ -4388,14 +4388,13 @@ available. If the selectivity value is small enough (less than 0.5), the optimal plan is to scan a narrow range of an index and then blocking sort. An alternative, suboptimal query plan does a full scan of an index that provides the right sort order, and requires fetching each document before running -a residual predicate. +a very selective residual predicate: this means each getNext() has to scan many index entries +(1/selectivity on average). -Because the SBE multiplanner can't round-robin, it has a heuristic "try nonblocking plans first". -This scenario is a worst case for that heuristic, because we'll try the best plan last. -Otherwise, an IXSCAN and FETCH non-blocking plan will be used. - -Another point of view: this scenario shows that "non-blocking" plans can still do an unbounded -amount of work per getNext(). +This case shows that it's important for the multiplanner to round-robin the execution of the candidate +plans. Historically (in an alternative multiplanner based on SBE execution) we have tried a simpler +strategy that runs the candidates sequentially, starting with the nonblocking plans--this heuristic +does not work, because a nonblocking plan can still do an unbounded amount of work per getNext(). From e52494b489873e32fe16fa67cbad683d22d6cb1e Mon Sep 17 00:00:00 2001 From: David Percy Date: Mon, 17 Jun 2024 21:51:39 +0000 Subject: [PATCH 19/25] appease linter by adding keyword to unchanged file --- src/workloads/query/multiplanner/NoSuchField.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/workloads/query/multiplanner/NoSuchField.yml b/src/workloads/query/multiplanner/NoSuchField.yml index cc19b15dd..fc486854d 100644 --- a/src/workloads/query/multiplanner/NoSuchField.yml +++ b/src/workloads/query/multiplanner/NoSuchField.yml @@ -14,6 +14,8 @@ Description: | candidate plans and none of them produce any results. 2) When there are zero results, each plan has a productivity ratio of zero. This makes ties likely during plan ranking, which can in turn lead to an incorrect plan choice. +Keywords: + - multiplanner GlobalDefaults: dbname: &db test From d5c7d7624bd263a0f39e35e2b9bac185e6a00c0c Mon Sep 17 00:00:00 2001 From: David Percy Date: Tue, 18 Jun 2024 15:28:56 +0000 Subject: [PATCH 20/25] dont compare --- src/workloads/query/multiplanner/MultikeyIndexes.yml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/workloads/query/multiplanner/MultikeyIndexes.yml b/src/workloads/query/multiplanner/MultikeyIndexes.yml index 2e0f2be76..cd5921339 100644 --- a/src/workloads/query/multiplanner/MultikeyIndexes.yml +++ b/src/workloads/query/multiplanner/MultikeyIndexes.yml @@ -3,9 +3,7 @@ Owner: "@mongodb/query" Description: | The goal of this test is to exercise multiplanning with multikey indexes. We create many indexes and run a query that makes all of them eligible, so we get as many competing plans as possible. Because - an IXSCAN of a multikey index has to deduplicate RIDs, a lot of space will be used. The classic - multi-planner will behave more optimally than the SBE multiplanner because it will cut off execution - when the one good plan reaches the end. + an IXSCAN of a multikey index has to deduplicate RIDs, a lot of space will be used. GlobalDefaults: dbname: &db test From 1830416a062f9beefd9ae162a79f8b093119aa1f Mon Sep 17 00:00:00 2001 From: David Percy Date: Tue, 18 Jun 2024 15:29:29 +0000 Subject: [PATCH 21/25] the multiplanner handles group --- src/workloads/query/multiplanner/MultiplannerWithGroup.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/workloads/query/multiplanner/MultiplannerWithGroup.yml b/src/workloads/query/multiplanner/MultiplannerWithGroup.yml index eb659fb24..5b740e68b 100644 --- a/src/workloads/query/multiplanner/MultiplannerWithGroup.yml +++ b/src/workloads/query/multiplanner/MultiplannerWithGroup.yml @@ -1,7 +1,7 @@ SchemaVersion: 2018-07-01 Owner: "@mongodb/query" Description: | - This test was created to show how three different multiplanners handle $group. + This test was created to show how the multiplanner handles $group. The query is essentially the one from 'Simple.yml': we have as many indexed predicates as possible, to create as many indexed plans as possible, but only one of those predicates is selective, which means only one of those plans is efficient. Where this test departs from From 42a536b5da16659b94681a82c2c8d49e036b116f Mon Sep 17 00:00:00 2001 From: David Percy Date: Tue, 18 Jun 2024 15:49:42 +0000 Subject: [PATCH 22/25] empty bounds --- src/workloads/query/multiplanner/NoResults.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/workloads/query/multiplanner/NoResults.yml b/src/workloads/query/multiplanner/NoResults.yml index 6e9c26589..43b43b5ed 100644 --- a/src/workloads/query/multiplanner/NoResults.yml +++ b/src/workloads/query/multiplanner/NoResults.yml @@ -2,9 +2,9 @@ SchemaVersion: 2018-07-01 Owner: "@mongodb/query" Description: | The goal of this test is to exercise multiplanning. We create as many indexes as possible, and run a - query that makes all of them eligible, so we get as many competing plans as possible. All predicates - are very selective (match 0% of the documents). With zero results, we do no hit the EOF optimization - and all competing plans hit the works limit instead of document limit. + query that makes all of them eligible, so we get as many competing plans as possible. However, all + the indexed predicates are very selective (match 0% of the documents). This should result in empty + index bounds, so multiplanning should finish immediately. GlobalDefaults: dbname: &db test From b5ba2ba3ba4f81aeaa392f3de34ba445a2422721 Mon Sep 17 00:00:00 2001 From: David Percy Date: Tue, 18 Jun 2024 17:11:42 +0000 Subject: [PATCH 23/25] not large strings --- src/workloads/query/multiplanner/UseClusteredIndex.yml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/workloads/query/multiplanner/UseClusteredIndex.yml b/src/workloads/query/multiplanner/UseClusteredIndex.yml index 2ed97ec03..704feb849 100644 --- a/src/workloads/query/multiplanner/UseClusteredIndex.yml +++ b/src/workloads/query/multiplanner/UseClusteredIndex.yml @@ -3,9 +3,8 @@ Owner: "@mongodb/query" Description: | The goal of this test is to exercise multiplanning in the presence of clustered indexes. We create as many indexes as possible, and run a query that makes all of them eligible, so we get - as many competing plans as possible. The collection is clustered and has very large strings as - _id; also, one of the predicates is on _id which means a clustered collection scan is included - in the competing plans. + as many competing plans as possible. The collection is clustered and one of the predicates is + on _id, which means a clustered collection scan is included in the competing plans. This workload is similar to 'Simple.yml' except for the collection being clustered, and the extra predicate. From 539fad1614ec44d4fd6186321cfc510892fe99c2 Mon Sep 17 00:00:00 2001 From: David Percy Date: Tue, 18 Jun 2024 17:17:40 +0000 Subject: [PATCH 24/25] update docs --- docs/generated/workloads.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/generated/workloads.md b/docs/generated/workloads.md index e00977802..52fa73dcd 100644 --- a/docs/generated/workloads.md +++ b/docs/generated/workloads.md @@ -4317,9 +4317,7 @@ indexes ### Description The goal of this test is to exercise multiplanning with multikey indexes. We create many indexes and run a query that makes all of them eligible, so we get as many competing plans as possible. Because -an IXSCAN of a multikey index has to deduplicate RIDs, a lot of space will be used. The classic -multi-planner will behave more optimally than the SBE multiplanner because it will cut off execution -when the one good plan reaches the end. +an IXSCAN of a multikey index has to deduplicate RIDs, a lot of space will be used. @@ -4330,7 +4328,7 @@ when the one good plan reaches the end. ### Description -This test was created to show how three different multiplanners handle $group. +This test was created to show how the multiplanner handles $group. The query is essentially the one from 'Simple.yml': we have as many indexed predicates as possible, to create as many indexed plans as possible, but only one of those predicates is selective, which means only one of those plans is efficient. Where this test departs from @@ -4346,9 +4344,9 @@ selective, which means only one of those plans is efficient. Where this test dep ### Description The goal of this test is to exercise multiplanning. We create as many indexes as possible, and run a -query that makes all of them eligible, so we get as many competing plans as possible. All predicates -are very selective (match 0% of the documents). With zero results, we do no hit the EOF optimization -and all competing plans hit the works limit instead of document limit. +query that makes all of them eligible, so we get as many competing plans as possible. However, all +the indexed predicates are very selective (match 0% of the documents). This should result in empty +index bounds, so multiplanning should finish immediately. @@ -4373,6 +4371,10 @@ multi-planner for a few reasons: 2) When there are zero results, each plan has a productivity ratio of zero. This makes ties likely during plan ranking, which can in turn lead to an incorrect plan choice. + + +### Keywords +multiplanner ## [NonBlockingVsBlocking](https://www.github.com/mongodb/genny/blob/master/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml) @@ -4439,9 +4441,8 @@ only one selective predicate. ### Description The goal of this test is to exercise multiplanning in the presence of clustered indexes. We create as many indexes as possible, and run a query that makes all of them eligible, so we get -as many competing plans as possible. The collection is clustered and has very large strings as -_id; also, one of the predicates is on _id which means a clustered collection scan is included -in the competing plans. +as many competing plans as possible. The collection is clustered and one of the predicates is +on _id, which means a clustered collection scan is included in the competing plans. This workload is similar to 'Simple.yml' except for the collection being clustered, and the extra predicate. From e9a2b5e41892311a72728a00da225e68f3bc09e3 Mon Sep 17 00:00:00 2001 From: David Percy Date: Thu, 20 Jun 2024 15:30:02 +0000 Subject: [PATCH 25/25] trailing spaces --- src/workloads/query/multiplanner/NonBlockingVsBlocking.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml b/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml index 22f8fe758..6a1186296 100644 --- a/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml +++ b/src/workloads/query/multiplanner/NonBlockingVsBlocking.yml @@ -9,7 +9,7 @@ Description: | of an index that provides the right sort order, and requires fetching each document before running a very selective residual predicate: this means each getNext() has to scan many index entries (1/selectivity on average). - + This case shows that it's important for the multiplanner to round-robin the execution of the candidate plans. Historically (in an alternative multiplanner based on SBE execution) we have tried a simpler strategy that runs the candidates sequentially, starting with the nonblocking plans--this heuristic