roachtest: unoptimized-query-oracle/disable-rules=half failed #90761
Assignees
Labels
branch-release-22.2
Used to mark release blockers and technical advisories for 22.2
C-test-failure
Broken test (automatically or manually discovered).
O-roachtest
O-robot
Originated from a bot.
S-0-visible-logical-error
Database stores inconsistent data in some cases, or queries return invalid results silently.
T-sql-queries
SQL Queries Team
Projects
Milestone
Comments
cockroach-teamcity
added
branch-release-22.2
|
Partially reduced: CREATE TABLE table1 (col1_0 TIMESTAMP, col1_1 INTERVAL, col1_2 CHAR NOT NULL);
INSERT INTO table1 (col1_0, col1_1, col1_2)
VALUES ('1991-05-10 14:18:54.000472', '-9 years -2 mons -188 days -21:52:32.812039', '');
SET testing_optimizer_random_seed = 726585762659783634;
SET testing_optimizer_disable_rule_probability = 0.500000;
SELECT 1
FROM table1 AS tab_7919
JOIN table1 AS tab_7920
JOIN table1 AS tab_7921
JOIN table1 AS tab_7922
JOIN table1 AS tab_7923 ON tab_7922.col1_2 = tab_7923.col1_2
LEFT JOIN table1 AS tab_7924 ON tab_7923.col1_2 = tab_7924.col1_2
LEFT JOIN table1 AS tab_7925
JOIN table1 AS tab_7926 ON tab_7925.col1_1 = tab_7926.col1_1
ON tab_7924.col1_1 = tab_7926.col1_1 AND tab_7924.col1_2 > tab_7926.col1_2
ON tab_7921.col1_1 = tab_7926.col1_1 AND tab_7921.col1_1 = tab_7924.col1_1
ON tab_7920.col1_0 = tab_7926.col1_0
AND tab_7920.crdb_internal_mvcc_timestamp = tab_7924.crdb_internal_mvcc_timestamp
ON tab_7919.col1_0 = tab_7920.col1_0
AND tab_7919.crdb_internal_mvcc_timestamp = tab_7921.crdb_internal_mvcc_timestamp
;
RESET testing_optimizer_random_seed;
RESET testing_optimizer_disable_rule_probability;
SELECT 1
FROM table1 AS tab_7919
JOIN table1 AS tab_7920
JOIN table1 AS tab_7921
JOIN table1 AS tab_7922
JOIN table1 AS tab_7923 ON tab_7922.col1_2 = tab_7923.col1_2
LEFT JOIN table1 AS tab_7924 ON tab_7923.col1_2 = tab_7924.col1_2
LEFT JOIN table1 AS tab_7925
JOIN table1 AS tab_7926 ON tab_7925.col1_1 = tab_7926.col1_1
ON tab_7924.col1_1 = tab_7926.col1_1 AND tab_7924.col1_2 > tab_7926.col1_2
ON tab_7921.col1_1 = tab_7926.col1_1 AND tab_7921.col1_1 = tab_7924.col1_1
ON tab_7920.col1_0 = tab_7926.col1_0
AND tab_7920.crdb_internal_mvcc_timestamp = tab_7924.crdb_internal_mvcc_timestamp
ON tab_7919.col1_0 = tab_7920.col1_0
AND tab_7919.crdb_internal_mvcc_timestamp = tab_7921.crdb_internal_mvcc_timestamp
; |
|
Further reduced: CREATE TABLE t (a INT, b INT, c INT);
INSERT INTO t (a, b, c)
VALUES (0, 0, 0);
SET testing_optimizer_random_seed = 726585762659783634;
SET testing_optimizer_disable_rule_probability = 0.500000;
SELECT 1
FROM t AS t1
JOIN t AS t2
LEFT JOIN t AS t3
JOIN t AS t4 ON true
ON t2.b > t4.b
ON t1.a = t4.a AND t1.c = t2.c
;
RESET testing_optimizer_random_seed;
RESET testing_optimizer_disable_rule_probability;
SELECT 1
FROM t AS t1
JOIN t AS t2
LEFT JOIN t AS t3
JOIN t AS t4 ON true
ON t2.b > t4.b
ON t1.a = t4.a AND t1.c = t2.c
; |
|
The symptoms here look similar to #76522. The |
|
Yeah looks similar but with null-rejection. I'll keep digging though. |
This comment was marked as off-topic.
This comment was marked as off-topic.
|
For the most recent failure - need to backport #88399 |
DrewKimball
added
the
S-0-visible-logical-error
|
roachtest.unoptimized-query-oracle/disable-rules=half failed with artifacts on release-22.2 @ 56e0cae2dd496aee737f273f6b0a77faae4f76bd: Parameters: Same failure on other branches
|
|
Stack trace for the most recent failure: |
DrewKimball
added a commit
to DrewKimball/cockroach
that referenced
this issue
Nov 7, 2022
This patch fixes a bug in the join reordering logic that can lead to incorrect results due to a dropped filter and incorrect conversion of a left join to an inner join. The bug can occur when the join tree contains an inner join with a left join as an input, where the inner join has two separate conjuncts in its ON condition that reference both inputs of the left join. Additionally, the inner join filters must not filter NULL values from the right side of the left join (or alternatively null-rejection rules must be disabled). The incorrect transformation looks something like this: ``` (INNER JOIN xy (LEFT JOIN ab (INNER JOIN uv wz ON v = w) ON b = v) ON a = x AND u = x) ``` => ``` (INNER JOIN ab (INNER JOIN xy (INNER JOIN uv wz ON v = w) ON u = x) ON a = x) ``` Notice how `xy` has been "pushed" into the right side of the left join and the left join's `b = v` filter (and the left join itself) dropped in the process. To understand what causes the bug, it is necessary to understand three points about the join reordering algorithm: 1. Cross products are never introduced in the enumerated plans. So, for two sub-plans, a join is only considered between them if there is an applicable edge between those sub-plans. 2. The original paper associates each join with exactly one edge in the hypergraph that encodes "reorderability" properties. 3. The `JoinOrderBuilder` departs from the paper by associating each inner join *conjunct* with a hypergraph edge. This allows each conjunct to be independently reordered from the others. See the `Special handling of inner joins` section in the `JoinOrderBuilder` comment for more details. (1) combined with (2) implies that a reordered join tree is only considered if every edge in the hypergraph could be applied to form joins in the join tree. This allows the original algorithm to prevent invalid orderings by making just a single edge inapplicable. However, because of (3) the same is no longer true for the `JoinOrderBuilder`. In the example given above, the left join fails the applicability check, indicating an invalid plan. However, the inner join's `a = x` filter passes the check and ends up replacing the left join. This prevents the the check in (1) from catching the invalid plan. This patch fixes the bug by keeping track of the edges that *should* be applied somewhere in each join tree based on the TES of each edge. This is then compared against the actual edges that are applied in the construction of the join tree. If the edge sets aren't equal, the plan is invalid and cannot be added to the memo. This allows the `JoinOrderBuilder` to recover the property that an inapplicable edge invalidates an enumerated plan. Fixes cockroachdb#90761 Release note (bug fix): Fixed a bug existing since 20.2 that could cause incorrect results in rare cases for queries with inner joins and left joins. For the bug to occur, the left join had to be in the input of the inner join and the inner join filters had to reference both inputs of the left join, and not filter NULL values from the right input of the left join. Additionally, the right input of the left join had to contain at least one join, with one input not referenced by the left join's ON condition.
craig bot
pushed a commit
that referenced
this issue
Nov 17, 2022
91425: opt: don't drop LeftJoin filter during join ordering r=DrewKimball a=DrewKimball **opt: use RelExpr instead of ColumnID for reorderjoins relation map** The reorderjoins opttester directive previously maintained a map from the first column ID of each base relation to the relation's label in the output. This could cause a panic for relations that didn't output any columns. This patch changes the map to use the relations themselves as keys, which prevents the panic. **opt: don't drop LeftJoin filter during join ordering** This patch fixes a bug in the join reordering logic that can lead to incorrect results due to a dropped filter and incorrect conversion of a left join to an inner join. The bug can occur when the join tree contains an inner join with a left join as an input, where the inner join has two separate conjuncts in its ON condition that reference both inputs of the left join. Additionally, the inner join filters must not filter NULL values from the right side of the left join (or alternatively null-rejection rules must be disabled). The incorrect transformation looks something like this: ``` (INNER JOIN xy (LEFT JOIN ab (INNER JOIN uv wz ON v = w) ON b = v) ON a = x AND u = x) ``` => ``` (INNER JOIN ab (INNER JOIN xy (INNER JOIN uv wz ON v = w) ON u = x) ON a = x) ``` Notice how `xy` has been "pushed" into the right side of the left join and the left join's `b = v` filter (and the left join itself) dropped in the process. To understand what causes the bug, it is necessary to understand three points about the join reordering algorithm: 1. Cross products are never introduced in the enumerated plans. So, for two sub-plans, a join is only considered between them if there is an applicable edge between those sub-plans. 2. The original paper associates each join with exactly one edge in the hypergraph that encodes "reorderability" properties. 3. The `JoinOrderBuilder` departs from the paper by associating each inner join *conjunct* with a hypergraph edge. This allows each conjunct to be independently reordered from the others. See the `Special handling of inner joins` section in the `JoinOrderBuilder` comment for more details. (1) combined with (2) implies that a reordered join tree is only considered if every edge in the hypergraph could be applied to form joins in the join tree. This allows the original algorithm to prevent invalid orderings by making just a single edge inapplicable. However, because of (3) the same is no longer true for the `JoinOrderBuilder`. In the example given above, the left join fails the applicability check, indicating an invalid plan. However, the inner join's `a = x` filter passes the check and ends up replacing the left join. This prevents the the check in (1) from catching the invalid plan. This patch fixes the bug by keeping track of the edges that *should* be applied somewhere in each join tree based on the TES of each edge. This is then compared against the actual edges that are applied in the construction of the join tree. If the edge sets aren't equal, the plan is invalid and cannot be added to the memo. This allows the `JoinOrderBuilder` to recover the property that an inapplicable edge invalidates an enumerated plan. Fixes #90761 Release note (bug fix): Fixed a bug existing since 20.2 that could cause incorrect results in rare cases for queries with inner joins and left joins. For the bug to occur, the left join had to be in the input of the inner join and the inner join filters had to reference both inputs of the left join, and not filter NULL values from the right input of the left join. Additionally, the right input of the left join had to contain at least one join, with one input not referenced by the left join's ON condition. Co-authored-by: DrewKimball <drewk@cockroachlabs.com>
blathers-crl bot
pushed a commit
that referenced
this issue
Nov 17, 2022
This patch fixes a bug in the join reordering logic that can lead to incorrect results due to a dropped filter and incorrect conversion of a left join to an inner join. The bug can occur when the join tree contains an inner join with a left join as an input, where the inner join has two separate conjuncts in its ON condition that reference both inputs of the left join. Additionally, the inner join filters must not filter NULL values from the right side of the left join (or alternatively null-rejection rules must be disabled). The incorrect transformation looks something like this: ``` (INNER JOIN xy (LEFT JOIN ab (INNER JOIN uv wz ON v = w) ON b = v) ON a = x AND u = x) ``` => ``` (INNER JOIN ab (INNER JOIN xy (INNER JOIN uv wz ON v = w) ON u = x) ON a = x) ``` Notice how `xy` has been "pushed" into the right side of the left join and the left join's `b = v` filter (and the left join itself) dropped in the process. To understand what causes the bug, it is necessary to understand three points about the join reordering algorithm: 1. Cross products are never introduced in the enumerated plans. So, for two sub-plans, a join is only considered between them if there is an applicable edge between those sub-plans. 2. The original paper associates each join with exactly one edge in the hypergraph that encodes "reorderability" properties. 3. The `JoinOrderBuilder` departs from the paper by associating each inner join *conjunct* with a hypergraph edge. This allows each conjunct to be independently reordered from the others. See the `Special handling of inner joins` section in the `JoinOrderBuilder` comment for more details. (1) combined with (2) implies that a reordered join tree is only considered if every edge in the hypergraph could be applied to form joins in the join tree. This allows the original algorithm to prevent invalid orderings by making just a single edge inapplicable. However, because of (3) the same is no longer true for the `JoinOrderBuilder`. In the example given above, the left join fails the applicability check, indicating an invalid plan. However, the inner join's `a = x` filter passes the check and ends up replacing the left join. This prevents the the check in (1) from catching the invalid plan. This patch fixes the bug by keeping track of the edges that *should* be applied somewhere in each join tree based on the TES of each edge. This is then compared against the actual edges that are applied in the construction of the join tree. If the edge sets aren't equal, the plan is invalid and cannot be added to the memo. This allows the `JoinOrderBuilder` to recover the property that an inapplicable edge invalidates an enumerated plan. Fixes #90761 Release note (bug fix): Fixed a bug existing since 20.2 that could cause incorrect results in rare cases for queries with inner joins and left joins. For the bug to occur, the left join had to be in the input of the inner join and the inner join filters had to reference both inputs of the left join, and not filter NULL values from the right input of the left join. Additionally, the right input of the left join had to contain at least one join, with one input not referenced by the left join's ON condition.
blathers-crl bot
pushed a commit
that referenced
this issue
Nov 17, 2022
This patch fixes a bug in the join reordering logic that can lead to incorrect results due to a dropped filter and incorrect conversion of a left join to an inner join. The bug can occur when the join tree contains an inner join with a left join as an input, where the inner join has two separate conjuncts in its ON condition that reference both inputs of the left join. Additionally, the inner join filters must not filter NULL values from the right side of the left join (or alternatively null-rejection rules must be disabled). The incorrect transformation looks something like this: ``` (INNER JOIN xy (LEFT JOIN ab (INNER JOIN uv wz ON v = w) ON b = v) ON a = x AND u = x) ``` => ``` (INNER JOIN ab (INNER JOIN xy (INNER JOIN uv wz ON v = w) ON u = x) ON a = x) ``` Notice how `xy` has been "pushed" into the right side of the left join and the left join's `b = v` filter (and the left join itself) dropped in the process. To understand what causes the bug, it is necessary to understand three points about the join reordering algorithm: 1. Cross products are never introduced in the enumerated plans. So, for two sub-plans, a join is only considered between them if there is an applicable edge between those sub-plans. 2. The original paper associates each join with exactly one edge in the hypergraph that encodes "reorderability" properties. 3. The `JoinOrderBuilder` departs from the paper by associating each inner join *conjunct* with a hypergraph edge. This allows each conjunct to be independently reordered from the others. See the `Special handling of inner joins` section in the `JoinOrderBuilder` comment for more details. (1) combined with (2) implies that a reordered join tree is only considered if every edge in the hypergraph could be applied to form joins in the join tree. This allows the original algorithm to prevent invalid orderings by making just a single edge inapplicable. However, because of (3) the same is no longer true for the `JoinOrderBuilder`. In the example given above, the left join fails the applicability check, indicating an invalid plan. However, the inner join's `a = x` filter passes the check and ends up replacing the left join. This prevents the the check in (1) from catching the invalid plan. This patch fixes the bug by keeping track of the edges that *should* be applied somewhere in each join tree based on the TES of each edge. This is then compared against the actual edges that are applied in the construction of the join tree. If the edge sets aren't equal, the plan is invalid and cannot be added to the memo. This allows the `JoinOrderBuilder` to recover the property that an inapplicable edge invalidates an enumerated plan. Fixes #90761 Release note (bug fix): Fixed a bug existing since 20.2 that could cause incorrect results in rare cases for queries with inner joins and left joins. For the bug to occur, the left join had to be in the input of the inner join and the inner join filters had to reference both inputs of the left join, and not filter NULL values from the right input of the left join. Additionally, the right input of the left join had to contain at least one join, with one input not referenced by the left join's ON condition.
DrewKimball
added a commit
to DrewKimball/cockroach
that referenced
this issue
Nov 18, 2022
This patch fixes a bug in the join reordering logic that can lead to incorrect results due to a dropped filter and incorrect conversion of a left join to an inner join. The bug can occur when the join tree contains an inner join with a left join as an input, where the inner join has two separate conjuncts in its ON condition that reference both inputs of the left join. Additionally, the inner join filters must not filter NULL values from the right side of the left join (or alternatively null-rejection rules must be disabled). The incorrect transformation looks something like this: ``` (INNER JOIN xy (LEFT JOIN ab (INNER JOIN uv wz ON v = w) ON b = v) ON a = x AND u = x) ``` => ``` (INNER JOIN ab (INNER JOIN xy (INNER JOIN uv wz ON v = w) ON u = x) ON a = x) ``` Notice how `xy` has been "pushed" into the right side of the left join and the left join's `b = v` filter (and the left join itself) dropped in the process. To understand what causes the bug, it is necessary to understand three points about the join reordering algorithm: 1. Cross products are never introduced in the enumerated plans. So, for two sub-plans, a join is only considered between them if there is an applicable edge between those sub-plans. 2. The original paper associates each join with exactly one edge in the hypergraph that encodes "reorderability" properties. 3. The `JoinOrderBuilder` departs from the paper by associating each inner join *conjunct* with a hypergraph edge. This allows each conjunct to be independently reordered from the others. See the `Special handling of inner joins` section in the `JoinOrderBuilder` comment for more details. (1) combined with (2) implies that a reordered join tree is only considered if every edge in the hypergraph could be applied to form joins in the join tree. This allows the original algorithm to prevent invalid orderings by making just a single edge inapplicable. However, because of (3) the same is no longer true for the `JoinOrderBuilder`. In the example given above, the left join fails the applicability check, indicating an invalid plan. However, the inner join's `a = x` filter passes the check and ends up replacing the left join. This prevents the the check in (1) from catching the invalid plan. This patch fixes the bug by keeping track of the edges that *should* be applied somewhere in each join tree based on the TES of each edge. This is then compared against the actual edges that are applied in the construction of the join tree. If the edge sets aren't equal, the plan is invalid and cannot be added to the memo. This allows the `JoinOrderBuilder` to recover the property that an inapplicable edge invalidates an enumerated plan. Fixes cockroachdb#90761 Release note (bug fix): Fixed a bug existing since 20.2 that could cause incorrect results in rare cases for queries with inner joins and left joins. For the bug to occur, the left join had to be in the input of the inner join and the inner join filters had to reference both inputs of the left join, and not filter NULL values from the right input of the left join. Additionally, the right input of the left join had to contain at least one join, with one input not referenced by the left join's ON condition.
DrewKimball
added a commit
to DrewKimball/cockroach
that referenced
this issue
Nov 24, 2022
This patch fixes a bug in the join reordering logic that can lead to incorrect results due to a dropped filter and incorrect conversion of a left join to an inner join. The bug can occur when the join tree contains an inner join with a left join as an input, where the inner join has two separate conjuncts in its ON condition that reference both inputs of the left join. Additionally, the inner join filters must not filter NULL values from the right side of the left join (or alternatively null-rejection rules must be disabled). The incorrect transformation looks something like this: ``` (INNER JOIN xy (LEFT JOIN ab (INNER JOIN uv wz ON v = w) ON b = v) ON a = x AND u = x) ``` => ``` (INNER JOIN ab (INNER JOIN xy (INNER JOIN uv wz ON v = w) ON u = x) ON a = x) ``` Notice how `xy` has been "pushed" into the right side of the left join and the left join's `b = v` filter (and the left join itself) dropped in the process. To understand what causes the bug, it is necessary to understand three points about the join reordering algorithm: 1. Cross products are never introduced in the enumerated plans. So, for two sub-plans, a join is only considered between them if there is an applicable edge between those sub-plans. 2. The original paper associates each join with exactly one edge in the hypergraph that encodes "reorderability" properties. 3. The `JoinOrderBuilder` departs from the paper by associating each inner join *conjunct* with a hypergraph edge. This allows each conjunct to be independently reordered from the others. See the `Special handling of inner joins` section in the `JoinOrderBuilder` comment for more details. (1) combined with (2) implies that a reordered join tree is only considered if every edge in the hypergraph could be applied to form joins in the join tree. This allows the original algorithm to prevent invalid orderings by making just a single edge inapplicable. However, because of (3) the same is no longer true for the `JoinOrderBuilder`. In the example given above, the left join fails the applicability check, indicating an invalid plan. However, the inner join's `a = x` filter passes the check and ends up replacing the left join. This prevents the the check in (1) from catching the invalid plan. This patch fixes the bug by keeping track of the edges that *should* be applied somewhere in each join tree based on the TES of each edge. This is then compared against the actual edges that are applied in the construction of the join tree. If the edge sets aren't equal, the plan is invalid and cannot be added to the memo. This allows the `JoinOrderBuilder` to recover the property that an inapplicable edge invalidates an enumerated plan. Fixes cockroachdb#90761 Release note (bug fix): Fixed a bug existing since 20.2 that could cause incorrect results in rare cases for queries with inner joins and left joins. For the bug to occur, the left join had to be in the input of the inner join and the inner join filters had to reference both inputs of the left join, and not filter NULL values from the right input of the left join. Additionally, the right input of the left join had to contain at least one join, with one input not referenced by the left join's ON condition.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
roachtest.unoptimized-query-oracle/disable-rules=half failed with artifacts on release-22.2 @ 4816df3a9d76d179ed135a2b1efb53babb5611a0:
Parameters:
ROACHTEST_cloud=gce,ROACHTEST_cpu=4,ROACHTEST_encrypted=false,ROACHTEST_ssd=0Help
See: roachtest README
See: How To Investigate (internal)
This test on roachdash | Improve this report!
Jira issue: CRDB-20932
The text was updated successfully, but these errors were encountered: