Fix bugs in MergeJoin when 'not_processed' is not null

[liql2007]

Changelog category:

• Bug Fix

Changelog entry:

Fix bugs in MergeJoin when 'not_processed' is not null.

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

[CLAassistant]

All committers have signed the CLA.

[vdimir]

Thank you for your contribution!

Could you please describe how the bug affects the result? Does it lead to duplicated rows in the result? Do you have a reproducible example (if yes, it's good to add a test case)?

Probably related to #31009

[liql2007]

@vdimir Thanks for your review！
This bug is not same as #31009.
It will cause loss of partial results, not duplicated results.

[liql2007]

If not reset skip_right, may be cause wrong right cursor position in the following leftJoin<is_all>(...)

[liql2007]

@vdimir

[liql2007]

Could not update the left_cursor yet, because, maybe there are still equal keys in the right side. Forwarding the left_cursor right now, will cause the loss of result.

[liql2007]

@vdimir

[liql2007]

Hi @vdimir, I can not see the details of the failing check. Could you please help me confirm the cause of the failure？ Thank you very much! :-)

[vdimir]

Performance Comparison Aarch64 [1/4] — Failed to parse the report
PullRequestCI / PerformanceComparisonAarch-0 (pull_request) Successful in 17s

Looks like internal CI failure for this task

[vdimir]

Overall looks reasonable. It's still quite tricky to understand without an example. Is it difficult to reproduce (maybe changing max_block_size) in any example? Existing tests are green, though.

[liql2007]

Set MergeJoin.max_joined_block_rows = 2, and in LEFT JOIN scenario, we have:

left block:
┌id┐
│ 1 │
│ 2 │
│ 3 │
└─┘

two blocks at right side:
┌id┬value┐
│ 1 │ 1 │
│ 2 │ 2 │
│ 2 │ 3 │
└─┴─┘

┌id┬value┐
│ 3 │ 4 │
│ 3 │ 5 │
│ 4 │ 6 │
└─┴─┘

Step 1: left column 1 matches with right column 1
Step 2: left column 2 matches with right column [2, 3], but the joined (result) block size can not exceed the threshold MergeJoin.max_joined_block_rows (= 2), then:

• generate the joined block:
  ┌id┬id┬value┐
  │ 1 │ 1 │ 1 │
  │ 2 │ 2 │ 2 │
  └─┴─┴─┘
• skip right cursor to the third row
• set skip_right = 2
• skip left cursor to the third row (this is a bug ) @vdimir
• create not_processed

Step 3: start next round merge join according to the not_processed. Because of the wrong left cursor position, and the wrong updating of skip_right, left block do not match anything with the second right block.

[vdimir]

Should this script reproduce the error?

SET join_algorithm='partial_merge';

SET max_block_size=3;
SET max_joined_block_size_rows = 2;

DROP TABLE IF EXISTS t1;
DROP TABLE IF EXISTS t2;
DROP TABLE IF EXISTS t3;

CREATE TABLE t1 (x UInt64) ENGINE = TinyLog;
INSERT INTO t1 VALUES (1), (2), (3);

CREATE TABLE t2 (x UInt64, value String) ENGINE = TinyLog;
INSERT INTO t2 VALUES (1, 'a'), (2, 'b'), (2, 'c');
INSERT INTO t2 VALUES (3, 'd'), (3, 'e'), (4, 'f');

SELECT * FROM t1 JOIN t2 ON t1.x = t2.x;

┌─x─┬─t2.x─┬─value─┐
│ 1 │    1 │ a     │
│ 2 │    2 │ b     │
│ 2 │    2 │ c     │
│ 3 │    3 │ d     │
│ 3 │    3 │ e     │
└───┴──────┴───────┘

[liql2007]

In INNER JOIN, after the join interrupt because of the too large joined block, we did not skip the left table cursor (code), so, INNER JOIN is okay.

This bug only exists in LEFT JOIN scenario.

At last, thanks for your review again !