Full sorting merge join, pt 1 #35796

vdimir · 2022-03-31T10:47:55Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Added full sorting merge join algorithm

Reference files for tests are generated by result of hash algorithm.
TODO (for next PR):
- support conditions from ON section
- lowcard keys
- use nullableCompare only if there's nullable columns
- Anti/Semi/Asof joins
- port more existing join tests for new algorithm
- get rid of virtual calls insertFrom/inserdDefault/... in internal loops (need benchmark)
- use defaults from type, not column (https://github.com/ClickHouse/ClickHouse/blob/master/tests/queries/0_stateless/01656_join_defaul_enum.sql)

vdimir · 2022-05-16T11:08:32Z

Stateless tests flaky check (address, actions) — Tests are not finished, fail: 0, passed: 360

Still need to investigate

UPD: seems that server just hadn't enough time to shut down

vdimir · 2022-05-18T12:02:31Z

This PR contains bunch of changed and added tests and lots of them are long, so seems that timeout 1800 bash -c run_tests in Stateless Tests Flaky Check is not enough. Should we fix test scripts to adjust timeout to number of tests, or just merge?

src/Interpreters/FullSortingMergeJoin.h

KochetovNicolai · 2022-06-09T08:39:05Z

src/Interpreters/FullSortingMergeJoin.h

+            bool type_equals
+                = table_join->hasUsing() ? left_type->equals(*right_type) : removeNullable(left_type)->equals(*removeNullable(right_type));


Hm, does it mean we can't join like UInt32 and UInt64 columns?
Maybe we need to find the least common type (at least for numeric columns)

We add conversion on previous steps, so UInt32 and UInt64 would work. Here is just to check once more that type conversion were added successfully

KochetovNicolai · 2022-06-09T08:40:15Z

src/Interpreters/FullSortingMergeJoin.h

+    }
+
+    /// Used just to get result header
+    void joinBlock(Block & block, std::shared_ptr<ExtraBlock> & /* not_processed */) override


Probably we need to update IJoin interface a little bit

To have something like IJoin::transformHeader?

KochetovNicolai · 2022-06-09T08:43:41Z

src/Interpreters/IJoin.h

@@ -43,7 +64,8 @@ class IJoin

    /// StorageJoin/Dictionary is already filled. No need to call addJoinedBlock.
    /// Different query plan is used for such joins.
-    virtual bool isFilled() const { return false; }
+    virtual bool isFilled() const { return pipelineType() == JoinPipelineType::FilledRight; }
+    virtual JoinPipelineType pipelineType() const { return JoinPipelineType::FillRightFirst; }


I like JoinPipelineType enum, but could you add a comment?

There's some comment at enum class JoinPipelineType declaration, but I've added short info here as well https://github.com/vdimir/ClickHouse/blob/c262d4d2c7c3551d49d1b8d4f324e4a3a550698f/src/Interpreters/IJoin.h#L20-L39

KochetovNicolai · 2022-06-09T10:30:47Z

src/QueryPipeline/QueryPipelineBuilder.h

@@ -105,6 +115,14 @@ class QueryPipelineBuilder
        bool keep_left_read_in_order,
        Processors * collected_processors = nullptr);

+    static std::unique_ptr<QueryPipelineBuilder> joinPipelines2(


We need a better name here :)

Changed to QueryPipelineBuilder::joinPipelinesYShaped and QueryPipelineBuilder::joinPipelinesRightLeft

yakov-olkhovskiy

LGTM mostly

vdimir · 2022-07-07T15:00:44Z

@KochetovNicolai @yakov-olkhovskiy let's merge?

alanpaulkwan · 2022-07-25T05:49:39Z

Hi, I have a question about how this is supposed to work. I posted an issue here: #39542

Alex-Cheng · 2023-09-08T09:58:02Z

src/Processors/Transforms/MergeJoinTransform.cpp

+    if (!cursors[1]->cursor.isValid() && !cursors[1]->fullyCompleted())
+        return Status(1);
+
+    if (auto result = handleAllJoinState())


Just I am curious, why call handleAllJoinState() method here instead of calling it in allJoin(), and calling handleAnyJoinState is located in anyJoin() method.

To be honest I don't remember exactly. Perhaps handle*State functions rely on the code below (/// check if blocks are not intersecting at all) in a different ways. Or it's also possible that there's no particular reason, need to dive into the code to understand.

Alex-Cheng · 2023-09-08T12:33:36Z

src/Processors/Transforms/MergeJoinTransform.cpp

+                {
+                    assert(state == nullptr);
+                    state = std::make_unique<AllJoinState>(left_cursor.cursor, lpos, right_cursor.cursor, rpos);
+                    state->addRange(0, left_cursor.getCurrent().clone(), lpos, lnum);


Shall we only copy the necessary rows from left_cursor.getCurrent() chunk and right_cursor.getCurrent() chunk?

vdimir added the do not test disable testing on pull request label Mar 31, 2022

robot-ch-test-poll2 added the pr-feature Pull request with new product feature label Mar 31, 2022

vdimir force-pushed the full-sorting-merge-join branch 3 times, most recently from 2710e9e to d6452fa Compare April 6, 2022 14:40

vdimir removed the do not test disable testing on pull request label Apr 6, 2022

vdimir force-pushed the full-sorting-merge-join branch 3 times, most recently from 2f92fc9 to 70a6f1a Compare May 3, 2022 10:04

vdimir force-pushed the full-sorting-merge-join branch 2 times, most recently from 58bc152 to 5fd0d4f Compare May 4, 2022 10:51

vdimir force-pushed the full-sorting-merge-join branch from 5fd0d4f to d8d3347 Compare May 13, 2022 15:49

vdimir marked this pull request as ready for review May 16, 2022 11:21

vdimir changed the title ~~[WIP] Full sorting merge join~~ Full sorting merge join, pt 1 May 18, 2022

vdimir force-pushed the full-sorting-merge-join branch 2 times, most recently from 130d235 to 96d67e0 Compare May 20, 2022 12:00

vdimir force-pushed the full-sorting-merge-join branch from 96d67e0 to f6d3300 Compare May 25, 2022 16:14

yakov-olkhovskiy self-assigned this May 27, 2022

KochetovNicolai reviewed Jun 9, 2022

View reviewed changes

src/Interpreters/FullSortingMergeJoin.h Outdated Show resolved Hide resolved

KochetovNicolai reviewed Jun 9, 2022

View reviewed changes

src/Interpreters/FullSortingMergeJoin.h Outdated Show resolved Hide resolved

KochetovNicolai reviewed Jun 9, 2022

View reviewed changes

yakov-olkhovskiy approved these changes Jun 14, 2022

View reviewed changes

vdimir force-pushed the full-sorting-merge-join branch from f6d3300 to c262d4d Compare July 5, 2022 16:21

vdimir requested a review from yakov-olkhovskiy July 5, 2022 16:26

Pipeline for full sorting merge join

fa8eb35

vdimir added 17 commits July 6, 2022 14:26

full sorting all join

a0144e1

tests full sort join

92ff43e

add not implemented checks, add using testcase to full sort join

a2a7abc

full sort join: check key types, more tests

d184e18

MergeJoinAlgorithm::createBlockWithDefaults

a90ac59

full sorting join with using

753a567

full sorting join using and nulls

c200ce4

couple of tests for full_sorting_merge

c8cd962

fix build

0b994bb

Fix column remap in MergeJoinTransform

f8e6660

minor changes in full sort join

aff6654

Add algorithms to 00172_parallel_join

a4b93c7

Add join_algorithm='full_sorting_merge' to stress tests

b0c0ec3

upd full sorting join

881d352

Minor updates for full soring merge join

7c586a9

Fix style FullSortingMergeJoin.h

2fd2294

Fix build

b27d3ec

vdimir force-pushed the full-sorting-merge-join branch from 95e806a to b27d3ec Compare July 6, 2022 15:36

vdimir mentioned this pull request Jul 7, 2022

Avoid redundant join block transformation during planning. #38943

Merged

yakov-olkhovskiy approved these changes Jul 7, 2022

View reviewed changes

vdimir merged commit db838f1 into ClickHouse:master Jul 7, 2022

vdimir deleted the full-sorting-merge-join branch July 7, 2022 17:16

This was referenced Jul 8, 2022

Disable instrumentation of sanitizer death callback #38977

Merged

Invalid number of columns in chunk pushed to OutputPort. Expected 3, found 1 #39008

Closed

rfraposa mentioned this pull request Aug 9, 2022

Add docs for the new types of JOINs ClickHouse/clickhouse-docs#283

Closed

zhanglistar mentioned this pull request Nov 17, 2022

Sort merge join support Kyligence/ClickHouse#196

Closed

Alex-Cheng reviewed Sep 8, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Full sorting merge join, pt 1 #35796

Full sorting merge join, pt 1 #35796

vdimir commented Mar 31, 2022 •

edited

vdimir commented May 16, 2022 •

edited

vdimir commented May 18, 2022

KochetovNicolai Jun 9, 2022

vdimir Jul 5, 2022 •

edited

KochetovNicolai Jun 9, 2022

KochetovNicolai Jun 9, 2022

vdimir Jul 5, 2022

KochetovNicolai Jun 9, 2022

vdimir Jul 5, 2022

KochetovNicolai Jun 9, 2022

vdimir Jul 5, 2022

yakov-olkhovskiy left a comment

vdimir commented Jul 7, 2022

alanpaulkwan commented Jul 25, 2022

Alex-Cheng Sep 8, 2023

vdimir Sep 8, 2023

Alex-Cheng Sep 8, 2023

		bool type_equals
		= table_join->hasUsing() ? left_type->equals(right_type) : removeNullable(left_type)->equals(removeNullable(right_type));

Full sorting merge join, pt 1 #35796

Full sorting merge join, pt 1 #35796

Conversation

vdimir commented Mar 31, 2022 • edited

vdimir commented May 16, 2022 • edited

vdimir commented May 18, 2022

Choose a reason for hiding this comment

vdimir Jul 5, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yakov-olkhovskiy left a comment

Choose a reason for hiding this comment

vdimir commented Jul 7, 2022

alanpaulkwan commented Jul 25, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vdimir commented Mar 31, 2022 •

edited

vdimir commented May 16, 2022 •

edited

vdimir Jul 5, 2022 •

edited