Direct (nested loop) join for merge tree tables

[vdimir]

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

• Support direct (nested loop) join for MergeTree tables. To use it, specify it as the single option in the setting: join_algorithm = 'direct'

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

[clickhouse-gh]

Workflow [PR], commit [8381b14]

Summary: ❌

job_name	test_name	status	info	comment
BuzzHouse (amd_debug)		failure
	Logical error: 'Inconsistent AST formatting in SelectQuery: the query:	FAIL	cidb

[rschu1ze]

Did direct joins for merge tree tables not work before?

[vdimir]

Did direct joins for merge tree tables not work before?

@rschu1ze

Not quite, it works only if you create LAYOUT(DIRECT()) on top of table and join with that dictionary

[rschu1ze]

Right, I misread the PR description somehow. Thanks.

[Copilot]

Pull request overview

This PR introduces support for direct (nested loop) join with MergeTree tables, controlled by setting join_algorithm = 'direct'. The implementation enables efficient joins between regular tables and MergeTree tables by performing index-based lookups instead of building hash tables.

Key Changes:

• Added DirectJoinMergeTreeEntity class that implements IKeyValueEntity interface for MergeTree direct join operations
• Extended IKeyValueEntity::getByKeys interface to support ALL join semantics via an out_offsets parameter
• Implemented query plan cloning capabilities for ReadFromMergeTree and storage snapshots to enable parallel lookups
• Added comprehensive test coverage for various join types (INNER, LEFT, SEMI LEFT, ANTI LEFT) with MergeTree tables

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
src/Interpreters/DirectJoinMergeTreeEntity.h/cpp	New entity implementing direct join logic for MergeTree tables with IN-based filtering
src/Interpreters/IKeyValueEntity.h	Extended interface with out_offsets parameter for ALL join semantics
src/Interpreters/DirectJoin.cpp	Updated to handle ALL join semantics with offsets from key-value entities
src/Planner/PlannerJoinTree.cpp	Added logic to detect and create DirectJoinMergeTreeEntity when conditions are met
src/Processors/QueryPlan/ReadFromMergeTree.h/cpp	Added cloning support and query condition cache controls for multiple lookups
src/Storages/StorageSnapshot.h	Added virtual clone() method to Data base class
src/Storages/MergeTree/MergeTreeData.h	Implemented clone() for MergeTree snapshot data
src/Storages/StorageMemory.h	Implemented clone() for Memory snapshot data
src/Storages/Storage*.h/cpp	Updated getByKeys signatures to include out_offsets parameter
tests/queries/0_stateless/03712_*.sql/reference	Functional test verifying indexed vs full scan behavior
tests/queries/0_stateless/03742_*.sql/reference	Long test verifying different join types with large datasets

[Fgrtue]

During the reading I had a though that there could be a slightly more meaningful names for these variables. Something that relates to what we actually collect in them, i.e. indices_corresponding_to_right_table_keys or something in that way. Do you think it would worth renaming them for improvement of clarity?

[vdimir]

Changed to selector

[Fgrtue]

The same about this variable. I thought something relating to its meaning might improve the readability

[vdimir]

Actually it's common for findKey result in codebase https://github.com/search?q=repo%3AClickHouse%2FClickHouse+%2Ffind_result.*findKey%2F&type=code

[Fgrtue]

Do I understand correctly that this insert to the end of column is made in order to have a default null value for the rows from the left table, for which we did not find anything?

[vdimir]

Yes, added a comment

[EmeraldShift]

Would it be possible to extend support to range conditions like BETWEEN? My use case is that I have a table sorted by (a, b, timestamp), and a materialized view that aggregates some_id, a, b, min(timestamp), max(timestamp) GROUP BY ALL. I'd like to query the materialized view to receive a, b, min_t, max_t "ranges", and then directly join these ranges against the original MergeTree table, where each condition is a single consecutive range and should be extremely efficient to find.

[vdimir]

@EmeraldShift does ASOF JOIN not work because it finds only a single first/last match, but you need all of them? Does your query work with a regular hash join (in some cases we can execute complex conditions, but maybe in an inefficient way)? So does your approach fail because of an error like "not supported" or because the query is too long/consumes too much memory?

Is your query like following?

SELECT * FROM mv JOIN table
ON table.timestamp BETWEEN mv.min_t AND mv.max_t AND table.a = mv.a AND table.b = mv.b

Overall, supporting nested loops for arbitrary conditions would be nice, but we haven't planned it yet. You may create an issue with a more detailed explanation of the use case and why you think nested loops might work better than hash, just to track it.