Perfect Hash Join #1959

diegomestre2 · 2021-07-05T14:46:25Z

This PR provides the implementation of perfect hash join optimization.

For now, the optimization happens if the following statements are true:

Inner Join
One condition
Build side with no duplicates
Integer keys

This is done inside the physical hash join as an alternative path. In case it is not possible, it falls back to a normal hash join.

The perfect hash join copies the build data from the hash table after the build phase and stores it in a columnar manner.
The probe is done using the difference between the min value and the probe keys as indexes into the perfect hash table.
The result is a reference to the probe data and a dictionary vector containing the build keys.

Experiments with a fixed build side of 1024 and varying the probe side from 10k to 10M tuples achieve speedups of almost 8x.

TODOs:

Fast path to reference both probes and build when the probe is part of the build domain.
Support inner joins with more conditions

…ctor

…t its use

…probe side

…ild domain

hannes · 2021-08-13T07:58:07Z

Did you need help getting this sorted in multi-threading @diegomestre2 ?

diegomestre2 · 2021-08-27T08:39:32Z

Sorry for not finishing this PR. I spent some time on it and couldn't figure it out. But I will try it again and get back to you if I can't sort it out. I was just too busy during the previous weeks.

…ating selection vector

Mytherin

Thanks for the fixes @pdet ! Some more comments:

Mytherin · 2021-09-03T08:56:06Z

src/execution/operator/join/perfect_hash_join_executor.cpp

+		TemplatedFillSelectionVectorBuild<int16_t>(source, sel_vec, seq_sel_vec, count);
+		break;
+	case PhysicalType::TIME32:
+	case PhysicalType::DATE32:


PhysicalType::TIME32/DATE32/TIME64/DATE64/TIMESTAMP do not exist in our system (they are arrow only)

Mytherin · 2021-09-03T08:56:28Z

src/execution/operator/join/perfect_hash_join_executor.cpp

+		TemplatedFillSelectionVectorProbe<int16_t>(source, build_sel_vec, probe_sel_vec, count, probe_sel_count);
+		break;
+	case PhysicalType::TIME32:
+	case PhysicalType::DATE32:


Same here, these types aren't used

Mytherin · 2021-09-03T08:57:08Z

src/execution/physical_plan/plan_comparison_join.cpp

+	}
+	// with integral types
+	for (auto &&join_stat : op.join_stats) {
+		if (!join_stat->type.IsIntegral() || join_stat->type == LogicalTypeId::HUGEINT) {


^ Can we check the physical type here instead of the logical type so this also works for dates/times/etc?

Plus, can we add hugeint support? Unless that is too much work.

Mytherin · 2021-09-03T08:59:13Z

src/execution/physical_plan/plan_comparison_join.cpp

+	}
+	// with equality condition
+	for (auto &&condition : op.conditions) {
+		if (condition.comparison != ExpressionType::COMPARE_EQUAL) {


Should we check null_values_are_equal here?

I'm ok with adding the other data types in the future, but I think for this PR I just want to get Diego's stuff working.

Mytherin · 2021-09-03T08:59:59Z

src/include/duckdb/common/types/vector_buffer.hpp

@@ -95,8 +95,7 @@ class DictionaryBuffer : public VectorBuffer {
 	void SetSelVector(const SelectionVector &vector) {
 		this->sel_vector.Initialize(vector);
 	}
-
-private:
+	// private:


Can this remain private?

test/sql/join/inner/test_join_invisible_probe.test

Mytherin · 2021-09-03T09:01:32Z

test/sql/join/inner/test_using_join.test

@@ -3,7 +3,6 @@
 # group: [inner]

 statement ok
-PRAGMA enable_verification


Should this be removed?

Mytherin · 2021-09-03T09:02:13Z

src/include/duckdb/execution/operator/join/physical_hash_join.hpp

@@ -34,6 +72,9 @@ class PhysicalHashJoin : public PhysicalComparisonJoin {
 	vector<LogicalType> build_types;
 	//! Duplicate eliminated types; only used for delim_joins (i.e. correlated subqueries)
 	vector<LogicalType> delim_types;
+	//! Checks and execute perfect hash join optimization
+	unique_ptr<PerfectHashJoinExecutor> pjoin_executor {nullptr};
+	bool use_perfect_hash {false};


Mytherin · 2021-09-03T09:02:56Z

src/execution/operator/join/physical_hash_join.cpp

-#include "duckdb/parallel/thread_context.hpp"
-#include "duckdb/storage/storage_manager.hpp"
+
+#include <utility>


Is this include added on purpose?

Grr, CLion again

test/sql/join/inner/test_join_perfect_hash.test

Mytherin · 2021-09-04T17:57:32Z

There still appears to be some failures remaining in the SQLancer tests. Could you have a look there?

…tats builds are null to avoid null comparisons when optimizing for perfect Ht

Mytherin

Thanks for the fixes. Looks good. One more comment:

Mytherin · 2021-09-05T19:34:08Z

test/sql/join/inner/test_join_perfect_hash.test

+statement ok
+insert into null_equal_sec values (4),(5),(NULL)
+
+#Unoptimized version can't handle this query


Why not? That seems like a bug. Optimizers should not be necessary for correctness.

It falls on a non-implemented type of the unoptimized join, if I remember correctly.

Unoptimized Result:
INTERNAL Error: Unimplemented comparison type for merge join!

With the comparison type being:
COMPARE_NOT_DISTINCT_FROM

That is a bug then. A merge join shouldn't be generated, but rather a hash join or a nested loop join, no? Probably a missed case in #1363.

Fair enough, I'll fix it here. :-)

Mytherin

Some more comments w.r.t. code coverage:

Mytherin · 2021-09-05T19:35:39Z

src/common/row_operations/row_gather.cpp

@@ -32,6 +33,10 @@ static void TemplatedGatherLoop(Vector &rows, const SelectionVector &row_sel, Ve
 		data[col_idx] = Load<T>(row + col_offset);
 		ValidityBytes row_mask(row);
 		if (!row_mask.RowIsValid(row_mask.GetValidityEntry(entry_idx), idx_in_entry)) {
+			if (build_size > STANDARD_VECTOR_SIZE && col_mask.AllValid()) {
+				//! We need to initialize the mask with the vector size.
+				col_mask.Initialize(build_size);


This line is not covered, could you add a test that covers this?

Mytherin · 2021-09-05T19:36:12Z

src/common/row_operations/row_gather.cpp

+	col.SetVectorType(VectorType::FLAT_VECTOR);
+	switch (col.GetType().InternalType()) {
+	case PhysicalType::UINT8:
+		TemplatedFullScanLoop<uint8_t>(rows, col, count, col_offset, col_no);


Almost none of these types are covered. Could you add tests for all these types?

You did some type loop for the tests right? Can you point me out to one of these files?

See e.g. here

Mytherin · 2021-09-05T19:36:23Z

src/common/row_operations/row_gather.cpp

+		data[i] = Load<T>(row + col_offset);
+		ValidityBytes row_mask(row);
+		if (!row_mask.RowIsValid(row_mask.GetValidityEntry(entry_idx), idx_in_entry)) {
+			col_mask.SetInvalid(i);


This line is uncovered. Could you add a test that covers this?

For this line to be hit we need null values in the hash table.
However null comparisons are not done in this join, they are skipped in a phase prior to that.
I don't think we should remove this code, because at some point we might implement it anyway, but it is "unreachable" through testing right now. What you think?

Perhaps turn this into an internal exception for now if it is never supposed to be triggered currently, with the line of code commented out? If we need it in the future we can remove the exception again.

Mytherin · 2021-09-05T19:36:59Z

src/execution/physical_plan/plan_comparison_join.cpp

+	join_state.build_max = stats_build->max;
+	join_state.estimated_cardinality = op.estimated_cardinality;
+	if (build_range.type().id() == LogicalTypeId::DECIMAL) {
+		switch (build_range.type().InternalType()) {


These lines are uncovered. Could you add a test that covers this?

Mytherin · 2021-09-05T19:37:09Z

src/execution/physical_plan/plan_comparison_join.cpp

+			join_state.build_range = build_range.value_.bigint;
+			break;
+		default:
+			throw std::runtime_error("Invalid Physical Type for Decimals");


InternalException

Mytherin · 2021-09-05T19:37:19Z

src/execution/physical_plan/plan_comparison_join.cpp

+		case PhysicalType::INT32:
+			join_state.build_range = build_range.value_.integer;
+			break;
+		case PhysicalType::INT64:


Decimals can also be HUGEINT

True, but that physical type is ignored for this join, I can add it anyway.

…perfect HJ

diegomestre2 added 30 commits April 21, 2021 16:01

Add test for hash join with perfect hash

33d309f

Add propagate stats to join for perfect hash

656c7f7

Adding includes

9925b05

Adding implementations for perfect hash join

5b833c5

uncommenting branch

2634e69

Perfect hash join should occur inside hash join in case it falls back

b31e717

Removing unwanted files and adding flag to physical hash join constru…

a545967

…ctor

Creating perfect hash join state object to store ranges and info abou…

244a35a

…t its use

Adding commented code

20fa952

Retrieving left side and right side

46ea7ea

Probe right side using the difference of the minimum value

d373392

Joins and copies the right side in the perfect hash join

1d61fc0

Fetching right side from hashtable

f438825

The build side should be fetched using the selection vector from the …

d568c5d

…probe side

Adding checks for perfect hash table

e6df3b7

Perfect hash table should be built right after hash table build is done

4acb1d2

Check for perfect hash join only when columns are numeric

48a3e0c

Adding Vector constructor with variable size

1dd4063

Build perfect hash table structure

e2a8841

Add gather mathods to class definition

49b3000

Fix for non numeric statistics

07b5f4b

Create scan over hash table to store offsets

6c3ddf1

Build columnar hash table for perfect hash join

852d0c1

Fix for nullptrsd

724e320

Merge with master

eca0065

Fix merge

a20b1cc

Adding changes in the hash table building

c3730cf

Replacing min-max checking with vectorized primitive

1dc5e93

Propagating build and probe stats to execution

cc5ed9f

Check during the planning whether the probe min-max is part of the bu…

4508830

…ild domain

diegomestre2 and others added 8 commits August 28, 2021 10:39

Merge with master

6464e9f

Fix for some tidy check errors

1e9c77f

Merge branch 'duckdb:master' into master

f2f9bb6

Fix for parallel issue on perfect HJ

0ee7b63

Fixing includes

8f767c4

Fixing the invalid key types on Perfect HJ

f7f9f23

More fixes for Perfect HJ, avoiding null comparisons and properly cre…

9e2b49d

…ating selection vector

Fixing validity mask creation in row gather function

a754dee

Mytherin reviewed Sep 3, 2021

View reviewed changes

pdet added 4 commits September 3, 2021 13:36

PR Requests

61de84c

Fixing Perfect HJ bugs related to null values and decimal types

7d42096

Fixing test join invisible probe

7a0e4e4

Hopefully fixing the build

8bfcc33

pdet added 2 commits September 5, 2021 12:59

Small fix when constructing nullmasks on row gather and checking if s…

5d53052

…tats builds are null to avoid null comparisons when optimizing for perfect Ht

Only run perfect HJ when build size <= 10pow6

1da8d54

Mytherin reviewed Sep 5, 2021

View reviewed changes

pdet and others added 8 commits September 6, 2021 14:09

PR requests

93f6502

Merge branch 'master' of github.com:diegomestre2/duckdb

5e0a200

Adding internal execeptions on code that is should be unreachable at …

ee8140c

…perfect HJ

Commenting out unused line so git doesn't get angry with me

2af736c

Update test_not_distinct_from.test

a0f9a93

Codecov on unsigned types of Perfect HJ

a77f703

Merge branch 'master' of github.com:diegomestre2/duckdb

a6ba64b

forgot the decimals

726173a

Mytherin merged commit e8a9967 into duckdb:master Sep 7, 2021

Mytherin mentioned this pull request Mar 24, 2023

INNER JOIN is dropping too many rows when operating over 2 larger parquet files #6854

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Perfect Hash Join #1959

Perfect Hash Join #1959

diegomestre2 commented Jul 5, 2021 •

edited by hannes

hannes commented Aug 13, 2021

diegomestre2 commented Aug 27, 2021

Mytherin left a comment

Mytherin Sep 3, 2021

Mytherin Sep 3, 2021

Mytherin Sep 3, 2021 •

edited

Mytherin Sep 3, 2021

pdet Sep 3, 2021

Mytherin Sep 3, 2021

Mytherin Sep 3, 2021

Mytherin Sep 3, 2021

Mytherin Sep 3, 2021

pdet Sep 3, 2021

Mytherin commented Sep 4, 2021

Mytherin left a comment

Mytherin Sep 5, 2021

pdet Sep 5, 2021 •

edited

pdet Sep 5, 2021

pdet Sep 5, 2021

Mytherin Sep 5, 2021

pdet Sep 5, 2021

Mytherin left a comment

Mytherin Sep 5, 2021

Mytherin Sep 5, 2021

pdet Sep 5, 2021

Mytherin Sep 5, 2021

Mytherin Sep 5, 2021

pdet Sep 6, 2021

Mytherin Sep 6, 2021

Mytherin Sep 5, 2021

Mytherin Sep 5, 2021

Mytherin Sep 5, 2021

pdet Sep 5, 2021

Perfect Hash Join #1959

Perfect Hash Join #1959

Conversation

diegomestre2 commented Jul 5, 2021 • edited by hannes

hannes commented Aug 13, 2021

diegomestre2 commented Aug 27, 2021

Mytherin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mytherin Sep 3, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mytherin commented Sep 4, 2021

Mytherin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdet Sep 5, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mytherin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

diegomestre2 commented Jul 5, 2021 •

edited by hannes

Mytherin Sep 3, 2021 •

edited

pdet Sep 5, 2021 •

edited