Skip to content

Parallel non-order preserving CREATE TABLE AS and INSERT INTO #5033

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 39 commits into from
Oct 20, 2022

Conversation

Mytherin
Copy link
Collaborator

This PR adds support for parallel processing of CREATE TABLE AS and INSERT INTO queries in a non-order preserving manner. That is to say, the order of the tuples in the table is not guaranteed to be the same as the insertion order.

This method is used when preservation of insertion order is disabled (using SET preserve_insertion_order=false) or when data is inserted from a pipeline that does not have a defined order, such as when inserting data from a GROUP BY statement or a join between tables.

The order-preserving parallel implementation will follow in a subsequent PR.

Below are the new timings of the h2oai "join" benchmark after this PR (join in quotation brackets because the cost of the queries is dominated by the CREATE TABLE statement materializing millions of rows - the join is mostly irrelevant).

Query Old New
Q01 1.35s 0.18s
Q02 1.63s 0.22s
Q03 1.91s 0.28s
Q04 1.59s 0.22s
Q05 2.31s 0.45s

result_query in benchmark suite

This PR also adds support for the result_query verification method in benchmarks. Using this verification method, rather than checking the result of a query directly, the specified result_query is executed and the result of that query is checked instead. This allows us to (1) add result verification to queries with large results (such as the h2oai queries), and (2) add result verification to queries that have underspecified results (such as several clickbench queries).

Example usage:

result_query IIIII
SELECT COUNT(DISTINCT id2), COUNT(DISTINCT id4), SUM(r2), COUNT(*) FROM ans;
----
95	95	9.940515516534347	9216

The addition of the result_query syntax breaks the h2oai regression CI run, as the current master does not understand the verification syntax.

Parallel Aggregate HT Scans

This PR also enables parallel aggregate HT scans. This was sort-of implemented before, but disabled as there were a few problems found with the implementation. This PR fixes those bugs and enables support for the parallel aggregate HT scan. This enables queries that perform further operations on e.g. the result of large DISTINCT or GROUP BY to run more in parallel.

General Clean-up

  • The CREATE TABLE AS and INSERT INTO statements now use the same physical operator underneath (PhysicalInsert)
  • SegmentTree clean-up: move ownership of nodes to the SegmentTree itself, and add more robust locking and more control over the locking of the SegmentTree
  • Add OptimisticDataWriter and LocalTableManager classes that take over certain logic of the LocalStorage class, and add thread safety to the operations that is required now that we have parallel writes
  • Move TableStatistics into the RowGroupCollection, rather than having this be maintained outside and passed into certain operations
  • Fix a race condition that would cause Parallel appends to table with index with transactions to fail occasionally

…fixes a race condition in "Parallel appends to table with index with transactions"
@Mytherin Mytherin merged commit 8fcee5d into duckdb:master Oct 20, 2022
@Mytherin Mytherin deleted the parallelcreatetable branch January 7, 2023 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant