-
Notifications
You must be signed in to change notification settings - Fork 2.5k
Parallel non-order preserving CREATE TABLE AS and INSERT INTO #5033
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…cation queries to h2oai benchmark
…he actual segment tree
…fixes a race condition in "Parallel appends to table with index with transactions"
…for thread safety
… values followed by a non-null value
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for parallel processing of
CREATE TABLE ASandINSERT INTOqueries in a non-order preserving manner. That is to say, the order of the tuples in the table is not guaranteed to be the same as the insertion order.This method is used when preservation of insertion order is disabled (using
SET preserve_insertion_order=false) or when data is inserted from a pipeline that does not have a defined order, such as when inserting data from aGROUP BYstatement or a join between tables.The order-preserving parallel implementation will follow in a subsequent PR.
Below are the new timings of the h2oai "join" benchmark after this PR (join in quotation brackets because the cost of the queries is dominated by the
CREATE TABLEstatement materializing millions of rows - the join is mostly irrelevant).result_queryin benchmark suiteThis PR also adds support for the
result_queryverification method in benchmarks. Using this verification method, rather than checking the result of a query directly, the specifiedresult_queryis executed and the result of that query is checked instead. This allows us to (1) add result verification to queries with large results (such as the h2oai queries), and (2) add result verification to queries that have underspecified results (such as several clickbench queries).Example usage:
The addition of the
result_querysyntax breaks the h2oai regression CI run, as the current master does not understand the verification syntax.Parallel Aggregate HT Scans
This PR also enables parallel aggregate HT scans. This was sort-of implemented before, but disabled as there were a few problems found with the implementation. This PR fixes those bugs and enables support for the parallel aggregate HT scan. This enables queries that perform further operations on e.g. the result of large
DISTINCTorGROUP BYto run more in parallel.General Clean-up
CREATE TABLE ASandINSERT INTOstatements now use the same physical operator underneath (PhysicalInsert)SegmentTreeitself, and add more robust locking and more control over the locking of the SegmentTreeOptimisticDataWriterandLocalTableManagerclasses that take over certain logic of theLocalStorageclass, and add thread safety to the operations that is required now that we have parallel writesTableStatisticsinto theRowGroupCollection, rather than having this be maintained outside and passed into certain operationsParallel appends to table with index with transactionsto fail occasionally