Fix dbtransfer and add unittests #330
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks! |
|
@hannesmuehleisen You are welcome! It's interesting for me to help this opensource project, even, still, with small changes. |
|
Also PRs that make small changes are very welcome! |
hawkfish
pushed a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 17, 2023
Add Python implementation of skip lists to libraries.
hawkfish
pushed a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 17, 2023
Convert basic scalar quantile window to use skip lists. Performance on 1e9 rows, single partition, 16 cores went from 11:11 to 00:36.
hawkfish
pushed a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 17, 2023
Convert scalar and list windowed quantile to use skip lists. Performance on 1e9 rows, single partition, 16 cores went from 11:11 to 00:36.
hawkfish
pushed a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 17, 2023
Create MST template for use in quantile and other aggregates.
hawkfish
pushed a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 17, 2023
* Define QuantileSortTree for quantile acceleration. * Add aggregate_wininit_t for constructing global windowing state. * Update aggregate_window_t to take optional global windowing state. * Add quantile WindowInit function (not used yet).
hawkfish
pushed a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 17, 2023
Call window initialisation if it is defined.
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 17, 2023
Switch quantile computations to use merge sort trees. MAD not converted yet.
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 17, 2023
Fix double decrement.
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 17, 2023
Add MIT License.
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 17, 2023
Toggle 32/64 bit versions.
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 17, 2023
Refactor, simplify and deploy to Median Absolute Deviation.
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 17, 2023
Plumb through statistics and extract min/max for ROWS.
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 17, 2023
Use frame expression statistics to choose between Skip Lists and Merge Sort Trees
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 17, 2023
Test statistics and handle no propagation situation.
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 17, 2023
Fix fanout misplacement.
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 17, 2023
Finish implementing and testing merge sort trees by supporting EXCLUDE/multiple frames.
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 17, 2023
Fix CI issues (formatting, includes)
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 18, 2023
Remove non-portable #pragma mark lines.
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 22, 2023
Standardise SubFrame typing.
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 22, 2023
Push subframes into the Merge Sort Tree.
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 23, 2023
Temporary hack around unnecessary validity mask construction in DataChunk::Append.
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 23, 2023
Try just fixing the validity mask allocation.
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 24, 2023
Find remaining shubs and zuuls.
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 26, 2023
Add a one element Node pool to avoid repeated new/delete calls.
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 30, 2023
Integrate Tania' feedback (thanks!)
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 30, 2023
Incorporate Carlo's review (thanks!)
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Oct 30, 2023
Fix signed/unsigned portability issue and extra include.
hawkfish
added a commit
to hawkfish/duckdb
that referenced
this pull request
Nov 1, 2023
Revert single-bit prng use as it trashes performance by 2x.
hawkfish
pushed a commit
to hawkfish/duckdb
that referenced
this pull request
Nov 3, 2023
Clean them up in response to PR feedback. Note that ListWindow is being naughty and modifying the partition data, which is not thread-safe.
hawkfish
pushed a commit
to hawkfish/duckdb
that referenced
this pull request
Nov 3, 2023
Restore the statistics API to use signed values. Add coverage tests for quantile variants.
Mytherin
added a commit
that referenced
this pull request
Nov 8, 2023
Internal #330: Quantile Performance
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I have implemented several tests for
dbtransfertool and made several fixes inQueryDatabase: add passingresult_typesandtypeidstoresultobject ofMaterializedQueryResulttype.