Length kernel returns bytes not character length #156
Assignees
Labels
datafusion
Changes in the datafusion crate
good first issue
Good for newcomers
help wanted
Extra attention is needed
Comments
|
Comment from Mike Seddon(MikeSeddonAU) @ 2021-01-30T05:45:15.085+0000: The function comments have been updated as part of https://github.com/apache/arrow/pull/9243 to clarify that `length` returns `bytes` not number of characters. Comment from Mike Seddon(MikeSeddonAU) @ 2021-01-30T05:46:25.295+0000: This has been addressed in https://github.com/apache/arrow/pull/9243 with updated comments to clarify that this function returns number of bytes not number of characters. |
houqp
added
good first issue
|
I've had a look around and see the I could use some help with understanding this issue please -- what is the desired outcome -- should the test referenced above change to say Thanks. |
|
Hi @novemberkilo -- I think you are correct that this ticket is ill formed Specifically, since this ticket was filed, DataFusion appears to have both Closing this one as I don't think it is actionable anymore |
mustafasrepo
added a commit
that referenced
this issue
Dec 25, 2023
mustafasrepo
added a commit
that referenced
this issue
Dec 25, 2023
* Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * Initial comments on working * License update (#157) This extension adds Synnada license information to the existing one. * Adding comments * Update sort_hash_join.rs * After merge silent error * Change the query in HashJoin * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Update rat_exclude_files.txt * Clippy solving. * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * After merge * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Before clippy fmt etc. * lazy loading tables * mini test * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes * Before rebase merge * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Update utils.rs * Print deletion * Update Cargo.lock * Refactor for review * Working without slt * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Change in test folders * Update join_pipeline_selection.rs * Update utils.rs * Before clippy * Before SLT * Tests are passing and clippy OK. * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Resolve errors introduced during rebase * After merge * Update rat_exclude_files.txt * Comments visited * Synnada Streaming SQL Tests (#190) * Adds a new method to construct window function for the given input * For mustafa * Final * Update rat_exclude_files.txt * More commenting * Fix linter errors, compile errors after rebase, Update commit hashes * After merge refactors * Dir * Additional test for coverage * Update join_disable_repartition_joins.slt * Review changes, remove code duplicates * Update subdirectory hashes --------- Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
mustafasrepo
added a commit
that referenced
this issue
Dec 25, 2023
* [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Resolve errors introduced during rebase * Synnada Streaming SQL Tests (#190) * Adds a new method to construct window function for the given input * Protobuf implementations with roundrobin tests (#193) * Protobuf implementations with roundrobin * Proto * Update mod.rs * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Resolve errors introduced during rebase * Synnada Streaming SQL Tests (#190) * Adds a new method to construct window function for the given input * Protobuf implementations with roundrobin tests (#193) * Protobuf implementations with roundrobin * Proto * Update mod.rs * Fix linter errors, compile errors after rebase, Update commit hashes, regenerate proto * Rewrite Filter Predicate (#192) * Global join selection (#183) * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * Initial comments on working * License update (#157) This extension adds Synnada license information to the existing one. * Adding comments * Update sort_hash_join.rs * After merge silent error * Change the query in HashJoin * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Update rat_exclude_files.txt * Clippy solving. * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * After merge * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Before clippy fmt etc. * lazy loading tables * mini test * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes * Before rebase merge * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Update utils.rs * Print deletion * Update Cargo.lock * Refactor for review * Working without slt * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Change in test folders * Update join_pipeline_selection.rs * Update utils.rs * Before clippy * Before SLT * Tests are passing and clippy OK. * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Resolve errors introduced during rebase * After merge * Update rat_exclude_files.txt * Comments visited * Synnada Streaming SQL Tests (#190) * Adds a new method to construct window function for the given input * For mustafa * Final * Update rat_exclude_files.txt * More commenting * Fix linter errors, compile errors after rebase, Update commit hashes * After merge refactors * Dir * Additional test for coverage * Update join_disable_repartition_joins.slt * Review changes, remove code duplicates * Update subdirectory hashes --------- Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * work in progress * fix after merge * joins are ok * code cleaning * CaseExpr handling * tests are updated * Handling sequential projections * Simplifications * partition expr update * ProjectionPushdown becomes a rule.bash de * fix after merge * remove the subrule * tpch update * Minor comment changes * remove unnecessary state struct * coalesce batches does not let to push projection down * tpch changes removed * minor changes * addresses reviews * Update projection_pushdown.rs * minor * Review Part 1 * Simplify the API of plan handlers * Review Part 2 * Review Part 3 * Review Part 4 * Review Part 5 * Review Part 6 * Review Part 7 * Remove duplication of physical_expr matching * fix documentation * Minor changes * Take upstream changes --------- Co-authored-by: Metehan Yıldırım <100111937+metesynnada@users.noreply.github.com> Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
mustafasrepo
added a commit
that referenced
this issue
Dec 25, 2023
mustafasrepo
added a commit
that referenced
this issue
Dec 25, 2023
* Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * Initial comments on working * License update (#157) This extension adds Synnada license information to the existing one. * Adding comments * Update sort_hash_join.rs * After merge silent error * Change the query in HashJoin * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Update rat_exclude_files.txt * Clippy solving. * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * After merge * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Before clippy fmt etc. * lazy loading tables * mini test * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes * Before rebase merge * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Update utils.rs * Print deletion * Update Cargo.lock * Refactor for review * Working without slt * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Change in test folders * Update join_pipeline_selection.rs * Update utils.rs * Before clippy * Before SLT * Tests are passing and clippy OK. * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Resolve errors introduced during rebase * After merge * Update rat_exclude_files.txt * Comments visited * Synnada Streaming SQL Tests (#190) * Adds a new method to construct window function for the given input * For mustafa * Final * Update rat_exclude_files.txt * More commenting * Fix linter errors, compile errors after rebase, Update commit hashes * After merge refactors * Dir * Additional test for coverage * Update join_disable_repartition_joins.slt * Review changes, remove code duplicates * Update subdirectory hashes --------- Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
mustafasrepo
added a commit
that referenced
this issue
Dec 25, 2023
* [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Resolve errors introduced during rebase * Synnada Streaming SQL Tests (#190) * Adds a new method to construct window function for the given input * Protobuf implementations with roundrobin tests (#193) * Protobuf implementations with roundrobin * Proto * Update mod.rs * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Resolve errors introduced during rebase * Synnada Streaming SQL Tests (#190) * Adds a new method to construct window function for the given input * Protobuf implementations with roundrobin tests (#193) * Protobuf implementations with roundrobin * Proto * Update mod.rs * Fix linter errors, compile errors after rebase, Update commit hashes, regenerate proto * Rewrite Filter Predicate (#192) * Global join selection (#183) * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * Initial comments on working * License update (#157) This extension adds Synnada license information to the existing one. * Adding comments * Update sort_hash_join.rs * After merge silent error * Change the query in HashJoin * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Update rat_exclude_files.txt * Clippy solving. * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * After merge * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Before clippy fmt etc. * lazy loading tables * mini test * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes * Before rebase merge * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Update utils.rs * Print deletion * Update Cargo.lock * Refactor for review * Working without slt * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Change in test folders * Update join_pipeline_selection.rs * Update utils.rs * Before clippy * Before SLT * Tests are passing and clippy OK. * [GITHUB ACTION] Refactor for license and actions (#148) * Delete datafusion main publication * Adding licence information, refactoring prunibility issues * Update SYNNADA-CONTRIBUTIONS.txt * Update rat_exclude_files.txt * Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132) * Very initial test passing algorithm * Working except a minor bug in interval calculations * After clippy * Plan * initial implemantation * Before prune check ability is added. Order equivalence implementations will vanish after we send a seperate PR * minor changes * Fix bug, ordering equivalence random head * minor changes * Add ordering equivalence for sort merge join * Improvement on tests * Upstream changes * Add ordering equivalence for sort merge join * Fmt issues * Update comment * Add ordering equivalence support for hash join * Make 1 file * Code enhancements/comment improvements * Add projection cast handling * Fix output ordering for sort merge join * projection bug fix * Minor changes * minor changes * simplify sort_merge_join * Update equivalence implementation * Update test_utils.rs * Update cast implementation * More idiomatic code * After merge * Comments visisted * Add key swap according to the children orders * Refactoring * After merge refactor * Update sort_enforcement.rs * Update datafusion/core/src/physical_optimizer/join_selection.rs Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * Comments are applied * Feature/determine prunability (#139) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition * Determine prunability of tables for join operations (#90) * ready to review * license added * simplifications * simplifications * sort expr's are taken separately for each table * we can return the sort info of the expression now * check filter conditions * simplifications * simplifications * functions are implemented for SortInfo calculations * node specialized tableSide functions * NotImplemented errors are added, test comments are added * Comment change * Simplify comparison node calculations * Simplfications and better commenting * is_prunable function is updated with new Prunability function * Indices of sort expressions are updated with intermediate schema columns of the filter * Unused function is removed * Future-proof index updating * An if let check is removed * simplifications * Simplifications * simplifications * Change if condition --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * fix the tables' unboundedness --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> * Comment improvements and minor code improvements * Splitting the order based join selection * Update rat_exclude_files.txt * Revert "Feature/determine prunability (#139)" This reverts commit cf56105. * Commented --------- Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> * Bug fix: Fix lexicographical column search among provided ordering (#156) * License update (#157) This extension adds Synnada license information to the existing one. * Sliding Nested Join Algorithm (#142) * Sliding Hash Join Algorithm (SWHJ) (#147) * Fix errors introduced during rebase * Keep Track of Global Ordering Requirement (#165) * Prunability of Join Filter Physical Expressions (#161) * BinaryExpr Equivalence (#116) * Fix errors introduced during rebase * Support multiple ordered columns on joins and expression graph (#163) * SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171) * Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file * Resolve errors introduced during rebase * After merge * Update rat_exclude_files.txt * Comments visited * Synnada Streaming SQL Tests (#190) * Adds a new method to construct window function for the given input * For mustafa * Final * Update rat_exclude_files.txt * More commenting * Fix linter errors, compile errors after rebase, Update commit hashes * After merge refactors * Dir * Additional test for coverage * Update join_disable_repartition_joins.slt * Review changes, remove code duplicates * Update subdirectory hashes --------- Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com> Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com> * work in progress * fix after merge * joins are ok * code cleaning * CaseExpr handling * tests are updated * Handling sequential projections * Simplifications * partition expr update * ProjectionPushdown becomes a rule.bash de * fix after merge * remove the subrule * tpch update * Minor comment changes * remove unnecessary state struct * coalesce batches does not let to push projection down * tpch changes removed * minor changes * addresses reviews * Update projection_pushdown.rs * minor * Review Part 1 * Simplify the API of plan handlers * Review Part 2 * Review Part 3 * Review Part 4 * Review Part 5 * Review Part 6 * Review Part 7 * Remove duplication of physical_expr matching * fix documentation * Minor changes * Take upstream changes --------- Co-authored-by: Metehan Yıldırım <100111937+metesynnada@users.noreply.github.com> Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com> Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Note: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-11434
The rust
lengthkernel currently counts number of bytes/octets rather than characters given that Arrow uses UTF8 encoding.This means that the result of the
lengthkernel on a string likejoséwill be 5 bytes rather than 4 characters.The text was updated successfully, but these errors were encountered: