Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Datafusion] Error with CAST: Unsupported SQL type Time #193

Closed
alamb opened this issue Apr 26, 2021 · 4 comments
Closed

[Datafusion] Error with CAST: Unsupported SQL type Time #193

alamb opened this issue Apr 26, 2021 · 4 comments
Labels
bug Something isn't working datafusion Changes in the datafusion crate

Comments

@alamb
Copy link
Contributor

alamb commented Apr 26, 2021

Note: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-12232

select cast(timestamp as Time) from foo limit 5;
NotImplemented("Unsupported SQL type Time")

@alamb alamb added the datafusion Changes in the datafusion crate label Apr 26, 2021
@alamb
Copy link
Contributor Author

alamb commented Apr 26, 2021

Comment from Evan Chan(velvia) @ 2021-04-14T20:47:50.722+0000:

After discussion for the above PR, seems like the above PR does not really solve this issue, and thus will be amended or left.  A related bug/feature will be filed instead.

@andygrove andygrove added the bug Something isn't working label May 1, 2021
@velvia
Copy link
Contributor

velvia commented May 17, 2021

I'll leave this ticket for tracking implementing CAST() for Time type, and the discussion from the above PR I'll open a separate issue for to_timestamp_x

@Jefffrey
Copy link
Contributor

Issue seems resolved as of 406c108

jeffrey:~/Code/arrow-datafusion/datafusion-cli$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.10s
     Running `/media/jeffrey/1tb_860evo_ssd/.cargo_target_cache/debug/datafusion-cli`
DataFusion CLI v14.0.0
❯ select cast(b.timestamp as time) from (select now() as timestamp) b;
+--------------------+
| b.timestamp        |
+--------------------+
| 11:10:31.506762991 |
+--------------------+
1 row in set. Query took 0.004 seconds.
❯ explain select cast(b.timestamp as time) from (select now() as timestamp) b;
+---------------+---------------------------------------------------------------------------------------+
| plan_type     | plan                                                                                  |
+---------------+---------------------------------------------------------------------------------------+
| logical_plan  | Projection: CAST(b.timestamp AS Time64(Nanosecond))                                   |
|               |   Projection: timestamp, alias=b                                                      |
|               |     Projection: TimestampNanosecond(1668597032907296503, Some("+00:00")) AS timestamp |
|               |       EmptyRelation                                                                   |
| physical_plan | ProjectionExec: expr=[CAST(timestamp@0 AS Time64(Nanosecond)) as b.timestamp]         |
|               |   ProjectionExec: expr=[timestamp@0 as timestamp]                                     |
|               |     ProjectionExec: expr=[1668597032907296503 as timestamp]                           |
|               |       EmptyExec: produce_one_row=true                                                 |
|               |                                                                                       |
+---------------+---------------------------------------------------------------------------------------+
2 rows in set. Query took 0.005 seconds.
❯

@alamb
Copy link
Contributor Author

alamb commented Nov 16, 2022

🎉 thanks for checking @Jefffrey

@alamb alamb closed this as completed Nov 16, 2022
mustafasrepo pushed a commit that referenced this issue Dec 25, 2023
* Protobuf implementations with roundrobin

* Proto

* Update mod.rs
mustafasrepo added a commit that referenced this issue Dec 25, 2023
* [GITHUB ACTION] Refactor for license and actions (#148)

* Delete datafusion main publication

* Adding licence information, refactoring prunibility issues

* Update SYNNADA-CONTRIBUTIONS.txt

* Update rat_exclude_files.txt

* Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132)

* Very initial test passing algorithm

* Working except a minor bug in interval calculations

* After clippy

* Plan

* initial implemantation

* Before prune check ability is added.

Order equivalence implementations will vanish after we send a seperate PR

* minor changes

* Fix bug, ordering equivalence random head

* minor changes

* Add ordering equivalence for sort merge join

* Improvement on tests

* Upstream changes

* Add ordering equivalence for sort merge join

* Fmt issues

* Update comment

* Add ordering equivalence support for hash join

* Make 1 file

* Code enhancements/comment improvements

* Add projection cast handling

* Fix output ordering for sort merge join

* projection bug fix

* Minor changes

* minor changes

* simplify sort_merge_join

* Update equivalence implementation

* Update test_utils.rs

* Update cast implementation

* More idiomatic code

* After merge

* Comments visisted

* Add key swap according to the children orders

* Refactoring

* After merge refactor

* Update sort_enforcement.rs

* Update datafusion/core/src/physical_optimizer/join_selection.rs

Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* Comments are applied

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* Comment improvements and minor code improvements

* Splitting the order based join selection

* Update rat_exclude_files.txt

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Commented

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>

* Bug fix: Fix lexicographical column search among provided ordering (#156)

* License update (#157)

This extension adds Synnada license information to the existing one.

* Sliding Nested Join Algorithm (#142)

* Sliding Hash Join Algorithm (SWHJ) (#147)

* Fix errors introduced during rebase

* Keep Track of Global Ordering Requirement (#165)

* Prunability of Join Filter Physical Expressions (#161)

* BinaryExpr Equivalence (#116)

* Fix errors introduced during rebase

* Support multiple ordered columns on joins and expression graph (#163)

* SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171)

* Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file

* Resolve errors introduced during rebase

* Synnada Streaming SQL Tests (#190)

* Adds a new method to construct window function for the given input

* Protobuf implementations with roundrobin tests (#193)

* Protobuf implementations with roundrobin

* Proto

* Update mod.rs

* [GITHUB ACTION] Refactor for license and actions (#148)

* Delete datafusion main publication

* Adding licence information, refactoring prunibility issues

* Update SYNNADA-CONTRIBUTIONS.txt

* Update rat_exclude_files.txt

* Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132)

* Very initial test passing algorithm

* Working except a minor bug in interval calculations

* After clippy

* Plan

* initial implemantation

* Before prune check ability is added.

Order equivalence implementations will vanish after we send a seperate PR

* minor changes

* Fix bug, ordering equivalence random head

* minor changes

* Add ordering equivalence for sort merge join

* Improvement on tests

* Upstream changes

* Add ordering equivalence for sort merge join

* Fmt issues

* Update comment

* Add ordering equivalence support for hash join

* Make 1 file

* Code enhancements/comment improvements

* Add projection cast handling

* Fix output ordering for sort merge join

* projection bug fix

* Minor changes

* minor changes

* simplify sort_merge_join

* Update equivalence implementation

* Update test_utils.rs

* Update cast implementation

* More idiomatic code

* After merge

* Comments visisted

* Add key swap according to the children orders

* Refactoring

* After merge refactor

* Update sort_enforcement.rs

* Update datafusion/core/src/physical_optimizer/join_selection.rs

Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* Comments are applied

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* Comment improvements and minor code improvements

* Splitting the order based join selection

* Update rat_exclude_files.txt

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Commented

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>

* Bug fix: Fix lexicographical column search among provided ordering (#156)

* License update (#157)

This extension adds Synnada license information to the existing one.

* Sliding Nested Join Algorithm (#142)

* Sliding Hash Join Algorithm (SWHJ) (#147)

* Fix errors introduced during rebase

* Keep Track of Global Ordering Requirement (#165)

* Prunability of Join Filter Physical Expressions (#161)

* BinaryExpr Equivalence (#116)

* Fix errors introduced during rebase

* Support multiple ordered columns on joins and expression graph (#163)

* SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171)

* Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file

* Resolve errors introduced during rebase

* Synnada Streaming SQL Tests (#190)

* Adds a new method to construct window function for the given input

* Protobuf implementations with roundrobin tests (#193)

* Protobuf implementations with roundrobin

* Proto

* Update mod.rs

* Fix linter errors, compile errors after rebase, Update commit hashes, regenerate proto

* Rewrite Filter Predicate (#192)

* Global join selection (#183)

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* [GITHUB ACTION] Refactor for license and actions (#148)

* Delete datafusion main publication

* Adding licence information, refactoring prunibility issues

* Update SYNNADA-CONTRIBUTIONS.txt

* Update rat_exclude_files.txt

* Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132)

* Very initial test passing algorithm

* Working except a minor bug in interval calculations

* After clippy

* Plan

* initial implemantation

* Before prune check ability is added.

Order equivalence implementations will vanish after we send a seperate PR

* minor changes

* Fix bug, ordering equivalence random head

* minor changes

* Add ordering equivalence for sort merge join

* Improvement on tests

* Upstream changes

* Add ordering equivalence for sort merge join

* Fmt issues

* Update comment

* Add ordering equivalence support for hash join

* Make 1 file

* Code enhancements/comment improvements

* Add projection cast handling

* Fix output ordering for sort merge join

* projection bug fix

* Minor changes

* minor changes

* simplify sort_merge_join

* Update equivalence implementation

* Update test_utils.rs

* Update cast implementation

* More idiomatic code

* After merge

* Comments visisted

* Add key swap according to the children orders

* Refactoring

* After merge refactor

* Update sort_enforcement.rs

* Update datafusion/core/src/physical_optimizer/join_selection.rs

Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* Comments are applied

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* Comment improvements and minor code improvements

* Splitting the order based join selection

* Update rat_exclude_files.txt

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Commented

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>

* Bug fix: Fix lexicographical column search among provided ordering (#156)

* Initial comments on working

* License update (#157)

This extension adds Synnada license information to the existing one.

* Adding comments

* Update sort_hash_join.rs

* After merge silent error

* Change the query in HashJoin

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Update rat_exclude_files.txt

* Clippy solving.

* [GITHUB ACTION] Refactor for license and actions (#148)

* Delete datafusion main publication

* Adding licence information, refactoring prunibility issues

* Update SYNNADA-CONTRIBUTIONS.txt

* Update rat_exclude_files.txt

* Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132)

* Very initial test passing algorithm

* Working except a minor bug in interval calculations

* After clippy

* Plan

* initial implemantation

* Before prune check ability is added.

Order equivalence implementations will vanish after we send a seperate PR

* minor changes

* Fix bug, ordering equivalence random head

* minor changes

* Add ordering equivalence for sort merge join

* Improvement on tests

* Upstream changes

* Add ordering equivalence for sort merge join

* Fmt issues

* Update comment

* Add ordering equivalence support for hash join

* Make 1 file

* Code enhancements/comment improvements

* Add projection cast handling

* Fix output ordering for sort merge join

* projection bug fix

* Minor changes

* minor changes

* simplify sort_merge_join

* Update equivalence implementation

* Update test_utils.rs

* Update cast implementation

* More idiomatic code

* After merge

* Comments visisted

* Add key swap according to the children orders

* Refactoring

* After merge refactor

* Update sort_enforcement.rs

* Update datafusion/core/src/physical_optimizer/join_selection.rs

Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* Comments are applied

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* Comment improvements and minor code improvements

* Splitting the order based join selection

* Update rat_exclude_files.txt

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Commented

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>

* Bug fix: Fix lexicographical column search among provided ordering (#156)

* License update (#157)

This extension adds Synnada license information to the existing one.

* [GITHUB ACTION] Refactor for license and actions (#148)

* Delete datafusion main publication

* Adding licence information, refactoring prunibility issues

* Update SYNNADA-CONTRIBUTIONS.txt

* Update rat_exclude_files.txt

* Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132)

* Very initial test passing algorithm

* Working except a minor bug in interval calculations

* After clippy

* Plan

* initial implemantation

* Before prune check ability is added.

Order equivalence implementations will vanish after we send a seperate PR

* minor changes

* Fix bug, ordering equivalence random head

* minor changes

* Add ordering equivalence for sort merge join

* Improvement on tests

* Upstream changes

* Add ordering equivalence for sort merge join

* Fmt issues

* Update comment

* Add ordering equivalence support for hash join

* Make 1 file

* Code enhancements/comment improvements

* Add projection cast handling

* Fix output ordering for sort merge join

* projection bug fix

* Minor changes

* minor changes

* simplify sort_merge_join

* Update equivalence implementation

* Update test_utils.rs

* Update cast implementation

* More idiomatic code

* After merge

* Comments visisted

* Add key swap according to the children orders

* Refactoring

* After merge refactor

* Update sort_enforcement.rs

* Update datafusion/core/src/physical_optimizer/join_selection.rs

Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* Comments are applied

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* Comment improvements and minor code improvements

* Splitting the order based join selection

* Update rat_exclude_files.txt

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Commented

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>

* Bug fix: Fix lexicographical column search among provided ordering (#156)

* License update (#157)

This extension adds Synnada license information to the existing one.

* Sliding Nested Join Algorithm (#142)

* Sliding Hash Join Algorithm (SWHJ) (#147)

* Fix errors introduced during rebase

* [GITHUB ACTION] Refactor for license and actions (#148)

* Delete datafusion main publication

* Adding licence information, refactoring prunibility issues

* Update SYNNADA-CONTRIBUTIONS.txt

* Update rat_exclude_files.txt

* Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132)

* Very initial test passing algorithm

* Working except a minor bug in interval calculations

* After clippy

* Plan

* initial implemantation

* Before prune check ability is added.

Order equivalence implementations will vanish after we send a seperate PR

* minor changes

* Fix bug, ordering equivalence random head

* minor changes

* Add ordering equivalence for sort merge join

* Improvement on tests

* Upstream changes

* Add ordering equivalence for sort merge join

* Fmt issues

* Update comment

* Add ordering equivalence support for hash join

* Make 1 file

* Code enhancements/comment improvements

* Add projection cast handling

* Fix output ordering for sort merge join

* projection bug fix

* Minor changes

* minor changes

* simplify sort_merge_join

* Update equivalence implementation

* Update test_utils.rs

* Update cast implementation

* More idiomatic code

* After merge

* Comments visisted

* Add key swap according to the children orders

* Refactoring

* After merge refactor

* Update sort_enforcement.rs

* Update datafusion/core/src/physical_optimizer/join_selection.rs

Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* Comments are applied

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* Comment improvements and minor code improvements

* Splitting the order based join selection

* Update rat_exclude_files.txt

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Commented

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>

* Bug fix: Fix lexicographical column search among provided ordering (#156)

* License update (#157)

This extension adds Synnada license information to the existing one.

* Sliding Nested Join Algorithm (#142)

* Sliding Hash Join Algorithm (SWHJ) (#147)

* Fix errors introduced during rebase

* Keep Track of Global Ordering Requirement (#165)

* Prunability of Join Filter Physical Expressions (#161)

* BinaryExpr Equivalence (#116)

* Fix errors introduced during rebase

* Support multiple ordered columns on joins and expression graph (#163)

* After merge

* SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171)

* Before clippy fmt etc.

* lazy loading tables

* mini test

* [GITHUB ACTION] Refactor for license and actions (#148)

* Delete datafusion main publication

* Adding licence information, refactoring prunibility issues

* Update SYNNADA-CONTRIBUTIONS.txt

* Update rat_exclude_files.txt

* Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132)

* Very initial test passing algorithm

* Working except a minor bug in interval calculations

* After clippy

* Plan

* initial implemantation

* Before prune check ability is added.

Order equivalence implementations will vanish after we send a seperate PR

* minor changes

* Fix bug, ordering equivalence random head

* minor changes

* Add ordering equivalence for sort merge join

* Improvement on tests

* Upstream changes

* Add ordering equivalence for sort merge join

* Fmt issues

* Update comment

* Add ordering equivalence support for hash join

* Make 1 file

* Code enhancements/comment improvements

* Add projection cast handling

* Fix output ordering for sort merge join

* projection bug fix

* Minor changes

* minor changes

* simplify sort_merge_join

* Update equivalence implementation

* Update test_utils.rs

* Update cast implementation

* More idiomatic code

* After merge

* Comments visisted

* Add key swap according to the children orders

* Refactoring

* After merge refactor

* Update sort_enforcement.rs

* Update datafusion/core/src/physical_optimizer/join_selection.rs

Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* Comments are applied

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* Comment improvements and minor code improvements

* Splitting the order based join selection

* Update rat_exclude_files.txt

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Commented

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>

* Bug fix: Fix lexicographical column search among provided ordering (#156)

* License update (#157)

This extension adds Synnada license information to the existing one.

* Sliding Nested Join Algorithm (#142)

* Sliding Hash Join Algorithm (SWHJ) (#147)

* Fix errors introduced during rebase

* Keep Track of Global Ordering Requirement (#165)

* Prunability of Join Filter Physical Expressions (#161)

* BinaryExpr Equivalence (#116)

* Fix errors introduced during rebase

* Support multiple ordered columns on joins and expression graph (#163)

* SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171)

* Add license, add contribution hash commits, minor changes

* Before rebase merge

* Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file

* Update utils.rs

* Print deletion

* Update Cargo.lock

* Refactor for review

* Working without slt

* [GITHUB ACTION] Refactor for license and actions (#148)

* Delete datafusion main publication

* Adding licence information, refactoring prunibility issues

* Update SYNNADA-CONTRIBUTIONS.txt

* Update rat_exclude_files.txt

* Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132)

* Very initial test passing algorithm

* Working except a minor bug in interval calculations

* After clippy

* Plan

* initial implemantation

* Before prune check ability is added.

Order equivalence implementations will vanish after we send a seperate PR

* minor changes

* Fix bug, ordering equivalence random head

* minor changes

* Add ordering equivalence for sort merge join

* Improvement on tests

* Upstream changes

* Add ordering equivalence for sort merge join

* Fmt issues

* Update comment

* Add ordering equivalence support for hash join

* Make 1 file

* Code enhancements/comment improvements

* Add projection cast handling

* Fix output ordering for sort merge join

* projection bug fix

* Minor changes

* minor changes

* simplify sort_merge_join

* Update equivalence implementation

* Update test_utils.rs

* Update cast implementation

* More idiomatic code

* After merge

* Comments visisted

* Add key swap according to the children orders

* Refactoring

* After merge refactor

* Update sort_enforcement.rs

* Update datafusion/core/src/physical_optimizer/join_selection.rs

Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* Comments are applied

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* Comment improvements and minor code improvements

* Splitting the order based join selection

* Update rat_exclude_files.txt

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Commented

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>

* Bug fix: Fix lexicographical column search among provided ordering (#156)

* License update (#157)

This extension adds Synnada license information to the existing one.

* Sliding Nested Join Algorithm (#142)

* Sliding Hash Join Algorithm (SWHJ) (#147)

* Fix errors introduced during rebase

* Keep Track of Global Ordering Requirement (#165)

* Prunability of Join Filter Physical Expressions (#161)

* BinaryExpr Equivalence (#116)

* Fix errors introduced during rebase

* Support multiple ordered columns on joins and expression graph (#163)

* SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171)

* Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file

* Change in test folders

* Update join_pipeline_selection.rs

* Update utils.rs

* Before clippy

* Before SLT

* Tests are passing and clippy OK.

* [GITHUB ACTION] Refactor for license and actions (#148)

* Delete datafusion main publication

* Adding licence information, refactoring prunibility issues

* Update SYNNADA-CONTRIBUTIONS.txt

* Update rat_exclude_files.txt

* Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132)

* Very initial test passing algorithm

* Working except a minor bug in interval calculations

* After clippy

* Plan

* initial implemantation

* Before prune check ability is added.

Order equivalence implementations will vanish after we send a seperate PR

* minor changes

* Fix bug, ordering equivalence random head

* minor changes

* Add ordering equivalence for sort merge join

* Improvement on tests

* Upstream changes

* Add ordering equivalence for sort merge join

* Fmt issues

* Update comment

* Add ordering equivalence support for hash join

* Make 1 file

* Code enhancements/comment improvements

* Add projection cast handling

* Fix output ordering for sort merge join

* projection bug fix

* Minor changes

* minor changes

* simplify sort_merge_join

* Update equivalence implementation

* Update test_utils.rs

* Update cast implementation

* More idiomatic code

* After merge

* Comments visisted

* Add key swap according to the children orders

* Refactoring

* After merge refactor

* Update sort_enforcement.rs

* Update datafusion/core/src/physical_optimizer/join_selection.rs

Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* Comments are applied

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* Comment improvements and minor code improvements

* Splitting the order based join selection

* Update rat_exclude_files.txt

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Commented

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>

* Bug fix: Fix lexicographical column search among provided ordering (#156)

* License update (#157)

This extension adds Synnada license information to the existing one.

* Sliding Nested Join Algorithm (#142)

* Sliding Hash Join Algorithm (SWHJ) (#147)

* Fix errors introduced during rebase

* Keep Track of Global Ordering Requirement (#165)

* Prunability of Join Filter Physical Expressions (#161)

* BinaryExpr Equivalence (#116)

* Fix errors introduced during rebase

* Support multiple ordered columns on joins and expression graph (#163)

* SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171)

* Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file

* Resolve errors introduced during rebase

* After merge

* Update rat_exclude_files.txt

* Comments visited

* Synnada Streaming SQL Tests (#190)

* Adds a new method to construct window function for the given input

* For mustafa

* Final

* Update rat_exclude_files.txt

* More commenting

* Fix linter errors, compile errors after rebase, Update commit hashes

* After merge refactors

* Dir

* Additional test for coverage

* Update join_disable_repartition_joins.slt

* Review changes, remove code duplicates

* Update subdirectory hashes

---------

Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>
Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* work in progress

* fix after merge

* joins are ok

* code cleaning

* CaseExpr handling

* tests are updated

* Handling sequential projections

* Simplifications

* partition expr update

* ProjectionPushdown becomes a rule.bash de

* fix after merge

* remove the subrule

* tpch update

* Minor comment changes

* remove unnecessary state struct

* coalesce batches does not let to push projection down

* tpch changes removed

* minor changes

* addresses reviews

* Update projection_pushdown.rs

* minor

* Review Part 1

* Simplify the API of plan handlers

* Review Part 2

* Review Part 3

* Review Part 4

* Review Part 5

* Review Part 6

* Review Part 7

* Remove duplication of physical_expr matching

* fix documentation

* Minor changes

* Take upstream changes

---------

Co-authored-by: Metehan Yıldırım <100111937+metesynnada@users.noreply.github.com>
Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
mustafasrepo pushed a commit that referenced this issue Dec 25, 2023
* Protobuf implementations with roundrobin

* Proto

* Update mod.rs
mustafasrepo added a commit that referenced this issue Dec 25, 2023
* [GITHUB ACTION] Refactor for license and actions (#148)

* Delete datafusion main publication

* Adding licence information, refactoring prunibility issues

* Update SYNNADA-CONTRIBUTIONS.txt

* Update rat_exclude_files.txt

* Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132)

* Very initial test passing algorithm

* Working except a minor bug in interval calculations

* After clippy

* Plan

* initial implemantation

* Before prune check ability is added.

Order equivalence implementations will vanish after we send a seperate PR

* minor changes

* Fix bug, ordering equivalence random head

* minor changes

* Add ordering equivalence for sort merge join

* Improvement on tests

* Upstream changes

* Add ordering equivalence for sort merge join

* Fmt issues

* Update comment

* Add ordering equivalence support for hash join

* Make 1 file

* Code enhancements/comment improvements

* Add projection cast handling

* Fix output ordering for sort merge join

* projection bug fix

* Minor changes

* minor changes

* simplify sort_merge_join

* Update equivalence implementation

* Update test_utils.rs

* Update cast implementation

* More idiomatic code

* After merge

* Comments visisted

* Add key swap according to the children orders

* Refactoring

* After merge refactor

* Update sort_enforcement.rs

* Update datafusion/core/src/physical_optimizer/join_selection.rs

Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* Comments are applied

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* Comment improvements and minor code improvements

* Splitting the order based join selection

* Update rat_exclude_files.txt

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Commented

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>

* Bug fix: Fix lexicographical column search among provided ordering (#156)

* License update (#157)

This extension adds Synnada license information to the existing one.

* Sliding Nested Join Algorithm (#142)

* Sliding Hash Join Algorithm (SWHJ) (#147)

* Fix errors introduced during rebase

* Keep Track of Global Ordering Requirement (#165)

* Prunability of Join Filter Physical Expressions (#161)

* BinaryExpr Equivalence (#116)

* Fix errors introduced during rebase

* Support multiple ordered columns on joins and expression graph (#163)

* SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171)

* Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file

* Resolve errors introduced during rebase

* Synnada Streaming SQL Tests (#190)

* Adds a new method to construct window function for the given input

* Protobuf implementations with roundrobin tests (#193)

* Protobuf implementations with roundrobin

* Proto

* Update mod.rs

* [GITHUB ACTION] Refactor for license and actions (#148)

* Delete datafusion main publication

* Adding licence information, refactoring prunibility issues

* Update SYNNADA-CONTRIBUTIONS.txt

* Update rat_exclude_files.txt

* Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132)

* Very initial test passing algorithm

* Working except a minor bug in interval calculations

* After clippy

* Plan

* initial implemantation

* Before prune check ability is added.

Order equivalence implementations will vanish after we send a seperate PR

* minor changes

* Fix bug, ordering equivalence random head

* minor changes

* Add ordering equivalence for sort merge join

* Improvement on tests

* Upstream changes

* Add ordering equivalence for sort merge join

* Fmt issues

* Update comment

* Add ordering equivalence support for hash join

* Make 1 file

* Code enhancements/comment improvements

* Add projection cast handling

* Fix output ordering for sort merge join

* projection bug fix

* Minor changes

* minor changes

* simplify sort_merge_join

* Update equivalence implementation

* Update test_utils.rs

* Update cast implementation

* More idiomatic code

* After merge

* Comments visisted

* Add key swap according to the children orders

* Refactoring

* After merge refactor

* Update sort_enforcement.rs

* Update datafusion/core/src/physical_optimizer/join_selection.rs

Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* Comments are applied

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* Comment improvements and minor code improvements

* Splitting the order based join selection

* Update rat_exclude_files.txt

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Commented

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>

* Bug fix: Fix lexicographical column search among provided ordering (#156)

* License update (#157)

This extension adds Synnada license information to the existing one.

* Sliding Nested Join Algorithm (#142)

* Sliding Hash Join Algorithm (SWHJ) (#147)

* Fix errors introduced during rebase

* Keep Track of Global Ordering Requirement (#165)

* Prunability of Join Filter Physical Expressions (#161)

* BinaryExpr Equivalence (#116)

* Fix errors introduced during rebase

* Support multiple ordered columns on joins and expression graph (#163)

* SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171)

* Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file

* Resolve errors introduced during rebase

* Synnada Streaming SQL Tests (#190)

* Adds a new method to construct window function for the given input

* Protobuf implementations with roundrobin tests (#193)

* Protobuf implementations with roundrobin

* Proto

* Update mod.rs

* Fix linter errors, compile errors after rebase, Update commit hashes, regenerate proto

* Rewrite Filter Predicate (#192)

* Global join selection (#183)

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* [GITHUB ACTION] Refactor for license and actions (#148)

* Delete datafusion main publication

* Adding licence information, refactoring prunibility issues

* Update SYNNADA-CONTRIBUTIONS.txt

* Update rat_exclude_files.txt

* Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132)

* Very initial test passing algorithm

* Working except a minor bug in interval calculations

* After clippy

* Plan

* initial implemantation

* Before prune check ability is added.

Order equivalence implementations will vanish after we send a seperate PR

* minor changes

* Fix bug, ordering equivalence random head

* minor changes

* Add ordering equivalence for sort merge join

* Improvement on tests

* Upstream changes

* Add ordering equivalence for sort merge join

* Fmt issues

* Update comment

* Add ordering equivalence support for hash join

* Make 1 file

* Code enhancements/comment improvements

* Add projection cast handling

* Fix output ordering for sort merge join

* projection bug fix

* Minor changes

* minor changes

* simplify sort_merge_join

* Update equivalence implementation

* Update test_utils.rs

* Update cast implementation

* More idiomatic code

* After merge

* Comments visisted

* Add key swap according to the children orders

* Refactoring

* After merge refactor

* Update sort_enforcement.rs

* Update datafusion/core/src/physical_optimizer/join_selection.rs

Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* Comments are applied

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* Comment improvements and minor code improvements

* Splitting the order based join selection

* Update rat_exclude_files.txt

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Commented

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>

* Bug fix: Fix lexicographical column search among provided ordering (#156)

* Initial comments on working

* License update (#157)

This extension adds Synnada license information to the existing one.

* Adding comments

* Update sort_hash_join.rs

* After merge silent error

* Change the query in HashJoin

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Update rat_exclude_files.txt

* Clippy solving.

* [GITHUB ACTION] Refactor for license and actions (#148)

* Delete datafusion main publication

* Adding licence information, refactoring prunibility issues

* Update SYNNADA-CONTRIBUTIONS.txt

* Update rat_exclude_files.txt

* Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132)

* Very initial test passing algorithm

* Working except a minor bug in interval calculations

* After clippy

* Plan

* initial implemantation

* Before prune check ability is added.

Order equivalence implementations will vanish after we send a seperate PR

* minor changes

* Fix bug, ordering equivalence random head

* minor changes

* Add ordering equivalence for sort merge join

* Improvement on tests

* Upstream changes

* Add ordering equivalence for sort merge join

* Fmt issues

* Update comment

* Add ordering equivalence support for hash join

* Make 1 file

* Code enhancements/comment improvements

* Add projection cast handling

* Fix output ordering for sort merge join

* projection bug fix

* Minor changes

* minor changes

* simplify sort_merge_join

* Update equivalence implementation

* Update test_utils.rs

* Update cast implementation

* More idiomatic code

* After merge

* Comments visisted

* Add key swap according to the children orders

* Refactoring

* After merge refactor

* Update sort_enforcement.rs

* Update datafusion/core/src/physical_optimizer/join_selection.rs

Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* Comments are applied

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* Comment improvements and minor code improvements

* Splitting the order based join selection

* Update rat_exclude_files.txt

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Commented

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>

* Bug fix: Fix lexicographical column search among provided ordering (#156)

* License update (#157)

This extension adds Synnada license information to the existing one.

* [GITHUB ACTION] Refactor for license and actions (#148)

* Delete datafusion main publication

* Adding licence information, refactoring prunibility issues

* Update SYNNADA-CONTRIBUTIONS.txt

* Update rat_exclude_files.txt

* Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132)

* Very initial test passing algorithm

* Working except a minor bug in interval calculations

* After clippy

* Plan

* initial implemantation

* Before prune check ability is added.

Order equivalence implementations will vanish after we send a seperate PR

* minor changes

* Fix bug, ordering equivalence random head

* minor changes

* Add ordering equivalence for sort merge join

* Improvement on tests

* Upstream changes

* Add ordering equivalence for sort merge join

* Fmt issues

* Update comment

* Add ordering equivalence support for hash join

* Make 1 file

* Code enhancements/comment improvements

* Add projection cast handling

* Fix output ordering for sort merge join

* projection bug fix

* Minor changes

* minor changes

* simplify sort_merge_join

* Update equivalence implementation

* Update test_utils.rs

* Update cast implementation

* More idiomatic code

* After merge

* Comments visisted

* Add key swap according to the children orders

* Refactoring

* After merge refactor

* Update sort_enforcement.rs

* Update datafusion/core/src/physical_optimizer/join_selection.rs

Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* Comments are applied

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* Comment improvements and minor code improvements

* Splitting the order based join selection

* Update rat_exclude_files.txt

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Commented

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>

* Bug fix: Fix lexicographical column search among provided ordering (#156)

* License update (#157)

This extension adds Synnada license information to the existing one.

* Sliding Nested Join Algorithm (#142)

* Sliding Hash Join Algorithm (SWHJ) (#147)

* Fix errors introduced during rebase

* [GITHUB ACTION] Refactor for license and actions (#148)

* Delete datafusion main publication

* Adding licence information, refactoring prunibility issues

* Update SYNNADA-CONTRIBUTIONS.txt

* Update rat_exclude_files.txt

* Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132)

* Very initial test passing algorithm

* Working except a minor bug in interval calculations

* After clippy

* Plan

* initial implemantation

* Before prune check ability is added.

Order equivalence implementations will vanish after we send a seperate PR

* minor changes

* Fix bug, ordering equivalence random head

* minor changes

* Add ordering equivalence for sort merge join

* Improvement on tests

* Upstream changes

* Add ordering equivalence for sort merge join

* Fmt issues

* Update comment

* Add ordering equivalence support for hash join

* Make 1 file

* Code enhancements/comment improvements

* Add projection cast handling

* Fix output ordering for sort merge join

* projection bug fix

* Minor changes

* minor changes

* simplify sort_merge_join

* Update equivalence implementation

* Update test_utils.rs

* Update cast implementation

* More idiomatic code

* After merge

* Comments visisted

* Add key swap according to the children orders

* Refactoring

* After merge refactor

* Update sort_enforcement.rs

* Update datafusion/core/src/physical_optimizer/join_selection.rs

Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* Comments are applied

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* Comment improvements and minor code improvements

* Splitting the order based join selection

* Update rat_exclude_files.txt

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Commented

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>

* Bug fix: Fix lexicographical column search among provided ordering (#156)

* License update (#157)

This extension adds Synnada license information to the existing one.

* Sliding Nested Join Algorithm (#142)

* Sliding Hash Join Algorithm (SWHJ) (#147)

* Fix errors introduced during rebase

* Keep Track of Global Ordering Requirement (#165)

* Prunability of Join Filter Physical Expressions (#161)

* BinaryExpr Equivalence (#116)

* Fix errors introduced during rebase

* Support multiple ordered columns on joins and expression graph (#163)

* After merge

* SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171)

* Before clippy fmt etc.

* lazy loading tables

* mini test

* [GITHUB ACTION] Refactor for license and actions (#148)

* Delete datafusion main publication

* Adding licence information, refactoring prunibility issues

* Update SYNNADA-CONTRIBUTIONS.txt

* Update rat_exclude_files.txt

* Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132)

* Very initial test passing algorithm

* Working except a minor bug in interval calculations

* After clippy

* Plan

* initial implemantation

* Before prune check ability is added.

Order equivalence implementations will vanish after we send a seperate PR

* minor changes

* Fix bug, ordering equivalence random head

* minor changes

* Add ordering equivalence for sort merge join

* Improvement on tests

* Upstream changes

* Add ordering equivalence for sort merge join

* Fmt issues

* Update comment

* Add ordering equivalence support for hash join

* Make 1 file

* Code enhancements/comment improvements

* Add projection cast handling

* Fix output ordering for sort merge join

* projection bug fix

* Minor changes

* minor changes

* simplify sort_merge_join

* Update equivalence implementation

* Update test_utils.rs

* Update cast implementation

* More idiomatic code

* After merge

* Comments visisted

* Add key swap according to the children orders

* Refactoring

* After merge refactor

* Update sort_enforcement.rs

* Update datafusion/core/src/physical_optimizer/join_selection.rs

Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* Comments are applied

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* Comment improvements and minor code improvements

* Splitting the order based join selection

* Update rat_exclude_files.txt

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Commented

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>

* Bug fix: Fix lexicographical column search among provided ordering (#156)

* License update (#157)

This extension adds Synnada license information to the existing one.

* Sliding Nested Join Algorithm (#142)

* Sliding Hash Join Algorithm (SWHJ) (#147)

* Fix errors introduced during rebase

* Keep Track of Global Ordering Requirement (#165)

* Prunability of Join Filter Physical Expressions (#161)

* BinaryExpr Equivalence (#116)

* Fix errors introduced during rebase

* Support multiple ordered columns on joins and expression graph (#163)

* SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171)

* Add license, add contribution hash commits, minor changes

* Before rebase merge

* Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file

* Update utils.rs

* Print deletion

* Update Cargo.lock

* Refactor for review

* Working without slt

* [GITHUB ACTION] Refactor for license and actions (#148)

* Delete datafusion main publication

* Adding licence information, refactoring prunibility issues

* Update SYNNADA-CONTRIBUTIONS.txt

* Update rat_exclude_files.txt

* Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132)

* Very initial test passing algorithm

* Working except a minor bug in interval calculations

* After clippy

* Plan

* initial implemantation

* Before prune check ability is added.

Order equivalence implementations will vanish after we send a seperate PR

* minor changes

* Fix bug, ordering equivalence random head

* minor changes

* Add ordering equivalence for sort merge join

* Improvement on tests

* Upstream changes

* Add ordering equivalence for sort merge join

* Fmt issues

* Update comment

* Add ordering equivalence support for hash join

* Make 1 file

* Code enhancements/comment improvements

* Add projection cast handling

* Fix output ordering for sort merge join

* projection bug fix

* Minor changes

* minor changes

* simplify sort_merge_join

* Update equivalence implementation

* Update test_utils.rs

* Update cast implementation

* More idiomatic code

* After merge

* Comments visisted

* Add key swap according to the children orders

* Refactoring

* After merge refactor

* Update sort_enforcement.rs

* Update datafusion/core/src/physical_optimizer/join_selection.rs

Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* Comments are applied

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* Comment improvements and minor code improvements

* Splitting the order based join selection

* Update rat_exclude_files.txt

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Commented

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>

* Bug fix: Fix lexicographical column search among provided ordering (#156)

* License update (#157)

This extension adds Synnada license information to the existing one.

* Sliding Nested Join Algorithm (#142)

* Sliding Hash Join Algorithm (SWHJ) (#147)

* Fix errors introduced during rebase

* Keep Track of Global Ordering Requirement (#165)

* Prunability of Join Filter Physical Expressions (#161)

* BinaryExpr Equivalence (#116)

* Fix errors introduced during rebase

* Support multiple ordered columns on joins and expression graph (#163)

* SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171)

* Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file

* Change in test folders

* Update join_pipeline_selection.rs

* Update utils.rs

* Before clippy

* Before SLT

* Tests are passing and clippy OK.

* [GITHUB ACTION] Refactor for license and actions (#148)

* Delete datafusion main publication

* Adding licence information, refactoring prunibility issues

* Update SYNNADA-CONTRIBUTIONS.txt

* Update rat_exclude_files.txt

* Enhanced Pipeline Execution: Now Supporting Complex Query Plans for Improved Performance (#132)

* Very initial test passing algorithm

* Working except a minor bug in interval calculations

* After clippy

* Plan

* initial implemantation

* Before prune check ability is added.

Order equivalence implementations will vanish after we send a seperate PR

* minor changes

* Fix bug, ordering equivalence random head

* minor changes

* Add ordering equivalence for sort merge join

* Improvement on tests

* Upstream changes

* Add ordering equivalence for sort merge join

* Fmt issues

* Update comment

* Add ordering equivalence support for hash join

* Make 1 file

* Code enhancements/comment improvements

* Add projection cast handling

* Fix output ordering for sort merge join

* projection bug fix

* Minor changes

* minor changes

* simplify sort_merge_join

* Update equivalence implementation

* Update test_utils.rs

* Update cast implementation

* More idiomatic code

* After merge

* Comments visisted

* Add key swap according to the children orders

* Refactoring

* After merge refactor

* Update sort_enforcement.rs

* Update datafusion/core/src/physical_optimizer/join_selection.rs

Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* Comments are applied

* Feature/determine prunability (#139)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

* Determine prunability of tables for join operations (#90)

* ready to review

* license added

* simplifications

* simplifications

* sort expr's are taken separately for each table

* we can return the sort info of the expression now

* check filter conditions

* simplifications

* simplifications

* functions are implemented for SortInfo calculations

* node specialized tableSide functions

* NotImplemented errors are added, test comments are added

* Comment change

* Simplify comparison node calculations

* Simplfications and better commenting

* is_prunable function is updated with new Prunability function

* Indices of sort expressions are updated with intermediate schema columns of the filter

* Unused function is removed

* Future-proof index updating

* An if let check is removed

* simplifications

* Simplifications

* simplifications

* Change if condition

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* fix the tables' unboundedness

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

* Comment improvements and minor code improvements

* Splitting the order based join selection

* Update rat_exclude_files.txt

* Revert "Feature/determine prunability (#139)"

This reverts commit cf56105.

* Commented

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>

* Bug fix: Fix lexicographical column search among provided ordering (#156)

* License update (#157)

This extension adds Synnada license information to the existing one.

* Sliding Nested Join Algorithm (#142)

* Sliding Hash Join Algorithm (SWHJ) (#147)

* Fix errors introduced during rebase

* Keep Track of Global Ordering Requirement (#165)

* Prunability of Join Filter Physical Expressions (#161)

* BinaryExpr Equivalence (#116)

* Fix errors introduced during rebase

* Support multiple ordered columns on joins and expression graph (#163)

* SlidingHashJoin and SlidingNestedLoopJoin planner integration (#171)

* Add license, add contribution hash commits, minor changes, add scripts to automate hash generation,Delete docs yaml file

* Resolve errors introduced during rebase

* After merge

* Update rat_exclude_files.txt

* Comments visited

* Synnada Streaming SQL Tests (#190)

* Adds a new method to construct window function for the given input

* For mustafa

* Final

* Update rat_exclude_files.txt

* More commenting

* Fix linter errors, compile errors after rebase, Update commit hashes

* After merge refactors

* Dir

* Additional test for coverage

* Update join_disable_repartition_joins.slt

* Review changes, remove code duplicates

* Update subdirectory hashes

---------

Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>
Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>

* work in progress

* fix after merge

* joins are ok

* code cleaning

* CaseExpr handling

* tests are updated

* Handling sequential projections

* Simplifications

* partition expr update

* ProjectionPushdown becomes a rule.bash de

* fix after merge

* remove the subrule

* tpch update

* Minor comment changes

* remove unnecessary state struct

* coalesce batches does not let to push projection down

* tpch changes removed

* minor changes

* addresses reviews

* Update projection_pushdown.rs

* minor

* Review Part 1

* Simplify the API of plan handlers

* Review Part 2

* Review Part 3

* Review Part 4

* Review Part 5

* Review Part 6

* Review Part 7

* Remove duplication of physical_expr matching

* fix documentation

* Minor changes

* Take upstream changes

---------

Co-authored-by: Metehan Yıldırım <100111937+metesynnada@users.noreply.github.com>
Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Co-authored-by: Mustafa Akur <106137913+mustafasrepo@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working datafusion Changes in the datafusion crate
Projects
None yet
Development

No branches or pull requests

4 participants