Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(sql): fix order by with index scans #3074

Merged
merged 11 commits into from
Mar 23, 2023
Merged

fix(sql): fix order by with index scans #3074

merged 11 commits into from
Mar 23, 2023

Conversation

bziobrowski
Copy link
Contributor

@bziobrowski bziobrowski commented Mar 16, 2023 •
edited

Fixes ORDER BY clause processing with index scanning factories .
Index factories may return data in neither ascending nor descending but in 'random' order and require additional sorting.

Example :

CREATE TABLE binance_dc (
  timestamp TIMESTAMP,
  symbol SYMBOL index 
) timestamp (timestamp);

SELECT * 
FROM binance_dc 
WHERE symbol= 'ETHUSDT' 
AND timestamp IN '2023-02-10' 
ORDER BY timestamp DESC;

produces incorrect plan :
plan

With this PR plan becomes :

fixedPlan

Additional changes :

  • add sort skipping for index scans with ORDER BY designated timestamp asc - by choosing table-order/heap cursor
  • add explicit symbolOrder meta information to FilterOnValues & FilterOnExcludedValues execution plans to show that actual runtime symbol order can be different from that shown on plan
  • add symbol sorting to FilterOnExcludedValues factory

@bziobrowski
fix(sql): fix order by with index scans
7c1848f
@bziobrowski
Merge remote-tracking branch 'origin/master' into bz_fix_order_by_wit…
9f67a1b 
…h_index_scan
@bziobrowski
Merge remote-tracking branch 'origin/master' into bz_fix_order_by_wit…
0aa5dd2 
…h_index_scan

# Conflicts:
#	core/src/main/java/io/questdb/griffin/engine/table/FilterOnExcludedValuesRecordCursorFactory.java
#	core/src/test/java/io/questdb/griffin/ExplainPlanTest.java
@bziobrowski
chore(tests): added tests to improve coverage
973ec72
@bziobrowski
chore(tests): added tests to improve coverage
3aabc5d
@ideoma ideoma self-requested a review March 21, 2023 16:40
ideoma
ideoma reviewed Mar 22, 2023
View reviewed changes
core/src/main/java/io/questdb/cairo/sql/RecordCursorFactory.java Show resolved Hide resolved
core/src/main/java/io/questdb/cairo/sql/RecordCursorFactory.java Show resolved Hide resolved
core/src/main/java/io/questdb/griffin/SqlCodeGenerator.java Show resolved Hide resolved
core/src/main/java/io/questdb/griffin/engine/orderby/SortedRecordCursorFactory.java Show resolved Hide resolved
...src/main/java/io/questdb/griffin/engine/table/FilterOnExcludedValuesRecordCursorFactory.java Outdated Show resolved Hide resolved
...src/main/java/io/questdb/griffin/engine/table/FilterOnExcludedValuesRecordCursorFactory.java Show resolved Hide resolved
core/src/main/java/io/questdb/griffin/engine/table/FilterOnValuesRecordCursorFactory.java Show resolved Hide resolved
core/src/main/java/io/questdb/griffin/engine/table/FilterOnValuesRecordCursorFactory.java Show resolved Hide resolved
@bziobrowski bziobrowski requested a review from ideoma March 23, 2023 09:27
ideoma
ideoma previously approved these changes Mar 23, 2023
View reviewed changes
@ideoma
Copy link
Collaborator

ideoma commented Mar 23, 2023

[PR Coverage check]

😍 pass : 139 / 140 (99.29%)

file detail

path covered line new line coverage
🔵 io/questdb/griffin/engine/table/DataFrameRecordCursorFactory.java 3 4 75.00%
🔵 io/questdb/griffin/engine/groupby/DistinctTimeSeriesRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/join/AsOfJoinRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/table/AsyncFilteredNegativeLimitRecordCursor.java 2 2 100.00%
🔵 io/questdb/griffin/engine/join/AsOfJoinNoKeyRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/join/HashOuterJoinFilteredRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/join/NestedLoopLeftJoinRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/union/ExceptRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/table/VirtualRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/table/AsyncFilteredRecordCursor.java 2 2 100.00%
🔵 io/questdb/griffin/engine/orderby/SortedLightRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/join/HashOuterJoinLightRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/table/SymbolIndexRowCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/table/FilterOnValuesRecordCursorFactory.java 9 9 100.00%
🔵 io/questdb/griffin/engine/orderby/LimitedSizeSortedLightRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/join/CrossJoinRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/table/FilteredRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/table/SelectedRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/join/LtJoinNoKeyRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/union/IntersectRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/join/LtJoinRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/table/AsyncJitFilteredRecordCursorFactory.java 5 5 100.00%
🔵 io/questdb/griffin/engine/orderby/SortedRecordCursorFactory.java 6 6 100.00%
🔵 io/questdb/griffin/engine/join/SpliceJoinLightRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/join/HashJoinRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/LimitRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/table/FilterOnExcludedValuesRecordCursorFactory.java 29 29 100.00%
🔵 io/questdb/griffin/SqlCodeGenerator.java 49 49 100.00%
🔵 io/questdb/griffin/engine/table/SymbolIndexFilteredRowCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/SqlOptimiser.java 2 2 100.00%
🔵 io/questdb/griffin/engine/join/HashOuterJoinFilteredLightRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/join/HashOuterJoinRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/join/HashJoinLightRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/table/AsyncFilteredRecordCursorFactory.java 5 5 100.00%
🔵 io/questdb/griffin/engine/join/LtJoinLightRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/cairo/sql/RecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/join/AsOfJoinLightRecordCursorFactory.java 1 1 100.00%

@ideoma ideoma merged commit 5abcfd2 into master Mar 23, 2023
@ideoma ideoma deleted the bz_fix_order_by_with_index_scan branch March 23, 2023 13:03
@puzpuzpuz puzpuzpuz added Bug Incorrect or unexpected behavior SQL Issues or changes relating to SQL execution labels Mar 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ideoma ideoma ideoma approved these changes

Assignees
No one assigned
Labels
Bug Incorrect or unexpected behavior SQL Issues or changes relating to SQL execution
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@bziobrowski @ideoma @puzpuzpuz