Enable more aggregation pushdown into scan #1188
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
mbasmanova
force-pushed
the
agg-pushdown-stats
branch
from
b233332
to
9de3f1d
Compare
mbasmanova
changed the title
mbasmanova
requested review from
aditi-pandit,
laithsakka,
majetideepak,
oerling and
pedroerp
|
@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Summary: Add runtime statistic "loadedToValueHook" to track number of values processed via aggregation pushdown into scan. This value contains total number of values, not rows, e.g. if we have 2 aggregates using pushdown for 10 rows each, loadedToValueHook will be 10 + 10 = 20. This statistics is reported by the LazyVector::load() method when it is called with non-null ValueHook. Use the new statistic to verify pushdown in TableScanTest.aggregationPushdown. These checks showed that aggregation pushdown is not enabled when it should be. Hence, a couple more fixes. Fix check for identity projections in FilterProject operator to identify these more reliably. This allows pushdown for queries like SELECT f(a), sum(b) FROM t GROUP BY 1. Fix can-pushdown check in GroupingSet to use the correct set of aggregation inputs. This check was consistently producing 'false' for non-first aggregates. This allows pushdown for all aggregates in queries like SELECT a, sum(b), sum (c), min(d) FROM t GROUP BY 1. Remaining TODOs for follow-up PRs: - Enable pushdown for global aggregations. - Fix runtime statistics reporting in LazyVector. As of now, aggregation pushdown statistics reported by LazyVector are added to the first operator upstream of aggregation. In some queries it is TableScan, while in others it is FilterProject. Pull Request resolved: facebookincubator#1188 Reviewed By: pedroerp Differential Revision: D34818362 Pulled By: mbasmanova fbshipit-source-id: bc200f1985f59eecd6119c69d8a3d4ac7bbbc1b9
|
This pull request was exported from Phabricator. Differential Revision: D34818362 |
mbasmanova
force-pushed
the
agg-pushdown-stats
branch
from
9de3f1d
to
3dd54f1
Compare
shiyu-bytedance
pushed a commit
to shiyu-bytedance/velox-1
that referenced
this pull request
Aug 18, 2022
Summary: Add runtime statistic "loadedToValueHook" to track number of values processed via aggregation pushdown into scan. This value contains total number of values, not rows, e.g. if we have 2 aggregates using pushdown for 10 rows each, loadedToValueHook will be 10 + 10 = 20. This statistics is reported by the LazyVector::load() method when it is called with non-null ValueHook. Use the new statistic to verify pushdown in TableScanTest.aggregationPushdown. These checks showed that aggregation pushdown is not enabled when it should be. Hence, a couple more fixes. Fix check for identity projections in FilterProject operator to identify these more reliably. This allows pushdown for queries like SELECT f(a), sum(b) FROM t GROUP BY 1. Fix can-pushdown check in GroupingSet to use the correct set of aggregation inputs. This check was consistently producing 'false' for non-first aggregates. This allows pushdown for all aggregates in queries like SELECT a, sum(b), sum (c), min(d) FROM t GROUP BY 1. Remaining TODOs for follow-up PRs: - Enable pushdown for global aggregations. - Fix runtime statistics reporting in LazyVector. As of now, aggregation pushdown statistics reported by LazyVector are added to the first operator upstream of aggregation. In some queries it is TableScan, while in others it is FilterProject. Pull Request resolved: facebookincubator#1188 Reviewed By: pedroerp Differential Revision: D34818362 Pulled By: mbasmanova fbshipit-source-id: 0fd2090e2ff48466921a09a4105fad2277ef5270
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add runtime statistic "loadedToValueHook" to track number of values processed
via aggregation pushdown into scan. This value contains total number of values,
not rows, e.g. if we have 2 aggregates using pushdown for 10 rows each,
loadedToValueHook will be 10 + 10 = 29. This statistics is reported by the
LazyVector::load() method when it is called with non-null ValueHook.
Use the new statistic to verify pushdown in TableScanTest.aggregationPushdown.
These checks showed that aggregation pushdown is not enabled when it should be.
Hence, a couple more fixes.
Fix check for identity projections in FilterProject operator to identify these
more reliably. This allows pushdown for queries like SELECT f(a), sum(b) FROM t
GROUP BY 1.
Fix can-pushdown check in GroupingSet to use the correct set of aggregation
inputs. This check was consistently producing 'false' for non-first aggregates.
This allows pushdown for all aggregates in queries like SELECT a, sum(b), sum
(c), min(d) FROM t GROUP BY 1.
Remaining TODOs for follow-up PRs:
pushdown statistics reported by LazyVector are added to the first operator
upstream of aggregation. In some queries it is TableScan, while in others it
is FilterProject.