Use sparse global ordered reader for unordered queries with no dups. #3207
Conversation
|
This pull request has been linked to Shortcut Story #10997: Support multiple ranges in sparse global order reader so unordered queries can be processed by it.. |
KiterLuc
force-pushed
the
lr/sparse-global-order-reader-unordered-queries/ch10997
branch
from
9252f1f
to
d1abd13
Compare
|
Is it possible to separate the fix from the feature change of using global order reader? In general I think it's best to have a separation of concerns in PRs. |
KiterLuc
force-pushed
the
lr/sparse-global-order-reader-unordered-queries/ch10997
branch
2 times, most recently
from
e331097
to
e082ed2
Compare
@Shelnutt2 Yeah you are right, it would be nice to include this in the current release and possibly back port without the other change. I opened #3208 and converted this to draft. Once it is merged, I'll rebase this. |
KiterLuc
force-pushed
the
lr/sparse-global-order-reader-unordered-queries/ch10997
branch
2 times, most recently
from
dab78c3
to
b630d08
Compare
| num_cells--; | ||
| } else { | ||
| // For overlapping ranges, create as many slabs as there are counts. | ||
| auto num = to_process_dup.tile_->bitmap_[to_process_dup.pos_]; |
There was a problem hiding this comment.
I am trying to understand how this works: the bitmap_ value can be something different than 0 or 1 ?
This change modifies the sparse global order reader to process unordered queries made on arrays with duplicates. As we don't care about the order of the data, we can ignore the constraint that we only support one range for global order. This also fixes an issue in bitmap computation for multiplicities, where the relevant ranges were not computed properly for count bitmaps. --- TYPE: IMPROVEMENT DESC: Use sparse global ordered reader for unordered queries with no dups.
KiterLuc
force-pushed
the
lr/sparse-global-order-reader-unordered-queries/ch10997
branch
from
6eeb8f7
to
6d2c381
Compare
KiterLuc
deleted the
lr/sparse-global-order-reader-unordered-queries/ch10997
branch
This change modifies the sparse global order reader to process unordered
queries made on arrays with duplicates. As we don't care about the
order of the data, we can ignore the constraint that we only support
one range for global order.
This also fixes an issue in bitmap computation for multiplicities,
where the relevant ranges were not computed properly for count bitmaps.
TYPE: IMPROVEMENT
DESC: Use sparse global ordered reader for unordered queries with no dups.