MSQ: Use frames to back broadcast joinables.#16913
MSQ: Use frames to back broadcast joinables.#16913gianm wants to merge 1 commit intoapache:masterfrom
Conversation
In BroadcastJoinSegmentMapFnProcessor, use FrameBasedInlineDataSource and FrameBasedIndexedTable to back broadcast joinables, rather than a regular InlineDataSource (which would use Java object arrays). Reduces memory usage and eliminates a copy while building the joinable.
github-actions
bot
added
Area - Batch Ingestion
Area - Segment Format and Ser/De
Area - MSQ
There was a problem hiding this comment.
I was wondering if there's a better way than converting the incoming frame to columnar frame. #14999 tries to have QueryableIndex for the row-based frames as well that I was using to test the join performance with row based frames. Should we implement QueryableIndex for row based frame so that the conversion isn't required?
|
AFAIR #14999 worked fine with the frame-based broadcast table and didn't have any performance degradation when compared with using columnar frames. |
|
@LakshSingla are you suggesting that in this patch I should use row-based frames rather than columnar? |
|
This pull request has been marked as stale due to 60 days of inactivity. |
|
This pull request/issue has been closed due to lack of activity. If you think that |
2 participants
In BroadcastJoinSegmentMapFnProcessor, use FrameBasedInlineDataSource and FrameBasedIndexedTable to back broadcast joinables, rather than a regular InlineDataSource (which would use Java object arrays).
Reduces memory usage and eliminates a copy while building the joinable.