Please sign in to comment.
[SPARK-25674][FOLLOW-UP] Update the stats for each ColumnarBatch
This PR is a follow-up of #22594 . This alternative can avoid the unneeded computation in the hot code path. - For row-based scan, we keep the original way. - For the columnar scan, we just need to update the stats after each batch. N/A Closes #22731 from gatorsmile/udpateStatsFileScanRDD. Authored-by: gatorsmile <firstname.lastname@example.org> Signed-off-by: Wenchen Fan <email@example.com> (cherry picked from commit 4cee191) Signed-off-by: Sean Owen <firstname.lastname@example.org>
- Loading branch information...
Showing with 8 additions and 7 deletions.