[GLUTEN-8580][CORE][Part-1] Clean up unnecessary code related to input file expression#8584
[GLUTEN-8580][CORE][Part-1] Clean up unnecessary code related to input file expression#8584zml1206 merged 2 commits intoapache:mainfrom
Conversation
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
The change looks good to me. @baibaichen Can you also take a look? |
| // To support input_file_name(). According to semantic we should return | ||
| // the exact file name a row belongs to. However in columnar engine it's | ||
| // not easy to accomplish this. so we return a list of file(part) names | ||
| split match { | ||
| case FirstZippedPartitionsPartition(_, g: GlutenPartition, _) => | ||
| InputFileBlockHolderProxy.set(g.files.mkString(",")) | ||
| case _ => | ||
| InputFileBlockHolderProxy.unset() | ||
| } | ||
|
|
There was a problem hiding this comment.
not sure if we can delete these codes. @gaoyangxiaozhu
There was a problem hiding this comment.
There are problems with the previous input file expression implementation. #7124 optimizes the solution and pushes the input file expression down to scanTransform or the project before scan. The results come from native scan or spark thread local, so there is no need to retain the information in InputFileBlockHolder.
|
Thanks for review, are there any other comments? If not, it will be merged later. @Yohahaha @baibaichen |
I have no more comments, thank you! |
…t file expression (apache#8584)
4 participants
What changes were proposed in this pull request?
(Fixes: #8580)
How was this patch tested?