Spark: Parse sort orders with zorder expressions#5185
Spark: Parse sort orders with zorder expressions#5185rdblue wants to merge 2 commits intoapache:masterfrom
Conversation
|
@ajantha-bhat, this demonstrates how to use the Spark extensions parser to parse a Zorder expression. This also updates the current procedure to use the parser directly. You can use this for your update to |
|
@rdblue : Thank you so much. I will update my PR with this change and add more test cases and a co-authored by message. |
| import java.util.List; | ||
|
|
||
| class SortOrderParserUtil { | ||
| public class Zorder implements Term { |
There was a problem hiding this comment.
should this implement UnboundTerm?
Because SortOrder only accepts UnboundTerm. https://github.com/apache/iceberg/blob/master/api/src/main/java/org/apache/iceberg/SortOrder.java#L258
Also, Now sortOrder needs some extra array or list that says which of the fields are Zordered in the sortOrder?
There was a problem hiding this comment.
No. UnboundTerm expects to have a single ref.
|
I have updated my PR as per this. |
|
Closing this in favor of #4902. |
2 participants
This updates the
RewriteDataFilesProcedureto directly call the parser to parse sort orders, instead of creating an ALTER TABLE statement and extracting the sort order.This also makes a few updates to allow creating a
Termfrom a zorder expression when parsed by Spark, so that the parser can pass back a zorder sort order representation (usingRawSortFieldbecauseSortOrdercannot contain a Zorder expression).