[SPARK-28962][SQL][FOLLOW-UP] Add the parameter description for the Scala function API filter #27336
Conversation
|
Test build #117290 has finished for PR 27336 at commit
|
|
All test passed, the R failure is a known flaky incoming feasibility test. cc @viirya |
|
@dongjoon-hyun Thanks for pinging me. Requested help from CRAN. |
|
Test build #117318 has finished for PR 27336 at commit
|
| * | ||
| * @param column: the input array column | ||
| * @param f: (col, index) => predicate, the boolean predicate to filter the input column | ||
| * given the index. Indices start at 0. |
There was a problem hiding this comment.
is this consistent within Spark that the indices parameter starts with 0 in higher-order functions? @ueshin @HyukjinKwon
There was a problem hiding this comment.
ArrayTransform with index argument starts with 0.
We might need to change it from 1 (with a legacy config and migration guide)?
There was a problem hiding this comment.
what's the behavior of presto?
There was a problem hiding this comment.
Actually presto's transform or filter don't take index argument.
I remember we had a discussion about the index argument in the PR for zip_with_index (#21121 (comment)).
And for filter, it was done later separately, but seems like the similar context https://issues.apache.org/jira/browse/SPARK-28962.
There was a problem hiding this comment.
It's too late to change now, let's keep using 0.
What changes were proposed in this pull request?
This PR is a follow-up PR #25666 for adding the description and example for the Scala function API
filter.Why are the changes needed?
It is hard to tell which parameter is the index column.
Does this PR introduce any user-facing change?
No
How was this patch tested?
N/A