[SPARK-37171][SQL]Add forany and forall to Datasets/Dataframes #34450
Conversation
Add forany and forall api for Dataframe/Datasets API To provide a higher level of abstraction for Spark customers Yes, forany and forall methods are added to the Dataframe/Dataset API Added new unit tests
|
Can one of the admins verify this patch? |
|
Why add these? |
|
I have mentioned in the ticket the potential for implementing the unoptimized version and provide a little higher abstraction. |
| * }}} | ||
| * | ||
| * @group typedrel | ||
| * @since 3.2.0 |
There was a problem hiding this comment.
This can't be since 3.2.0
This would have to be in R, Scala APIs too
But this doesn't seem worth it - it isn't faster to execute
There was a problem hiding this comment.
That’s true. So do you recommend we drop this as it would be too much to implement in R and Python?
I believe the code is for Scala API.
Just FYI, isEmpty method is absent in Python API although present in Scala API.
Let me know if we still wanna go forward, I can change the version then. If you let me know the right version.
There was a problem hiding this comment.
Sorry yes I meant Python/R. I don't think this is worthwhile if it isn't optimizing or saving the caller much code
There was a problem hiding this comment.
No problems :)
i will close the PR
What changes were proposed in this pull request?
Add forany and forall api for Dataframe/Datasets API
Why are the changes needed?
To provide a higher level of abstraction for Spark customers
Does this PR introduce any user-facing change?
Yes, forany and forall methods are added to the Dataframe/Dataset API
How was this patch tested?
Added new unit tests