feat(functions-nested): add array_filter higher-order function#21895
feat(functions-nested): add array_filter higher-order function#21895ologlogn wants to merge 1 commit intoapache:mainfrom
Conversation
6ff8773 to
07e4548
Compare
|
Hi @gabotechs, could you please trigger CI? Thanks! |
07e4548 to
44715ac
Compare
cbf076a to
36c8f36
Compare
cb94b16 to
ec92925
Compare
36c8f36 to
406f85b
Compare
There was a problem hiding this comment.
Forgive me if I missed something, but how does this function behave if the array argument is null or the lambda itself is null? Could we have some tests for those as well?
There was a problem hiding this comment.
null array -> should return null.
if lambda returns null for some elements -> those will be filtered out. Null is treated as false.
if lambda always returns null -> output will be empty list.
i will try to add sql tests for this
406f85b to
4e0caa6
Compare
| impl ArrayFilter { | ||
| pub fn new() -> Self { | ||
| Self { | ||
| signature: HigherOrderSignature::user_defined(Volatility::Immutable), |
There was a problem hiding this comment.
I plan to open a PR soon so we can be more specific about the Lambda signature we want (e.g. exact types) so all the validation can be hidden into the planner (and potentially be able to remove value_lambda_pair)
Which issue does this PR close?
Partially addresses #14509 — implements
array_filter/list_filter.Rationale for this change
array_transform(#21679) added the firstHigherOrderUDF.array_filteris the natural companion: filter array elements with a boolean lambda, matching Sparkfilter/ DuckDBlist_filtersemantics.What changes are included in this PR?
HigherOrderUDFArrayFilter(array_filter/list_filteralias)truekeeps,false/null drops (matches Spark semantics)List,LargeList, sliced arrays, null sublistsx -> true/x -> false)arrow::compute::filter)lambda_utils.rs: shared HOF helpers extracted fromarray_transform(value_lambda_pair,coerce_single_list_arg,single_list_lambda_parameters,extract_list_values)test_utils.rs: shared unit test helpers (create_i32_list,eval_hof_on_i32_list)Are these changes tested?
higher_order.slt: filter variants,array_filter+array_transformcombinations, error casesAre there any user-facing changes?
Yes —
array_filter(array, lambda)and aliaslist_filter(array, lambda)are now available as SQL functions.