-
Notifications
You must be signed in to change notification settings - Fork 133
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Up until #906, PyScalarUDF has used datafusion_physical_expr::functions::make_scalar_function which has been deprecated since v36 and is being removed from the public API in v43.
The deprecation warning advised:
Implement your function directly in terms of ColumnarValue or use
ScalarUDFinstead
datafusion-python users can not currently implement ScalarUDF in that recommended way.
Describe the solution you'd like
- Change
PyScalarUDFto take functions that operate onColumnarValue, as recommended upstream. - Add
PyarrowScalarUDFthat let's python users continue writing UDF's as they do today.
Describe alternatives you've considered
We could remove the ability to write pyarrow UDF's, forcing users to migrate as directed in the deprecation warning.
Additional context
It may be useful to review how upstream previously converted functions that use ArrayRef to ScalarUDF.