[SPARK-43939][CONNECT][PYTHON] Add try_* functions to Scala and Python #41653
Conversation
| * Returns `dividend``/``divisor`. It always performs floating point division. Its result is | ||
| * always null if `divisor` is 0. | ||
| * | ||
| * @note |
There was a problem hiding this comment.
in this PR, let's use call_udf for better parity
| * Returns the sum of `left` and `right` and the result is null on overflow. The acceptable | ||
| * input types are the same with the `+` operator. | ||
| * | ||
| * @note |
There was a problem hiding this comment.
it should be supported naturally in Connect
| * Returns `left``*``right` and the result is null on overflow. The acceptable input types are the | ||
| * same with the `*` operator. | ||
| * | ||
| * @note |
| * Returns `left`-`right` and the result is null on overflow. The acceptable input types are the | ||
| * same with the `-` operator. | ||
| * | ||
| * @note |
python/pyspark/sql/functions.py
Outdated
|
|
||
| .. versionadded:: 3.5.0 | ||
|
|
||
| Notes |
python/pyspark/sql/functions.py
Outdated
|
|
||
| .. versionadded:: 3.5.0 | ||
|
|
||
| Notes |
python/pyspark/sql/functions.py
Outdated
|
|
||
| .. versionadded:: 3.5.0 | ||
|
|
||
| Notes |
| * Returns the sum of `left` and `right` and the result is null on overflow. The acceptable | ||
| * input types are the same with the `+` operator. | ||
| * | ||
| * @note |
There was a problem hiding this comment.
please check similar function in vanilla scala apis
| * @since 3.5.0 | ||
| */ | ||
| def try_add(left: Column, right: Column): Column = { | ||
| call_udf("try_add", left, right) |
There was a problem hiding this comment.
sorry, let's directly use UnresolvedFunction for this case
since there were already some similar cases in functions, e.g.
spark/sql/core/src/main/scala/org/apache/spark/sql/functions.scala
Lines 3454 to 3463 in 476c58e
There was a problem hiding this comment.
Okay, let me test the issue with scaladoc first.
|
thank you @panbingkun merged to master |
| @try_remote_functions | ||
| def try_element_at(col: "ColumnOrName", extraction: "ColumnOrName") -> Column: | ||
| """ | ||
| (array, index) - Returns element of array at given (1-based) index. If Index is 0, Spark will |
There was a problem hiding this comment.
@panbingkun for try_element_at, maybe we should unify the second parameter's name to index, in both scala and python
There was a problem hiding this comment.
Let me fix it in the following PR.
There was a problem hiding this comment.
@panbingkun I just notice that try_element_at use the same names as element_at, so current commit is fine. we don't need any follow up.
What changes were proposed in this pull request?
Add following functions:
to:
Why are the changes needed?
for parity
Does this PR introduce any user-facing change?
Yes, new functions.
How was this patch tested?