[SPARK-43935][SQL][PYTHON][CONNECT] Add xpath_* functions to Scala and Python #41470
Conversation
|
@panbingkun many thanks for working on this. since a XPathXXX expression takes an also cc @HyukjinKwon @beliefer |
Ok, let me fix it. |
| * @group "xml_funcs" | ||
| * @since 3.5.0 | ||
| */ | ||
| def xpath(xml: Column, path: String): Column = |
There was a problem hiding this comment.
According to @HyukjinKwon 's suggestion, please change path as Column.
| @@ -28,6 +29,7 @@ class XPathFunctionsSuite extends QueryTest with SharedSparkSession { | |||
| test("xpath_boolean") { | |||
| val df = Seq("<a><b>b</b></a>").toDF("xml") | |||
| checkAnswer(df.selectExpr("xpath_boolean(xml, 'a/b')"), Row(true)) | |||
| checkAnswer(df.select(xpath_boolean(col("xml"), lit("a/b"))), Row(true)) | |||
There was a problem hiding this comment.
Could you update them with
checkAnswer(df.select(xpath_boolean(col("xml"), lit("a/b"))), df.selectExpr("xpath_boolean(xml, 'a/b')"))
There was a problem hiding this comment.
Is it really necessary to do so in such an easily observable situation?
@zhengruifeng
There was a problem hiding this comment.
I think it is fine in this case, since it implicitly compares with above df.selectExpr("xpath_boolean(xml, 'a/b')")
|
LGTM+1. |
|
merged to master |
…d Python ### What changes were proposed in this pull request? Add following functions: - xpath - xpath_boolean - xpath_double - xpath_float - xpath_int - xpath_long - xpath_number - xpath_short - xpath_string to: - Scala API - Python API - Spark Connect Scala Client - Spark Connect Python Client ### Why are the changes needed? for parity ### Does this PR introduce _any_ user-facing change? Yes, new functions. ### How was this patch tested? - Add New UT. Closes apache#41470 from panbingkun/SPARK-43935. Authored-by: panbingkun <pbk1982@gmail.com> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
What changes were proposed in this pull request?
Add following functions:
to:
Why are the changes needed?
for parity
Does this PR introduce any user-facing change?
Yes, new functions.
How was this patch tested?