[SPARK-43926][CONNECT][PYTHON] Add array_agg, array_size, cardinality, count_min_sketch,mask,named_struct,json_* to Scala and Python #41718

ivoson · 2023-06-25T00:53:24Z

What changes were proposed in this pull request?

Add following functions:

array_agg
array_size
cardinality
count_min_sketch
named_struct
json_array_length
json_object_keys
mask

To:

Scala API
Python API
Spark Connect Scala Client
Spark Connect Python Client

Why are the changes needed?

Add Scala, Python and Connect API for these sql functions: array_agg, array_size, cardinality, count_min_sketch, named_struct, json_array_length, json_object_keys, mask

Does this PR introduce any user-facing change?

Yes, added new functions.

How was this patch tested?

New UT added.

ivoson · 2023-06-25T00:53:45Z

cc @zhengruifeng

LuciferYang · 2023-06-25T02:28:52Z

also cc @HyukjinKwon @panbingkun @beliefer FYI

beliefer · 2023-06-25T03:36:55Z

sql/core/src/main/scala/org/apache/spark/sql/functions.scala

+    upperChar: Column,
+    lowerChar: Column,
+    digitChar: Column,
+    otherChar: Column): Column = {


Please supplement the API with other constructor of Mask.

Sure, will add the other constuctors. Thanks.

panbingkun · 2023-06-26T03:11:19Z

sql/core/src/main/scala/org/apache/spark/sql/functions.scala

@@ -368,6 +368,24 @@ object functions {
   */
  def collect_set(columnName: String): Column = collect_set(Column(columnName))

+  /**


Returns a count-min sketch of a column with the given esp,
confidence and seed. The result is an array of bytes, which can be deserialized to a
CountMinSketch before usage. Count-min sketch is a probabilistic data structure used for
cardinality estimation using sub-linear space.

Is the comment above more aligned?

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/CountMinSketchAgg.scala

Lines 45 to 48 in 00e7c08

_FUNC_(col, eps, confidence, seed) - Returns a count-min sketch of a column with the given esp,

confidence and seed. The result is an array of bytes, which can be deserialized to a

`CountMinSketch` before usage. Count-min sketch is a probabilistic data structure used for

cardinality estimation using sub-linear space.

Thanks, done.

panbingkun · 2023-06-26T03:35:17Z

python/pyspark/sql/functions.py

@@ -14394,6 +14394,260 @@ def nvl2(col1: "ColumnOrName", col2: "ColumnOrName", col3: "ColumnOrName") -> Co
    return _invoke_function_over_columns("nvl2", col1, col2, col3)


+@try_remote_functions


Suggest placing the function in the appropriate location of the corresponding file

relocate the functions by group, thanks.

panbingkun · 2023-06-26T03:36:13Z

python/pyspark/sql/connect/functions.py

@@ -3603,6 +3603,80 @@ def nvl2(col1: "ColumnOrName", col2: "ColumnOrName", col3: "ColumnOrName") -> Co
 nvl2.__doc__ = pysparkfuncs.nvl2.__doc__




thanks, done

panbingkun · 2023-06-26T03:37:56Z

sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala

@@ -5537,6 +5568,61 @@ class DataFrameFunctionsSuite extends QueryTest with SharedSparkSession {
    )
  }

+  test("json_array_length function") {


Can these tests be placed in JsonFunctionsSuite?

thanks, moved to JsonFunctionsSuite

beliefer · 2023-06-26T03:58:42Z

sql/core/src/main/scala/org/apache/spark/sql/functions.scala

+  // scalastyle:off line.size.limit
+  /**
+   * Masks the given string value. The function replaces upper-case, lower-case characters
+   * with specific characters, and numbers with 'n'.


replaces upper-case and lower-case characters with the characters specified respectively.

beliefer · 2023-06-26T03:59:14Z

sql/core/src/main/scala/org/apache/spark/sql/functions.scala

+  // scalastyle:off line.size.limit
+  /**
+   * Masks the given string value. The function replaces upper-case, lower-case characters and
+   * numbers with specific characters.


Thanks, done.

beliefer · 2023-06-26T04:01:56Z

python/pyspark/sql/functions.py

+    _lowerChar = lit("x") if lowerChar is None else lowerChar
+    _digitChar = lit("n") if digitChar is None else digitChar
+    _otherChar = lit(None) if otherChar is None else otherChar
+    return _invoke_function_over_columns(


If user passed lowerChar without upperChar, what will happens?

In this case, will repace lowerChar with specified character, and other cases with default value.

For example,

mask("col", lowerChar = 'y')

will replace:
lowerChar -> 'y'
upperChar -> 'X'
digitChar -> 'n'
otherChar -> retain original

I mean is mask("col", 'y'), then user will be confused.

According to the discussion between @zhengruifeng and me, the analyzer will check it too. You can temporarily ignore the above comment now.

LuciferYang · 2023-06-27T11:08:13Z

connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/functions.scala

+   */
+  def mask(input: Column): Column = Column.fn("mask", input)
+
+  // scalastyle:off line.size.limit


I think we should fix the code style and remove these scalastyle:off line.size.limit

+1, please remove them,

I know there are some usage of scalastyle:off line.size.limit in this file, but they are due to long hyperlink string which can not be broken into multi line.

In this PR, i think it is not needed

Done. thanks @LuciferYang @zhengruifeng

LuciferYang · 2023-06-27T11:33:05Z

sql/core/src/main/scala/org/apache/spark/sql/functions.scala

+    new Mask(input.expr)
+  }
+
+  // scalastyle:off line.size.limit


ditto, we should avoid to use scalastyle:off for the new code

got it. thanks.

zhengruifeng · 2023-06-30T05:36:45Z

thanks, merged to master

ivoson · 2023-06-30T09:16:49Z

thanks a lot for your review. @zhengruifeng @LuciferYang @beliefer @panbingkun

ivoson added 4 commits June 24, 2023 19:43

add scala functions

6a9ec8b

add python apis

99079f5

add connect scala client

84dba93

add connect python apis

2ae4eb5

github-actions bot added CONNECT CORE PYTHON SQL labels Jun 25, 2023

beliefer reviewed Jun 25, 2023

View reviewed changes

ivoson added 2 commits June 25, 2023 22:22

format connect scala functions

7d96928

more mask functions

99ec23a

panbingkun reviewed Jun 26, 2023

View reviewed changes

beliefer reviewed Jun 26, 2023

View reviewed changes

ivoson and others added 3 commits June 27, 2023 00:02

address comments

d52e19d

Merge branch 'master' into SPARK-43926

08130d3

format connect scala functions

e742336

LuciferYang reviewed Jun 27, 2023

View reviewed changes

format comments

f1f5551

zhengruifeng approved these changes Jun 30, 2023

View reviewed changes

zhengruifeng closed this in de8ec74 Jun 30, 2023

ivoson deleted the SPARK-43926 branch June 30, 2023 09:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-43926][CONNECT][PYTHON] Add array_agg, array_size, cardinality, count_min_sketch,mask,named_struct,json_* to Scala and Python #41718

[SPARK-43926][CONNECT][PYTHON] Add array_agg, array_size, cardinality, count_min_sketch,mask,named_struct,json_* to Scala and Python #41718

ivoson commented Jun 25, 2023

ivoson commented Jun 25, 2023

LuciferYang commented Jun 25, 2023

beliefer Jun 25, 2023

ivoson Jun 25, 2023

panbingkun Jun 26, 2023

ivoson Jun 26, 2023

panbingkun Jun 26, 2023

ivoson Jun 26, 2023

panbingkun Jun 26, 2023

ivoson Jun 26, 2023

panbingkun Jun 26, 2023

ivoson Jun 26, 2023

beliefer Jun 26, 2023

ivoson Jun 26, 2023

beliefer Jun 26, 2023

ivoson Jun 26, 2023

beliefer Jun 26, 2023

ivoson Jun 26, 2023

beliefer Jun 27, 2023

beliefer Jun 27, 2023

LuciferYang Jun 27, 2023

zhengruifeng Jun 27, 2023

ivoson Jun 28, 2023

LuciferYang Jun 27, 2023

ivoson Jun 28, 2023

zhengruifeng commented Jun 30, 2023

ivoson commented Jun 30, 2023

	_FUNC_(col, eps, confidence, seed) - Returns a count-min sketch of a column with the given esp,
	confidence and seed. The result is an array of bytes, which can be deserialized to a
	`CountMinSketch` before usage. Count-min sketch is a probabilistic data structure used for
	cardinality estimation using sub-linear space.

		@@ -14394,6 +14394,260 @@ def nvl2(col1: "ColumnOrName", col2: "ColumnOrName", col3: "ColumnOrName") -> Co
		return _invoke_function_over_columns("nvl2", col1, col2, col3)


		@try_remote_functions

		@@ -3603,6 +3603,80 @@ def nvl2(col1: "ColumnOrName", col2: "ColumnOrName", col3: "ColumnOrName") -> Co
		nvl2.__doc__ = pysparkfuncs.nvl2.__doc__

[SPARK-43926][CONNECT][PYTHON] Add array_agg, array_size, cardinality, count_min_sketch,mask,named_struct,json_* to Scala and Python #41718

[SPARK-43926][CONNECT][PYTHON] Add array_agg, array_size, cardinality, count_min_sketch,mask,named_struct,json_* to Scala and Python #41718

Conversation

ivoson commented Jun 25, 2023

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

ivoson commented Jun 25, 2023

LuciferYang commented Jun 25, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhengruifeng commented Jun 30, 2023

ivoson commented Jun 30, 2023