[SPARK-41264][CONNECT][PYTHON] Make Literal support more datatypes #38800

zhengruifeng · 2022-11-25T08:28:36Z

What changes were proposed in this pull request?

1, in the sever side, try to match https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala#L63-L101, and use CreateArray, CreateStruct, CreateMap for complex inputs;

2, in the client side, try to match https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L1335-L1349 ,
but do not support datetime.time since I don't find a corrsponding sql type for it.

Why are the changes needed?

try to support all datatype

Does this PR introduce any user-facing change?

No

How was this patch tested?

updated tests

grundprinzip

Thanks for doing the tedious work!

grundprinzip · 2022-11-25T09:11:35Z

connector/connect/src/main/protobuf/spark/connect/expressions.proto

+    message CalendarInterval {
+      int32 months = 1;
+      int32 days = 2;
+      int64 microseconds = 3;
    }

    message IntervalYearToMonth {


if you're removing the reference above, please remove it here as well.

will update

grundprinzip · 2022-11-25T09:13:33Z

connector/connect/src/main/protobuf/spark/connect/expressions.proto

      // Date in units of days since the UNIX epoch.
      int32 date = 16;
-      // Time in units of microseconds past midnight
-      int64 time = 17;


this is the ``datetime.time`?

yes, I don't find a corresponding sql type for datetime.time, so remove it.

grundprinzip · 2022-11-25T09:14:04Z

connector/connect/src/main/protobuf/spark/connect/expressions.proto

-      DataType null = 29; // a typed null literal
-      List list = 30;
-      DataType.Array empty_array = 31;
-      DataType.Map empty_map = 32;


Please clean up the below elements as well, thank you!

zhengruifeng · 2022-11-28T03:30:58Z

also cc @HyukjinKwon @amaliujia @cloud-fan

cloud-fan · 2022-11-28T09:06:34Z

connector/connect/src/main/protobuf/spark/connect/expressions.proto

@@ -40,37 +40,34 @@ message Expression {

  message Literal {
    oneof literal_type {


In catalyst, Literal is consist of value and type. The benefit is, there aren't many physical values (timestamp and timestamp ntz are both int64). When we need to add a new data type with existing physical values (e.g. char are varchar), we don't need to update Literal.

This is not a design decision made by this PR, so I won't block this PR for this reason.

This only works in Scala because of the inherited type hierarchy, the ability of an Any type and deferring the necessary apply / unaply methods.

Since proto does not have this imperative concept of inheritance, we need to embed the physical type with the logical one.

amaliujia · 2022-11-28T18:42:10Z

python/pyspark/sql/tests/connect/test_connect_column_expressions.py

@@ -134,6 +138,37 @@ def test_list_to_literal(self):
        lit_list_plan = fun.lit([fun.lit(10), fun.lit("str")]).to_plan(None)
        self.assertIsNotNone(lit_list_plan)

+    def test_tuple_to_literal(self):


QQ is this Python specific?

yes, scala lit do not support tuple and map.

amaliujia · 2022-11-28T18:45:27Z

LGTM

A big left work is testing coverage over literals. For example does NaN, +inf, -inf can pass through Connect proto and any test case? I also have tons of questions about value range for timestamp and timezones.

This is not because of current PR so that is not a blocker.

hvanhovell · 2022-11-28T18:50:17Z

connector/connect/src/main/protobuf/spark/connect/expressions.proto

-      // (ignoring precision) Always 16 bytes in length
-      bytes value = 1;
+      // the string representation.
+      string value = 1;


Why not bytes?

refer to the constructors of Decimal https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala#L541-L564

I think we can use :

def apply(unscaled: Long, precision: Int, scale: Int): Decimal = new Decimal().set(unscaled, precision, scale) def apply(value: String): Decimal = new Decimal().set(BigDecimal(value))

I just choose string for two reasons:
1, it is convenient;
2, not sure about how to get the unscaled and scale from python decimal

support tuple 2 struct fix test address comments init

zhengruifeng · 2022-11-29T04:31:59Z

A big left work is testing coverage over literals. For example does NaN, +inf, -inf can pass through Connect proto and any test case? I also have tons of questions about value range for timestamp and timezones.

@amaliujia add a simple test for nan and inf. Will add more tests as follow ups.

zhengruifeng · 2022-11-29T06:38:07Z

since I have other PRs depending on this work, let me merge it into master now, thanks for reviews

HyukjinKwon

LGTM thanks!

### What changes were proposed in this pull request? 1, in the sever side, try to match https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala#L63-L101, and use `CreateArray`, `CreateStruct`, `CreateMap` for complex inputs; 2, in the client side, try to match https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L1335-L1349 , but do not support `datetime.time` since I don't find a corrsponding sql type for it. ### Why are the changes needed? try to support all datatype ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? updated tests Closes apache#38800 from zhengruifeng/connect_update_literal. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>

zhengruifeng changed the title ~~[SPARK-41264][CONNECT][PYTHON] Make Literal support more datatypes~~ [SPARK-41264][CONNECT][PYTHON][WIP] Make Literal support more datatypes Nov 25, 2022

github-actions bot added CONNECT CORE PYTHON SQL labels Nov 25, 2022

grundprinzip reviewed Nov 25, 2022

View reviewed changes

zhengruifeng force-pushed the connect_update_literal branch 2 times, most recently from e3e5ccb to e271a41 Compare November 28, 2022 01:41

zhengruifeng marked this pull request as ready for review November 28, 2022 01:42

zhengruifeng changed the title ~~[SPARK-41264][CONNECT][PYTHON][WIP] Make Literal support more datatypes~~ [SPARK-41264][CONNECT][PYTHON] Make Literal support more datatypes Nov 28, 2022

zhengruifeng force-pushed the connect_update_literal branch from e271a41 to d1fc8e8 Compare November 28, 2022 05:51

cloud-fan reviewed Nov 28, 2022

View reviewed changes

cloud-fan approved these changes Nov 28, 2022

View reviewed changes

amaliujia reviewed Nov 28, 2022

View reviewed changes

hvanhovell reviewed Nov 28, 2022

View reviewed changes

grundprinzip approved these changes Nov 28, 2022

View reviewed changes

init

1d2e1ce

support tuple 2 struct fix test address comments init

amaliujia approved these changes Nov 29, 2022

View reviewed changes

add test for nan inf

85b6c9e

zhengruifeng force-pushed the connect_update_literal branch from d1fc8e8 to 85b6c9e Compare November 29, 2022 04:30

zhengruifeng closed this in 4c35c5b Nov 29, 2022

zhengruifeng deleted the connect_update_literal branch November 29, 2022 06:38

HyukjinKwon reviewed Nov 29, 2022

View reviewed changes

zhengruifeng mentioned this pull request Dec 7, 2022

[SPARK-41436][CONNECT][PYTHON] Implement collection functions: A~C #38961

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-41264][CONNECT][PYTHON] Make Literal support more datatypes #38800

[SPARK-41264][CONNECT][PYTHON] Make Literal support more datatypes #38800

zhengruifeng commented Nov 25, 2022

grundprinzip left a comment

grundprinzip Nov 25, 2022

zhengruifeng Nov 25, 2022

grundprinzip Nov 25, 2022

zhengruifeng Nov 25, 2022

grundprinzip Nov 25, 2022

zhengruifeng Nov 25, 2022

zhengruifeng commented Nov 28, 2022

cloud-fan Nov 28, 2022

grundprinzip Nov 28, 2022

amaliujia Nov 28, 2022

zhengruifeng Nov 29, 2022

amaliujia commented Nov 28, 2022 •

edited

hvanhovell Nov 28, 2022

zhengruifeng Nov 29, 2022

zhengruifeng commented Nov 29, 2022

zhengruifeng commented Nov 29, 2022

HyukjinKwon left a comment

		@@ -40,37 +40,34 @@ message Expression {

		message Literal {
		oneof literal_type {

[SPARK-41264][CONNECT][PYTHON] Make Literal support more datatypes #38800

[SPARK-41264][CONNECT][PYTHON] Make Literal support more datatypes #38800

Conversation

zhengruifeng commented Nov 25, 2022

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

grundprinzip left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhengruifeng commented Nov 28, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amaliujia commented Nov 28, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhengruifeng commented Nov 29, 2022

zhengruifeng commented Nov 29, 2022

HyukjinKwon left a comment

Choose a reason for hiding this comment

amaliujia commented Nov 28, 2022 •

edited