[Core] Spark assert_true and raise_error function support

[gaoyangxiaozhu]

Description

spark assert_true and raise_error function not compatible with current velox type system and function register interface.
as below query a example,

val df = spark.sql("select assert_true(1 > 0) as v, id from t")
df: org.apache.spark.sql.DataFrame = [v: void, id: bigint]

df.schema
res3: org.apache.spark.sql.types.StructType = StructType(StructField(v,NullType,true),StructField(id,LongType,true))

it would generate void type and NullType in output schema which would convert to unknown type in velox, for raise_error, it expect void return type but velox function always expect a non-void return type.

One way i tried is convert raise_error from void return type to string type but it need we update not only expression itself but also the output type schema also from NullType to StringType, otherwise would encounter mem allocation issue based on my local test.

Update:
RaiseError's data type is NullType. Ditto for AssertTrue as it is replaced by IF + RaiseError.
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala#L82

@zhouyuan / @PHILO-HE / @FelixYBW

[FelixYBW]

Is the two function implemented in Presto list or Spark list?

[gaoyangxiaozhu]

want to use this item to track how do support for those specific spark functions, @zhouyuan / @PHILO-HE / @FelixYBW

could you put your thought here, actully the issue can be summaried as how to support NullType schema and void return type spark function.

[gaoyangxiaozhu]

Is the two function implemented in Presto list or Spark list?

spark list https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.assert_true.html

[gaoyangxiaozhu]

looks if we can make a generic solution to let gluten always call vanilla spark to project values for the function, it would be nice. like facebookincubator/velox#9957 encounter similar issue

[FelixYBW]

I see. just talked with @zhli1142015 on the interval issue.

We need a fundamental way to test all the Spark functions. @rui-mo Can your velox_Spark fuzzer test achieve this?

[FelixYBW]

looks if we can make a generic solution to let gluten always call vanilla spark to project values for the function, it would be nice. like facebookincubator/velox#9957 encounter similar issue

It's different issue from input_file_name(). If we can implement the function in Velox, we should do that firstly.

[gaoyangxiaozhu]

looks if we can make a generic solution to let gluten always call vanilla spark to project values for the function, it would be nice. like facebookincubator/velox#9957 encounter similar issue

It's different issue from input_file_name(). If we can implement the function in Velox, we should do that firstly.

so please check my this try

One way i tried is convert raise_error from void return type to string type but it need we update not only expression itself but also the output type schema also from NullType to StringType, otherwise would encounter mem allocation issue based on my local test.
``

this need one change to apply if we detect there is a `NullType` in output schema, then we need convert it always `StringType`, does that change ok ? 

[FelixYBW]

this need one change to apply if we detect there is a NullType in output schema, then we need convert it always StringType, does that change ok ?

No, we should use the same data type as Spark.

Can assert_true return unknown_type in velox? null_type is implemented as unknown_type in velox.

[PHILO-HE]

@gaoyangxiaozhu, velox doesn't support UnknownType (NullType) for function output? I remember this type is supported for input.

[gaoyangxiaozhu]

@gaoyangxiaozhu, velox doesn't support UnknownType (NullType) for function output? I remember this type is supported for input.

got ! let me check and try! thank you @PHILO-HE / @FelixYBW

[rui-mo]

We need a fundamental way to test all the Spark functions. @rui-mo Can your velox_Spark fuzzer test achieve this?

@FelixYBW With this tool, the result of a Velox Spark function can be compared with the result of vanilla Spark. Currently we are working on enabling it in aggregate fuzzer test.