[VL] spark.read.csv("/tmp/test.csv") throws Exception

[xumingming]

Backend

VL (Velox)

Bug description

scala> var df = spark.read.csv("/tmp/test.csv");
E20240320 13:21:15.767925 16464018 Exceptions.h:69] Line: /path-to-gluten/ep/build-velox/build/velox_ep/velox/exec/Task.cpp:1811, Function:terminate, Expression:  Cancelled, Source: RUNTIME, ErrorCode: INVALID_STATE
df: org.apache.spark.sql.DataFrame = [_c0: string, _c1: string ... 1 more field]

Spark version

Spark-3.2.x

Spark configurations

No response

System information

Velox System Info v0.0.2
Commit: 57d36593a8f936814f0a8e2f75488c97225d8bdb
CMake Version: 3.24.4
System: Darwin-21.6.0
Arch: arm64
C++ Compiler: /Library/Developer/CommandLineTools/usr/bin/c++
C++ Compiler Version: 14.0.0.14000029
C Compiler: /Library/Developer/CommandLineTools/usr/bin/cc
C Compiler Version: 14.0.0.14000029
CMake Prefix Path: /Library/Developer/CommandLineTools/SDKs/MacOSX13.1.sdk/usr;/opt/homebrew;/usr/local;/usr;/;/usr/local/cmake-3.24.4/CMake.app/Contents;/usr/local;/usr/X11R6;/usr/pkg;/opt;/sw;/opt/local

Relevant logs

No response

[zhztheplayer]

Was the query result correct?

Sometimes this log doesn't mean there is anything wrong. If it's the case we can find a way to suppress that log from Velox.

[xumingming]

@zhztheplayer The result is correct, just a simple select.

[xumingming]

Full log:

scala> val path = "/tmp/person.csv"
path: String = /tmp/person.csv

scala> val person = spark.read.csv(path);
E20240322 16:52:40.086603 25592971 Exceptions.h:69] Line: /Users/abei/Code/gluten/ep/build-velox/build/velox_ep/velox/exec/Task.cpp:1811, Function:terminate, Expression:  Cancelled, Source: RUNTIME, ErrorCode: INVALID_STATE
person: org.apache.spark.sql.DataFrame = [_c0: string, _c1: string ... 1 more field]

scala> person.registerTempTable("person");
warning: one deprecation (since 2.0.0); for details, enable `:setting -deprecation' or `:replay -deprecation'

scala> spark.sql("select count(*) from person limit 10").show();
+--------+
|count(1)|
+--------+
|       2|
+--------+

If the log is expected, I will close this issue.

[PHILO-HE]

Hi @xumingming, the exception looks irrelevant. Currently, Gluten doesn't support CSV format. So reading CSV will fall back to vanilla Spark. You can check the fallback summary in Gluten SQL / DataFrame tab of history server UI.

[xumingming]

@PHILO-HE Thanks for the information! I tried with parquet data(nation table in TPCH), the details are the following:

== Fallback Summary ==
(4) Project: Not supported to map spark function name to substrait function name: toprettystring(n_nationkey#23, Some(Asia/Shanghai)), class name: ToPrettyString.
(5) CollectLimit: Gluten does not touch it or does not support it

== Physical Plan ==
CollectLimit (5)
+- Project (4)
   +- VeloxColumnarToRowExec (3)
      +- ^ Scan parquet  (1)

Is the fallback for Project expected?

[PHILO-HE]

@PHILO-HE Thanks for the information! I tried with parquet data(nation table in TPCH), the details are the following:

== Fallback Summary ==
(4) Project: Not supported to map spark function name to substrait function name: toprettystring(n_nationkey#23, Some(Asia/Shanghai)), class name: ToPrettyString.
(5) CollectLimit: Gluten does not touch it or does not support it

== Physical Plan ==
CollectLimit (5)
+- Project (4)
   +- VeloxColumnarToRowExec (3)
      +- ^ Scan parquet  (1)

Is the fallback for Project expected?

@xumingming, yes, because toprettystring is not supported in Gluten.

[xumingming]

@PHILO-HE Thanks for the information, closing it.