[SPARK-32025][SQL] Csv schema inference problems with different types in the same column

[planga82]

What changes were proposed in this pull request?

This pull request fixes a bug present in the csv type inference.
We have problems when we have different types in the same column.

Previously:

$ cat /example/f1.csv
col1
43200000
true

spark.read.csv(path="file:///example/*.csv", header=True, inferSchema=True).show()
+----+
|col1|
+----+
|null|
|true|
+----+

root
 |-- col1: boolean (nullable = true)

Now

spark.read.csv(path="file:///example/*.csv", header=True, inferSchema=True).show()
+-------------+
|col1          |
+-------------+
|43200000 |
|true           |
+-------------+

root                                                                            
 |-- col1: string (nullable = true)

Previously the hierarchy of type inference is the following:

IntegerType

LongType

DecimalType

DoubleType

TimestampType

BooleanType

StringType

So, when, for example, we have integers in one column, and the last element is a boolean, all the column is inferred as a boolean column incorrectly and all the number are shown as null when you see the data

We need the following hierarchy. When we have different numeric types in the column it will be resolved correctly. And when we have other different types it will be resolved as a String type column

IntegerType

LongType

DecimalType

DoubleType

StringType

TimestampType

StringType

BooleanType

StringType

StringType

Why are the changes needed?

Fix the bug explained

Does this PR introduce any user-facing change?

No

How was this patch tested?

Unit test and manual tests

[planga82]

CC @HyukjinKwon @cloud-fan @MaxGekk

[MaxGekk]

If StringType is not the last type in any type inference chains, this can break existing users apps, I guess. Can't it?

[planga82]

@MaxGekk Yes, the last type in inference is StringType in all inference chains. I have changed the description to show it better.
The principal difference is that now we have different chains instead of one

[cloud-fan]

Does JSON source have the same problem?

[planga82]

@HyukjinKwon It's exactly what you say, it only happens when the incompatibility is inside one partition. I will change the PR to use compatibleType, and I will make some performance test. Thanks for your help!

@cloud-fan I tested the same situation with json and it works fine, we don't have problems there

[planga82]

I have done some performance tests in my local machine

• File: 999999999 lines integers / 10 GB
• Resources: 4 Cores

Result without changes:

• 21m10,114s
• 20m18,626s
• 19m46,158s

Results after changes

• 19m46,388s
• 22m10,784s
• 21m15,937s

It seems that we don't have a very significant impact but the tests in local are not the best way to be sure

[HyukjinKwon]

ok to test

[SparkQA]

Test build #124510 has finished for PR 28896 at commit 4629bb5.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[planga82]

retest this please

[HyukjinKwon]

retest this please

[SparkQA]

Test build #124512 has finished for PR 28896 at commit 4629bb5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Merged to master.

[gatorsmile]

Is that possible we can document the type inference rule? something like the traditional database like https://www.ibm.com/support/knowledgecenter/SSEPGG_10.1.0/com.ibm.db2.luw.sql.ref.doc/doc/r0008477.html

cc @huaxingao @planga82

[huaxingao]

@gatorsmile I will take a look.

[planga82]

Thanks @huaxingao If you can't do it for any reason tell me.