[SPARK-15143][SPARK-15144][SQL] Add CSV tests with HadoopFsRelationTest and support for nullValue for other types

[HyukjinKwon]

What changes were proposed in this pull request?

Currently, nullValue option does not work for some types, BooleanType, TimestampType, DateType and StringType. So, currently there is no way to read null for those types. This PR adds the support just like the other types.

Also, CSV data source is not being tested with HadoopFsRelationTest as a HadoopFsRelation. HadoopFsRelationTest includes 50 more tests (eg. partitioned table tests).

This PR adds two variables, extraReadOptions and extraWriteOptions in HadoopFsRelationTest so that the child class gives some options for reading and writing. In order to get the tests in HadoopFsRelationTest passed, CSV data source needs to give options header and inferSchema as true for reading and header as true for writing.

How was this patch tested?

Unittests in CSVHadoopFsRelationTest, CSVTypeCastSuite and edited tests in CSVSuite

[HyukjinKwon]

This is being tested against the data as below:

year,make,model,comment,blank
"2012","Tesla","S",null,

1997,Ford,E350,"Go get one now they are going fast",
null,Chevy,Volt

Since the header is year,make,model,comment,blank, this should produce the values 2012,Tesla,S,null,null because nullValue is set to "null".

[HyukjinKwon]

cc @rxin @falaki

[SparkQA]

Test build #57845 has finished for PR 12921 at commit ef71599.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #57846 has finished for PR 12921 at commit f80df8c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #57847 has finished for PR 12921 at commit 2ea702c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Simply the logic below was added just like inferField():

if (datum == null || (datum == options.nullValue && nullable)) {
  null
} else {
 ... 

[SparkQA]

Test build #57856 has finished for PR 12921 at commit 33288c8.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

@falaki I just noticed and thought this test implies nullValue does not apply for StringType. Is this intendedly being exclusive? I thought nullValue should be applied for all the types equivalently.

[HyukjinKwon]

Otherwise, nulls for StringType will be lost in the roundtrip of reading and writing.

[SparkQA]

Test build #57871 has finished for PR 12921 at commit f31f69e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #58045 has finished for PR 12921 at commit 1233bd7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class NoSuchPermanentFunctionException(db: String, func: String)
  • class NoSuchFunctionException(db: String, func: String)
  • case class GetExternalRowField(

[SparkQA]

Test build #58319 has finished for PR 12921 at commit 75f1cb8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Please excuse my ping @rxin @falaki

[HyukjinKwon]

Hi @cloud-fan, Could you please take a look?

[HyukjinKwon]

Closing this since another PR has (I think) a better change. I will maybe submit another PR for adding some tests in the future.