[SPARK-19847][SQL] port hive read to FileFormat API

[cloud-fan]

What changes were proposed in this pull request?

implement the read logic in HiveFileFormat, to unify the table read path between data source and hive serde tables.

The major change is, hive partition may have a different serde, so the planner should put more information in PartitionedFile and send it to executors.

Tow things need to be improved in the future:

1. Due to the way we read hive table files, we do not support reading a partial file yet, which may reduce the parallelism for large files.
2. Hive tables with storage handler(non-file-based) still go to the old code path.

How was this patch tested?

existing tests.

[cloud-fan]

cc @sameeragarwal @rxin @gatorsmile

[SparkQA]

Test build #74075 has started for PR 17187 at commit ad47887.

[gatorsmile]

retest this please

[SparkQA]

Test build #74080 has finished for PR 17187 at commit ad47887.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• case class PartitionDirectory(
• case class PartitionPath(values: InternalRow, path: Path, @Nullable metadata: AnyRef = null)
• class ReadHiveDataSource(session: SparkSession) extends Rule[LogicalPlan]
• class HiveFileIndex(

[windpiger]

with be -> will be