[SPARK-14551][SQL] Reduce number of NameNode calls in OrcRelation

[rajeshbalamohan]

What changes were proposed in this pull request?

When FileSourceStrategy is used, record reader is created which incurs a NN call internally. Later in OrcRelation.unwrapOrcStructs, it ends ups reading the file information to get the ObjectInspector. This incurs additional NN call. It would be good to avoid this additional NN call (specifically for partitioned datasets).

Added OrcRecordReader which is very similar to OrcNewInputFormat.OrcRecordReader with an option of exposing the ObjectInspector. This eliminates the need to look up the file later for generating the object inspector. This would be specifically be useful for partitioned tables/datasets.

How was this patch tested?

Ran tpc-ds queries manually and also verified by running org.apache.spark.sql.hive.orc.OrcSuite,org.apache.spark.sql.hive.orc.OrcQuerySuite,org.apache.spark.sql.hive.orc.OrcPartitionDiscoverySuite,OrcPartitionDiscoverySuite.OrcHadoopFsRelationSuite,org.apache.spark.sql.hive.execution.HiveCompatibilitySuite

…SourceStrategy mode

[rxin]

@rajeshbalamohan can you fix your title / description? You are having the title spilling over to the end of the description.

[rxin]

Two other things:

1. can you follow the format used by other prs, i.e. [SPARK-14551][SQL] ...
2. can you spell out name node? Most people don't know that NN = namenode. It is a very Hadoop specific thing.

[rajeshbalamohan]

Sure @rxin. I just updated it. Thanks

[rxin]

Thanks!

[rxin]

if this is based on a file from hive, can you say that in the classdoc and explain what the differences are?

[rajeshbalamohan]

Sure. This is based on OrcNewInputFormat.OrcRecordReader (which is marked private). Only addition is the getObjectInspector targeted to reduce namenode calls later. I will update the doc.

[rxin]

cc @liancheng for review

[liancheng]

Nit: Please split this call into the following two for better readability:

          val orcReader = OrcFile.createReader(
            new Path(new URI(file.filePath)), OrcFile.readerOptions(conf))
          new OrcRecordReader(orcReader, conf, fileSplit.getStart(), fileSplit.getLength())

[liancheng]

@rajeshbalamohan Sorry for the late review and thanks for working on this! My major concern is that the newly added OrcRecordReader should be live in spark-hive rather than spark-sql. Otherwise it looks good except for a few styling issues.

[rajeshbalamohan]

Thanks for the review @liancheng
Latest commit addresses the review comments. Changes are as follows

• Moved OrcRecordReader changes to SparkOrcNewRecordReader in spark-hive
• Fixed comment in OrcRelation stating it is a custom Orc record reader. Also, SparkOrcNewRecordReader would suggest that this is specific to Spark.
• Removed pom.xml related changes
• Fixed styling issues.

[liancheng]

test this please

[liancheng]

add to whitelist

[liancheng]

LGTM pending Jenkins.

[SparkQA]

Test build #2859 has finished for PR 12319 at commit d6bc52d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #56770 has finished for PR 12319 at commit d6bc52d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Thanks - merging in master.

[rajeshbalamohan]

Thanks @liancheng , @rxin