[SPARK-23456][SPARK-21783] Turn on native ORC impl and PPD by default

[dongjoon-hyun]

What changes were proposed in this pull request?

Apache Spark 2.3 introduced native ORC supports with vectorization and many fixes. However, it's shipped as a not-default option. This PR enables native ORC implementation and predicate-pushdown by default for Apache Spark 2.4. We will improve and stabilize ORC data source before Apache Spark 2.4. And, eventually, Apache Spark will drop old Hive-based ORC code.

How was this patch tested?

Pass the Jenkins with existing tests.

[SparkQA]

Test build #87525 has finished for PR 20634 at commit bde6818.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

LGTM

[gatorsmile]

Thanks! Merged to master.

[dongjoon-hyun]

Thank you, @gatorsmile .

[pan3793]

... eventually, Apache Spark will drop old Hive-based ORC code.

@dongjoon-hyun Do you think it's a good time to drop the implementation spark.sql.orc.impl=hive in Spark 4.0? If you don't object, I will open a JIRA to track it.

[dongjoon-hyun]

Yes, it's a good chance. BTW, IIRC, there was a different at Hive ORC CHAR implementation before. So, we couldn't remove it for backward-compatibility issues. Since Spark implements many CHAR features, we need to re-verify that native implementation has all legacy Hive-based ORC features, @pan3793 .

[pan3793]

@dongjoon-hyun thanks for your response and detailed comment. SPARK-44677 is created to track the dropping work, will start to take a look after 3.5.0 released

[dongjoon-hyun]

Thank you, @pan3793 !