[SPARK-35096][SQL] SchemaPruning should adhere spark.sql.caseSensitive config

[sandeep-katta]

What changes were proposed in this pull request?

As a part of the SPARK-26837 pruning of nested fields from object serializers are supported. But it is missed to handle case insensitivity nature of spark

In this PR I have resolved the column names to be pruned based on spark.sql.caseSensitive config
Exception Before Fix

Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
  at org.apache.spark.sql.types.StructType.apply(StructType.scala:414)
  at org.apache.spark.sql.catalyst.optimizer.ObjectSerializerPruning$$anonfun$apply$4.$anonfun$applyOrElse$3(objects.scala:216)
  at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
  at scala.collection.immutable.List.foreach(List.scala:392)
  at scala.collection.TraversableLike.map(TraversableLike.scala:238)
  at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
  at scala.collection.immutable.List.map(List.scala:298)
  at org.apache.spark.sql.catalyst.optimizer.ObjectSerializerPruning$$anonfun$apply$4.applyOrElse(objects.scala:215)
  at org.apache.spark.sql.catalyst.optimizer.ObjectSerializerPruning$$anonfun$apply$4.applyOrElse(objects.scala:203)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDown$1(TreeNode.scala:309)
  at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:72)
  at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:309)
  at 

Why are the changes needed?

After Upgrade to Spark 3 foreachBatch API throws java.lang.ArrayIndexOutOfBoundsException. This issue will be fixed using this PR

Does this PR introduce any user-facing change?

No, Infact fixes the regression

How was this patch tested?

Added tests and also tested verified manually

[github-actions]

Test build #752951989 for PR 32194 at commit db4a74a.

[github-actions]

Test build #752985634 for PR 32194 at commit f6e4b6b.

[sandeep-katta]

CC @viirya @cloud-fan

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42011/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42011/

[SparkQA]

Test build #137434 has finished for PR 32194 at commit db4a74a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #137436 has finished for PR 32194 at commit f6e4b6b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

Good catch! Thanks for the fix.

[dongjoon-hyun]

Thank you, @sandeep-katta . BTW, this is [SQL] PR instead of [CORE].

[github-actions]

Test build #754432050 for PR 32194 at commit e477e91.

[github-actions]

Test build #754495960 for PR 32194 at commit baf8125.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42039/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42039/

[wangyum]

filter -> find?

[sandeep-katta]

Done updated, just curious is there any performance difference between these two

[github-actions]

Test build #754870665 for PR 32194 at commit 004d56c.

[github-actions]

Test build #754876451 for PR 32194 at commit 04b24c9.

[SparkQA]

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42049/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42051/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42051/

[SparkQA]

Test build #137464 has finished for PR 32194 at commit baf8125.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #137475 has finished for PR 32194 at commit 04b24c9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sandeep-katta]

@dongjoon-hyun @viirya , can this PR be merged ?. If not, I am happy to address review comments if there are any

[cloud-fan]

thanks, merging to master/3.1/3.0!

[dongjoon-hyun]

Hi, All.
This broke branch-3.0 because there is no SQLConfHelper.

Error: ] /home/runner/work/spark/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruning.scala:20: object SQLConfHelper is not a member of package org.apache.spark.sql.catalyst
Error: ] /home/runner/work/spark/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruning.scala:23: not found: type SQLConfHelper
Error: ] /home/runner/work/spark/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruning.scala:32: not found: value conf

spark-3.0:branch-3.0 $ git grep SQLConfHelper
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruning.scala:import org.apache.spark.sql.catalyst.SQLConfHelper
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruning.scala:object SchemaPruning extends SQLConfHelper {

I'll revert this from branch-3.0. Please make a backporting PR to branch-3.0.

[cloud-fan]

@sandeep-katta can you help to resubmit the PR for 3.0? thanks!

[sandeep-katta]

Sure will raise it soon

[dongjoon-hyun]

Thank you!

[sandeep-katta]

backport PR #32284