[SPARK-37569][SQL] Don't mark nested view fields as nullable

[shardulm94]

What changes were proposed in this pull request?

When analyzing a view, we should not unnecessarily mark nested fields as nullable. If the columns projected by the view define themselves as non-nullable, their nullability should be preserved.

Why are the changes needed?

Consider a view as follows with all fields non-nullable (required)

spark.sql("""
    CREATE OR REPLACE VIEW v AS 
    SELECT id, named_struct('a', id) AS nested
    FROM RANGE(10)
""")

When trying to read this view, it incorrectly marks nested column a as nullable

scala> spark.table("v2").printSchema
root
 |-- id: long (nullable = false)
 |-- nested: struct (nullable = false)
 |    |-- a: long (nullable = true)

However, we can see that the view schema has been correctly stored as non-nullable

scala> System.out.println(spark.sessionState.catalog.externalCatalog.getTable("default", "v2"))
CatalogTable(
Database: default
Table: v2
.
.
.
Schema: root
 |-- id: long (nullable = false)
 |-- nested: struct (nullable = false)
 |    |-- a: long (nullable = false)
)

This is caused by this line in Analyzer.scala. Going through the history of changes for this block of code, it seems like asNullable is a remnant of a time before we added checks to ensure that the from and to types of the cast were compatible. As nullability is already checked, it should be safe to add a cast without converting the target datatype to nullable.

Does this PR introduce any user-facing change?

Yes. View analysis will preserve nullability of nested fields instead of marking all nested fields as nullable.

How was this patch tested?

Added unit test

[shardulm94]

cc: @cloud-fan @gengliangwang @viirya @xkrogen

[xkrogen]

Great find @shardulm94 ! Nice to see that it's a simple fix. Appreciate the short history lesson as well.

[cloud-fan]

ok to test

[SparkQA]

Test build #146028 has finished for PR 34839 at commit 4b8e2ca.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50504/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50504/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50520/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50520/

[SparkQA]

Test build #146045 has finished for PR 34839 at commit e34dbd6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50527/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50527/

[viirya]

Seems okay. This change doesn't break other tests too.

[SparkQA]

Test build #146052 has finished for PR 34839 at commit be8b9df.

• This patch fails from timeout after a configured wait of 500m.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

We should call the createView method to create view, otherwise we just test the permanent view 3 times...

[shardulm94]

Fixed it now

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50549/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50549/

[SparkQA]

Test build #146075 has finished for PR 34839 at commit 16b44a3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to master!