[SPARK-51496][SQL] CaseInsensitiveStringMap comparison should ignore case

[huaxingao]

What changes were proposed in this pull request?

Since both commandOptions and dsOptions are CaseInsensitiveStringMap objects, I think we should convert the keys and values to lowercase before comparing them

Why are the changes needed?

In iceberg/spark4.0 integration, I got a few assertion errors:

assertion failed
java.lang.AssertionError: assertion failed
	at scala.Predef$.assert(Predef.scala:264)
	at org.apache.spark.sql.execution.datasources.v2.V2Writes$.org$apache$spark$sql$execution$datasources$v2$V2Writes$$mergeOptions(V2Writes.scala:128)

The assertion error occurs when comparing commandOptions and dsOptions; the cases of the keys don't match.

assert(commandOptions == dsOptions)

Does this PR introduce any user-facing change?

No

How was this patch tested?

I have verified that the iceberg tests can pass with this fix.

Was this patch authored or co-authored using generative AI tooling?

no

[pan3793]

because one of commandOptions and dsOptions was canonicalized but another wasn't? if so, should it be better to do canonicalization on the caller side too?

[huaxingao]

@pan3793
When setting the option, Spark doesn't change the key to lower case. When creating a DataSourceV2Relation, the keys in dsOption are converted to lower case.
I think there are two ways to fix the problem, the first way is as what i did in the PR, the second way is to convert key to lower case in DataFrameWriter.option. The second way doesn't work, because in the iceberg test

    df2.select("id", "data")
            .sort("data")
            .write()
            .format("org.apache.iceberg.spark.source.ManualSource")
            .option(ManualSource.TABLE_NAME, manualTableName)

ManualSource.TABLE_NAME is upper case. If calling .option(ManualSource.TABLE_NAME, manualTableName) changing the key to lower case, I will get

Missing property TABLE_NAME
java.lang.IllegalArgumentException: Missing property TABLE_NAME
	at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:143)
	at org.apache.iceberg.spark.source.ManualSource.getTable(ManualSource.java:64)
	at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.getTableFromProvider(DataSourceV2Utils.scala:98)

[huaxingao]

cc @cloud-fan

[cloud-fan]

shall we be case-preserving and do not lower case the keys when creating DataSourceV2Relation?

[huaxingao]

shall we be case-preserving and do not lower case the keys when creating DataSourceV2Relation?

When creating a CaseInsensitiveStringMap, the keys are changed to lower case in the constructor. This dsOptions is used to create DataSourceV2Relation.
Since DataSourceV2Relation.option is a CaseInsensitiveStringMap, I feel it's the correct behavior to change the key to lower case.

[cloud-fan]

The way we turn CaseInsensitiveStringMap to a scala Map is r.options.asScala.toMap, I think we should change it to r.options.asCaseSensitiveMap.asScala.toMap to be case preserving

[cloud-fan]

thanks, merging to master/4.0!

[huaxingao]

Thanks @cloud-fan