fix: Add complex types testing for lance

[rahil-c]

Describe the issue this Pull Request addresses

Issue: #17665

Add complex type testing for lance file format integration, currently we have only added basic primitive type tests in TestHoodieSparkLanceWriter, but we will need to test types such as (decimal, timestamp, structs, arrays, maps)

Note currently map type in default version of lance file writer is not supported. When trying to run the test testReadMapType encountered the following:

Caused by: java.lang.RuntimeException: LanceError(Schema): Unsupported data type: Map(Field { name: "entries", data_type: Struct([Field { name: "key", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "value", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), /Users/rahil/workplace/lance/rust/lance-core/src/datatypes.rs:171:31
	at com.lancedb.lance.file.LanceFileWriter.writeNative(Native Method)
	at com.lancedb.lance.file.LanceFileWriter.write(LanceFileWriter.java:119)
	at org.apache.hudi.io.lance.HoodieBaseLanceWriter.flushBatch(HoodieBaseLanceWriter.java:214)
	at org.apache.hudi.io.lance.HoodieBaseLanceWriter.close(HoodieBaseLanceWriter.java:136)
	... 6 more

For now will keep the map test disabled.

Summary and Changelog

Added complex type testing in TestHoodieSparkLanceReader which rounds trips using the HoodieSparkLanceWriter

Impact

none

Risk Level

low

Documentation Update

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[rahil-c]

feedback: move tests to data source level, and consolidate tests since we dont one test per type.

[voonhous]

Let's add an extra test case to check if we can read null data types correctly as Spark's ArrayData supports null elements. It will be good to verify that Lance preservers them.

expectedRows.add(createRow(4, new Object[]{1, null, 3}, new Object[]{"A", null, "C"}));

[the-other-tim-brown]

feedback: move tests to data source level, and consolidate tests since we dont one test per type.

@rahil-c I tried migrating the code to the datasource tests but that suite is all in scala and it was causing a large amount of changes to get the code compatible with the datasource APIs. We'll just leave the tests where they are.

[voonhous]

LGTM

[hudi-bot]

CI report:

• d53259b Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[voonhous]

Azure + GitHub CI is green, merging this in.