[AURON #2140] Implement native function of map_concat

[weimingdiit]

Which issue does this PR close?

Closes ##2140

Rationale for this change

map_concat(...) was not supported in Auron’s native execution path, so queries using it fell back instead of being executed natively.

This change adds support through the extension-function path, which fits the existing pattern for Spark-specific functions implemented outside the standard builtin ScalarFunction chain.

What changes are included in this PR?

This PR:

• adds Spark MapConcat expression conversion in NativeConverters
• introduces a dedicated native implementation in spark_map.rs
• registers Spark_MapConcat in datafusion-ext-functions
• adds a regression test in AuronFunctionSuite

Are there any user-facing changes?

NO.

How was this patch tested?

CI.

[Copilot]

Pull request overview

Adds native execution support for Spark SQL map_concat(...) by converting Spark’s MapConcat expression into an extension scalar function and implementing the function in the native engine, with a regression test to ensure queries no longer fall back.

Changes:

• Add Spark MapConcat expression conversion to NativeConverters via buildExtScalarFunction("Spark_MapConcat", ...).
• Implement Spark_MapConcat as a new native extension function (spark_map::map_concat) and register it in datafusion-ext-functions.
• Add a Spark-side regression test in AuronFunctionSuite.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeConverters.scala	Converts Spark MapConcat to the native extension-function call.
native-engine/datafusion-ext-functions/src/spark_map.rs	New native implementation of map_concat operating on Arrow MapArrays.
native-engine/datafusion-ext-functions/src/lib.rs	Registers Spark_MapConcat so the planner can resolve and execute it.
spark-extension-shims-spark/src/test/scala/org/apache/auron/AuronFunctionSuite.scala	Adds regression coverage to ensure map_concat runs natively and matches Spark results.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.