[AURON #2173] [Correctness Testing] Add function-related tests

[lyne7-sc]

Which issue does this PR close?

Closes #2173

Rationale for this change

This pr extends some functions correctness tests to other supported Spark versions.

It helps identify version-specific behavior differences and existing unsupported cases across Spark versions.

What changes are included in this PR?

• enable functions correctness tests for:
  • Spark 3.1
  • Spark 3.2
  • Spark 3.4
  • Spark 3.5
  • Spark 4.0
  • Spark 4.1
• annotate existing failing cases with preliminary failure analysis generated with LLM assistance

Are there any user-facing changes?

no.

How was this patch tested?

n/a

[Copilot]

Pull request overview

Extends Auron’s Spark correctness test coverage for built-in functions across additional supported Spark versions (3.1, 3.2, 3.4, 3.5, 4.0, 4.1), and updates shared test infrastructure to handle Spark 4 API differences.

Changes:

• Add per-version wrapper suites to run Spark’s existing function suites under SparkQueryTestsBase.
• Enable these suites in each Spark-version test module via AuronSparkTestSettings, with documented exclusions for known native-engine mismatches.
• Update common test utilities for Spark 4 compatibility (e.g., expression-to-Column creation and DataFrame internals), plus minor formatting-only adjustments in the extension code.

Reviewed changes

Copilot reviewed 38 out of 42 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
spark-extension/src/main/scala/org/apache/spark/sql/auron/util/TaskContextHelper.scala	Formatting-only adjustment in thread name construction.
spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeHelper.scala	Formatting-only adjustment in memory logging statement.
spark-extension/src/main/scala/org/apache/spark/sql/auron/AuronSparkSessionExtension.scala	Formatting-only adjustment in plan logging.
spark-extension/src/main/scala/org/apache/spark/sql/auron/AuronConverters.scala	Formatting-only adjustment in error string interpolation/indentation.
auron-spark-tests/common/src/test/scala/org/apache/spark/sql/SparkQueryTestsBase.scala	Use queryExecution APIs for plan/sparkSession access (Spark 4-compatible).
auron-spark-tests/common/src/test/scala/org/apache/spark/sql/SparkExpressionTestsBase.scala	Add Spark-version-specific helpers for Column creation and internal DF creation (Spark 4 compatibility).
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/AuronDataFrameFunctionsSuite.scala	Spark 3.1: enable DataFrame function suite under Auron test base.
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/AuronDateFunctionsSuite.scala	Spark 3.1: enable date function suite under Auron test base.
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/AuronMathFunctionsSuite.scala	Spark 3.1: enable math function suite under Auron test base.
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/AuronMiscFunctionsSuite.scala	Spark 3.1: enable misc function suite under Auron test base.
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/AuronStringFunctionsSuite.scala	Spark 3.1: enable string function suite under Auron test base.
auron-spark-tests/spark31/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala	Spark 3.1: register/enable function suites with known-failure exclusions.
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/AuronDataFrameFunctionsSuite.scala	Spark 3.2: enable DataFrame function suite under Auron test base.
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/AuronDateFunctionsSuite.scala	Spark 3.2: enable date function suite under Auron test base.
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/AuronMathFunctionsSuite.scala	Spark 3.2: enable math function suite under Auron test base.
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/AuronMiscFunctionsSuite.scala	Spark 3.2: enable misc function suite under Auron test base.
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/AuronStringFunctionsSuite.scala	Spark 3.2: enable string function suite under Auron test base.
auron-spark-tests/spark32/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala	Spark 3.2: register/enable function suites with known-failure exclusions.
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/AuronDataFrameFunctionsSuite.scala	Spark 3.4: enable DataFrame function suite under Auron test base.
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/AuronDateFunctionsSuite.scala	Spark 3.4: enable date function suite under Auron test base.
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/AuronMathFunctionsSuite.scala	Spark 3.4: enable math function suite under Auron test base.
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/AuronMiscFunctionsSuite.scala	Spark 3.4: enable misc function suite under Auron test base.
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/AuronStringFunctionsSuite.scala	Spark 3.4: enable string function suite under Auron test base.
auron-spark-tests/spark34/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala	Spark 3.4: register/enable function suites with known-failure exclusions.
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/AuronDataFrameFunctionsSuite.scala	Spark 3.5: enable DataFrame function suite under Auron test base.
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/AuronDateFunctionsSuite.scala	Spark 3.5: enable date function suite under Auron test base.
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/AuronMathFunctionsSuite.scala	Spark 3.5: enable math function suite under Auron test base.
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/AuronMiscFunctionsSuite.scala	Spark 3.5: enable misc function suite under Auron test base.
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/AuronStringFunctionsSuite.scala	Spark 3.5: enable string function suite under Auron test base.
auron-spark-tests/spark35/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala	Spark 3.5: register/enable function suites with known-failure exclusions.
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/AuronDataFrameFunctionsSuite.scala	Spark 4.0: enable DataFrame function suite under Auron test base.
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/AuronDateFunctionsSuite.scala	Spark 4.0: enable date function suite under Auron test base.
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/AuronMathFunctionsSuite.scala	Spark 4.0: enable math function suite under Auron test base.
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/AuronMiscFunctionsSuite.scala	Spark 4.0: enable misc function suite under Auron test base.
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/AuronStringFunctionsSuite.scala	Spark 4.0: enable string function suite under Auron test base.
auron-spark-tests/spark40/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala	Spark 4.0: register/enable function suites with known-failure exclusions.
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/AuronDataFrameFunctionsSuite.scala	Spark 4.1: enable DataFrame function suite under Auron test base.
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/AuronDateFunctionsSuite.scala	Spark 4.1: enable date function suite under Auron test base.
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/AuronMathFunctionsSuite.scala	Spark 4.1: enable math function suite under Auron test base.
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/AuronMiscFunctionsSuite.scala	Spark 4.1: enable misc function suite under Auron test base.
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/AuronStringFunctionsSuite.scala	Spark 4.1: enable string function suite under Auron test base.
auron-spark-tests/spark41/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala	Spark 4.1: register/enable function suites with known-failure exclusions.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[slfan1989]

@lyne7-sc Thanks for the contribution! @ShreyeshArangath @richox @yew1eb Thanks for the review!

[lyne7-sc]

Thanks everyone for the reviews!