[SPARK-45826][SQL] Add a SQL config for stack traces in DataFrame query context

[MaxGekk]

What changes were proposed in this pull request?

In the PR, I propose to add new SQL config spark.sql.stackTracesInDataFrameContext which defines how many non-Spark stack traces should be captured into DataFrame query context. By default, the config is set to 1.

Why are the changes needed?

To improve user experience with Spark SQL. When users troubleshoot an issue, they might need more stack traces in the DataFrame context. For example:

scala> spark.conf.set("spark.sql.ansi.enabled", true)
scala> spark.conf.set("spark.sql.stackTracesInDataFrameContext", 3)
scala> spark.range(1).select(lit(1) / lit(0)).collect()
org.apache.spark.SparkArithmeticException: [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error. SQLSTATE: 22012
== DataFrame ==
"div" was called from
<init>(<console>:1)
<init>(<console>:16)
.<clinit>(<console>:1)

Does this PR introduce any user-facing change?

No, it doesn't change the default behaviour.

How was this patch tested?

By running the modified test suite:

$ build/sbt "test:testOnly *QueryContextSuite"

Was this patch authored or co-authored using generative AI tooling?

No.

[cloud-fan]

should it be > 0?

[MaxGekk]

This is extra traces, see the config description:

When it is set to 0, captured one Spark traces and a followed non-Spark trace.

For instance

when it is 0, we return 4 and 5
when it is 1, we return 4, 5, 6

[beliefer]

I'm a bit confused.
Intuitively, I feel that 0 should represent the absence of non-Spark trace.

[MaxGekk]

Intuitively, I feel that 0 should represent the absence of non-Spark trace.

Actually, it works in this way. Let me modify the config and PR description. The slice method excludes the until index.

[beliefer]

I feel that:
default 1: this is consistent with before.
The number of extraOriginTraces should consistent with the number of non-Spark trace size.
Then we use > 0 as the constraint.

[cloud-fan]

Can we put a real example in the PR description?

[cloud-fan]

Can we show the impact to the real error message, instead of val ctx = try { df.select(explode($"*")) } catch { case e: AnalysisException => e.context.head }?

[MaxGekk]

Can we show the impact to the real error message

@cloud-fan I added an example, please, take a look at the PR.

[cloud-fan]

@MaxGekk not quite related in this PR, but what if the expression creation is different from the df creation? like

val divCol = lit(1) / lit(0)
spark.range(1).select(divCol).collect()

[MaxGekk]

@cloud-fan Quite the same:

scala> val divCol = lit(1) / lit(0)
val divCol: org.apache.spark.sql.Column = `/`(1, 0)

scala> spark.range(1).select(divCol).collect()
org.apache.spark.SparkArithmeticException: [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error. SQLSTATE: 22012
== DataFrame ==
"div" was called from
<init>(<console>:1)
<init>(<console>:15)
.<clinit>(<console>:1)

but when I create it in an object:

scala> object Obj1 {
     | val divCol = lit(1) / lit(0)
     | }
object Obj1

scala> spark.range(1).select(Obj1.divCol).collect()
org.apache.spark.SparkArithmeticException: [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error. SQLSTATE: 22012
== DataFrame ==
"div" was called from
Obj1$.<init>(<console>:2)
Obj1$lzycompute$1(<console>:1)
Obj1(<console>:1)

[MaxGekk]

Merging to master. Thank you, @cloud-fan and @beliefer for review.

[beliefer]

late LGTM.