[SPARK-57262][SQL][WEBUI] Job description derived from a query should respect spark.sql.redaction.string.regex

[sarutak]

What changes were proposed in this pull request?

This PR changes SparkSQLDriver.scala to redact a query before setJobDescription.

Why are the changes needed?

In the current implementation, redaction is done in SQLExecution.scala so the description in the table on the top of /SQL/execution is redacted.

But the description in the table on the /jobs page and the one in the table on the bottom of /SQL/execution page are not redacted.

Does this PR introduce any user-facing change?

Yes.

How was this patch tested?

Added new test.
Also confirmed descriptions are redacted in UI.

$ bin/spark-sql --conf spark.sql.redaction.string.regex="secret.*=.*"
spark-sql (default)>  CREATE TABLE test1(secret string);
spark-sql (default)> SELECT * FROM test1 WHERE secret=1;

Was this patch authored or co-authored using generative AI tooling?

Kiro CLI / Claude

[dongjoon-hyun]

cc @cloud-fan , too

[sarutak]

Merged to master/branch-4.x/branch-4.2. Will open backport-PRs for other branches later.
Thank you @dongjoon-hyun for reviewing!

[dongjoon-hyun]

Oh, very sorry, @sarutak .

I found a critical regression with the following use cases. For SQL plan executions without SparkSQLDriver, this PR lost the security redaction. For the record, Hive JDBC and Spark Connect has its own redactions, so they are not affected by this regression.

$ bin/spark-shell -c spark.sql.redaction.string.regex="secret.*=.*"
scala> val s = "SELECT * FROM (SELECT 'secret=1')"
scala> sc.setJobDescription(s)
scala> sql(s).show()
+--------+
|secret=1|
+--------+
|secret=1|
+--------+

BEFORE (4.2.0-preview5)

AFTER (this PR)

Let me revert this commit, @sarutak , because we are under security scan in these days as you know in the security channel.

[dongjoon-hyun]

This is reverted from master/4.x/4.2.

[dongjoon-hyun]

For the record, the above usage is a common use case which we need to consider.

[dongjoon-hyun]

I made a test case addition PR to prevent a regression.

• [SPARK-57297][SQL][TESTS] Add a test that SQL execution description respects spark.sql.redaction.string.regex #56358