[SPARK-35576][SQL] Redact the sensitive info in the result of Set command

[gengliangwang]

What changes were proposed in this pull request?

Currently, the results of following SQL queries are not redacted:

SET [KEY];
SET;

For example:

scala> spark.sql("set javax.jdo.option.ConnectionPassword=123456").show()
+--------------------+------+
|                 key| value|
+--------------------+------+
|javax.jdo.option....|123456|
+--------------------+------+


scala> spark.sql("set javax.jdo.option.ConnectionPassword").show()
+--------------------+------+
|                 key| value|
+--------------------+------+
|javax.jdo.option....|123456|
+--------------------+------+


scala> spark.sql("set").show()
+--------------------+--------------------+
|                 key|               value|
+--------------------+--------------------+
|javax.jdo.option....|              123456|

We should hide the sensitive information and redact the query output.

Why are the changes needed?

Security.

Does this PR introduce any user-facing change?

Yes, the sensitive information in the output of Set commands are redacted

How was this patch tested?

Unit test

[gengliangwang]

I will have another PR for set -v command, which has a different code path and we need to keep the <undefined> results instead of redacting them.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43630/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43631/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43630/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43631/

[SparkQA]

Test build #139110 has finished for PR 32712 at commit a691720.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #139111 has finished for PR 32712 at commit 6716d54.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

+1, LGTM. Thanks, @gengliangwang and all.

[dongjoon-hyun]

Merged to master. There is a conflict. Could you make a backport PR, @gengliangwang ?

[gengliangwang]

@dongjoon-hyun Thanks for merging. I have opened a cherry-pick PR in #32720

[gengliangwang]

I will have another PR for set -v command, which has a different code path and we need to keep the results instead of redacting them.

I just went over the result of "set -v", it seems that there is no such a configuration requiring redaction. I will leave it alone.