[SPARK-34970][3.0][SQL] Redact map-type options in the output of explain() #32085
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…f explain() The `explain()` method prints the arguments of tree nodes in logical/physical plans. The arguments could contain a map-type option that contains sensitive data. We should map-type options in the output of `explain()`. Otherwise, we will see sensitive data in explain output or Spark UI.  Data security. Yes, redact the map-type options in the output of `explain()` Unit tests Closes apache#32066 from gengliangwang/redactOptions. Authored-by: Gengliang Wang <ltnwgl@gmail.com> Signed-off-by: Gengliang Wang <ltnwgl@gmail.com>
gengliangwang
changed the title
|
This is to backport #32066 to branch-3.0 |
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
maropu
approved these changes
Apr 8, 2021
|
Test build #137052 has finished for PR 32085 at commit
|
HyukjinKwon
approved these changes
Apr 8, 2021
There was a problem hiding this comment.
Thank you for making a backport, @gengliangwang .
The second test case fails like the following. Could you fix it?
[info] ExplainSuite:
08:38:16.800 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[info] - SPARK-34970: Redact Map type options in explain output (1 second, 409 milliseconds)
[info] - SPARK-34970: Redact CaseInsensitiveMap type options in explain output *** FAILED *** (1 second, 884 milliseconds)
[info] "== Parsed Logical Plan ==
[info] 'UnresolvedRelation [t]
[info]
[info] == Analyzed Logical Plan ==
[info] id: bigint
[info] SubqueryAlias spark_catalog.default.t
[info] +- Relation[id#xL] json
[info]
[info] == Optimized Logical Plan ==
[info] Relation[id#xL] json
[info]
[info] == Physical Plan ==
[info] FileScan json default.t[id#xL] Batched: false, DataFilters: [], Format: JSON, Location: InMemoryFileIndex[file:/Users/dongjoon/PRS/SPARK-PR-32085/sql/core/spark-warehouse/org.apache.spa..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:bigint>
[info]
[info] " did not contain "value" (ExplainSuite.scala:66)
|
@dongjoon-hyun yes. |
dongjoon-hyun
changed the title
dongjoon-hyun
approved these changes
Apr 8, 2021
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
dongjoon-hyun
pushed a commit
that referenced
this pull request
Apr 8, 2021
…ain() ### What changes were proposed in this pull request? The `explain()` method prints the arguments of tree nodes in logical/physical plans. The arguments could contain a map-type option that contains sensitive data. We should map-type options in the output of `explain()`. Otherwise, we will see sensitive data in explain output or Spark UI.  ### Why are the changes needed? Data security. ### Does this PR introduce _any_ user-facing change? Yes, redact the map-type options in the output of `explain()` ### How was this patch tested? Unit tests Closes #32085 from gengliangwang/redact3.0. Authored-by: Gengliang Wang <ltnwgl@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
|
Merged to branch-3.0. |
|
Test build #137091 has finished for PR 32085 at commit
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The
explain()method prints the arguments of tree nodes in logical/physical plans. The arguments could contain a map-type option that contains sensitive data.We should map-type options in the output of
explain(). Otherwise, we will see sensitive data in explain output or Spark UI.Why are the changes needed?
Data security.
Does this PR introduce any user-facing change?
Yes, redact the map-type options in the output of
explain()How was this patch tested?
Unit tests