Skip to content

Commit

Permalink
[KYUUBI #1675][FEATURE] Redact secret information from SparkSQLEngine…
Browse files Browse the repository at this point in the history
… log

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Secret informations on log is dangerous on the production environment. Secret info should be redacted in some way.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
capture from Spark Driver's STDERR
![Screen Shot 2022-01-04 at 11 56 43 AM](https://user-images.githubusercontent.com/1802676/148022789-f005748e-4efe-47da-8d2d-87ecdd233910.png)

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1677 from minyk/kyuubi-1675.

Closes #1675

6a39057 [minyk] redact statement in the log

Authored-by: minyk <minykreva@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
  • Loading branch information
minyk authored and yaooqinn committed Jan 4, 2022
1 parent 4753310 commit 0f8ebb6
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 2 deletions.
Expand Up @@ -58,7 +58,8 @@ abstract class SparkOperation(opType: OperationType, session: Session)

protected def resultSchema: StructType

def redactedStatement: String = redact(spark.sessionState.conf.stringRedactionPattern, statement)
override def redactedStatement: String =
redact(spark.sessionState.conf.stringRedactionPattern, statement)

protected def cleanup(targetState: OperationState): Unit = state.synchronized {
if (!isTerminalState(state)) {
Expand Down
Expand Up @@ -60,6 +60,8 @@ abstract class AbstractOperation(opType: OperationType, session: Session)

def statement: String = opType.toString

def redactedStatement: String = statement

protected def setHasResultSet(hasResultSet: Boolean): Unit = {
this.hasResultSet = hasResultSet
handle.setHasResultSet(hasResultSet)
Expand All @@ -80,7 +82,7 @@ abstract class AbstractOperation(opType: OperationType, session: Session)
case _ =>
}
info(s"Processing ${session.user}'s query[$statementId]: ${state.name} -> ${newState.name}," +
s" statement: $statement$timeCost")
s" statement: $redactedStatement$timeCost")
state = newState
lastAccessTime = System.currentTimeMillis()
}
Expand Down

0 comments on commit 0f8ebb6

Please sign in to comment.