Skip to content

Commit 0f8ebb6

Browse files
minykyaooqinn
authored andcommitted
[KYUUBI #1675][FEATURE] Redact secret information from SparkSQLEngine log
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Secret informations on log is dangerous on the production environment. Secret info should be redacted in some way. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate capture from Spark Driver's STDERR ![Screen Shot 2022-01-04 at 11 56 43 AM](https://user-images.githubusercontent.com/1802676/148022789-f005748e-4efe-47da-8d2d-87ecdd233910.png) - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1677 from minyk/kyuubi-1675. Closes #1675 6a39057 [minyk] redact statement in the log Authored-by: minyk <minykreva@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>
1 parent 4753310 commit 0f8ebb6

File tree

2 files changed

+5
-2
lines changed

2 files changed

+5
-2
lines changed

externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/SparkOperation.scala

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,8 @@ abstract class SparkOperation(opType: OperationType, session: Session)
5858

5959
protected def resultSchema: StructType
6060

61-
def redactedStatement: String = redact(spark.sessionState.conf.stringRedactionPattern, statement)
61+
override def redactedStatement: String =
62+
redact(spark.sessionState.conf.stringRedactionPattern, statement)
6263

6364
protected def cleanup(targetState: OperationState): Unit = state.synchronized {
6465
if (!isTerminalState(state)) {

kyuubi-common/src/main/scala/org/apache/kyuubi/operation/AbstractOperation.scala

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,8 @@ abstract class AbstractOperation(opType: OperationType, session: Session)
6060

6161
def statement: String = opType.toString
6262

63+
def redactedStatement: String = statement
64+
6365
protected def setHasResultSet(hasResultSet: Boolean): Unit = {
6466
this.hasResultSet = hasResultSet
6567
handle.setHasResultSet(hasResultSet)
@@ -80,7 +82,7 @@ abstract class AbstractOperation(opType: OperationType, session: Session)
8082
case _ =>
8183
}
8284
info(s"Processing ${session.user}'s query[$statementId]: ${state.name} -> ${newState.name}," +
83-
s" statement: $statement$timeCost")
85+
s" statement: $redactedStatement$timeCost")
8486
state = newState
8587
lastAccessTime = System.currentTimeMillis()
8688
}

0 commit comments

Comments
 (0)