Skip to content
Browse files

[SPARK-27244][CORE] Redact Passwords While Using Option logConf=true

When logConf is set to true, config keys that contain password were printed in cleartext in driver log. This change uses the already present redact method in Utils, to redact all the passwords based on redact pattern in SparkConf and then print the conf to driver log thus ensuring that sensitive information like passwords is not printed in clear text.

This patch was tested through `SparkConfSuite` & then entire unit test through sbt

Please review before opening a pull request.

Closes #24196 from ninadingole/SPARK-27244.

Authored-by: Ninad Ingole <>
Signed-off-by: Sean Owen <>
(cherry picked from commit dbc7ce1)
Signed-off-by: Sean Owen <>
  • Loading branch information...
Ninad Ingole authored and srowen committed Mar 29, 2019
1 parent 530ec52 commit 0975fe9f7e6ed4154988b8df1487ee90976eef49
@@ -579,7 +579,7 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging with Seria
* configuration out for debugging.
def toDebugString: String = {{case (k, v) => k + "=" + v}.mkString("\n")
Utils.redact(this, getAll) { case (k, v) => k + "=" + v }.mkString("\n")

@@ -30,7 +30,7 @@ import org.apache.spark.deploy.history.config._
import org.apache.spark.internal.config._
import org.apache.spark.serializer.{JavaSerializer, KryoRegistrator, KryoSerializer}
import org.apache.spark.util.{ResetSystemProperties, RpcUtils}
import org.apache.spark.util.{ResetSystemProperties, RpcUtils, Utils}

class SparkConfSuite extends SparkFunSuite with LocalSparkContext with ResetSystemProperties {
test("Test byteString conversion") {
@@ -339,6 +339,14 @@ class SparkConfSuite extends SparkFunSuite with LocalSparkContext with ResetSyst

test("SPARK-27244 toDebugString should redact passwords") {
val conf = new SparkConf().set("dummy.password", "dummy-password")

assert(conf.get("dummy.password") === "dummy-password")


class Class1 {}

0 comments on commit 0975fe9

Please sign in to comment.
You can’t perform that action at this time.