Velox backend does not apply/enforce spark.redaction.regex config property when logging out spark config

[xushichangdesmond]

Description

Ideally, spark.redaction.regex should be applied as well

[Yohahaha]

would you submit a PR to support it?

[ArnavBalyan]

Hi I can add support for this. I see two ways to do this:

1. Handle all logs centrally via glog. Glog supports custom sinks where users can redirect logs to addition sinks. But this logic needs to be hacked to disable the default glog behaviour, introduce a custom sink, and redact information there. Currently there is no simple way to do it since glog does not support it. (I have raised a request in glog)
2. Until #1 can be supported, a custom macro can be introduced for sensitive logs. Something like SENSITIVE_LOG(INFO). Which can be called by callers. This would be hard to enforce since LOG(severity) is used commonly.

I feel #2 may be simpler until glog provides an easy way to do this. Would love to get your thoughts cc @FelixYBW @philo-he

[philo-he]

@xushichangdesmond, @ArnavBalyan, if we only care about sensitive information in Spark config, maybe, we can just locate the code places where Spark config properties are printed and redact the log there.

See the below Gluten's log when I run a test.

VeloxRuntime.cc:142] VeloxRuntime session config:
 [spark.sql.session.timeZone, Asia/Shanghai]
 [spark.gluten.ugi.tokens, ]
 [spark.sql.files.ignoreMissingFiles, false]
 [spark.sql.caseSensitive, false]
 [spark.gluten.memory.task.offHeap.size.in.bytes, 17895697066]
 [spark.gluten.sql.columnar.backend.velox.window.type, sort]
 [spark.gluten.sql.session.timeZone.default, Etc/UTC]
 [spark.gluten.ugi.username, root]
 [spark.gluten.sql.debug, true]

[ArnavBalyan]

Yes, this makes sense, maybe we can keep a util function anyone in the future can use if logging sensitive configs etc. Will wait for @xushichangdesmond, but I think the ask should be fulfilled with this thanks

[xushichangdesmond]

thanks much for this