[SPARK-47688][CORE] Support three methods of the log concatenation in the structured logging framework

[panbingkun]

What changes were proposed in this pull request?

The pr aims to support three methods of the log concatenation in the structured logging framework. eg:

• 1.log"... MDC(...) $normal_variables ..."

log"Driver terminated with exit code ${MDC(EXIT_CODE, exitCode)}! Shutting down. $remoteAddress"

• 2.s"... $normal_variables ..." ++ log"... MDC(...) ..."

log"Driver terminated with exit code ${MDC(EXIT_CODE, exitCode)}! Shutting down." ++ s"$remoteAddress"

• 3.log"... MDC(...) ..." ++ s"... $normal_variables ..."

s"$remoteAddress" ++ log"Driver terminated with exit code ${MDC(EXIT_CODE, exitCode)}! Shutting down."

Why are the changes needed?

Sometimes we just want to make some simple variable replacements in the log text, without the need for MDC
At present, the above logs, compile will fail.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Add new UT.

Was this patch authored or co-authored using generative AI tooling?

No.

[panbingkun]

cc @gengliangwang @cloud-fan @HyukjinKwon

[panbingkun]

The method name cannot be called + and does not take effect, so it is called ++

[panbingkun]

In addition, in String ++ String, ++ is an alias for contact, which sounds reasonable.

[panbingkun]

Some of the comments were not added. I want to add them to another PR #45808 , after this is merged, as they should conflict during the merge, I can handle them all at once.

[gengliangwang]

The args here is intended to MDC. So that we can enforce all the variables to be MDC.

[gengliangwang]

log"Driver terminated with exit code ${MDC(EXIT_CODE, exitCode)}! Shutting down. $remoteAddress"

In this case, we should convert remoteAddress as MDC as well.

[panbingkun]

@gengliangwang
Do we need to support the second(Sting + MDC) and third(MDC + Sting) methods besides the first one?

[gengliangwang]

Do we need to support the second(Sting + MDC) and third(MDC + Sting) methods besides the first one?

So by the end of this project, all the log entries containing variables will use logError(log"..."). logError(s"...") will not be allowed.
I am not sure about how easy to build such scala style check if we allow log"..." + str . If it is complicated, we can stay with MDC concat only.

[panbingkun]

Do we need to support the second(Sting + MDC) and third(MDC + Sting) methods besides the first one?

So by the end of this project, all the log entries containing variables will use logError(log"..."). logError(s"...") will not be allowed. I am not sure about how easy to build such scala style check if we allow log"..." + str . If it is complicated, we can stay with MDC concat only.

Okay, regarding this issue, I will investigate it carefully again.