[MINOR][TESTS] Use SparkContext.setLogLevel in QuietTest

[HyukjinKwon]

What changes were proposed in this pull request?

This PR proposes to use SparkContext.setLogLevel in QuietTest

Why are the changes needed?

To avoid the direct update of logger but reuses SparkContext.setLogLevel.

Does this PR introduce any user-facing change?

No, test-only.

How was this patch tested?

Manually tested.

Was this patch authored or co-authored using generative AI tooling?

No.

[HyukjinKwon]

Merged to master.

[LuciferYang]

This may have some negative impacts, as I have noticed some new flaky tests in recent GAs that might be related to this:

• https://github.com/apache/spark/actions/runs/12044265847/job/33581052605
• https://github.com/apache/spark/actions/runs/12043870433/job/33579994907
• https://github.com/LuciferYang/spark/actions/runs/12034481725/job/33551257684
• https://github.com/LuciferYang/spark/actions/runs/12034633745/job/33554178490

py4j.protocol.Py4JJavaError: An error occurred while calling o20.setLogLevel.
4768: java.util.ConcurrentModificationException
4769	at java.base/java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1784)
4770	at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
4771	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
4772	at java.base/java.util.WeakHashMap$ValueSpliterator.forEachRemaining(WeakHashMap.java:1217)
4773	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
4774	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
4775	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
4776	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
4777	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
4778	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
4779	at org.apache.logging.log4j.core.LoggerContext.updateLoggers(LoggerContext.java:776)
4780	at org.apache.logging.log4j.core.LoggerContext.updateLoggers(LoggerContext.java:766)
4781	at org.apache.spark.util.Utils$.setLogLevel(Utils.scala:2322)
4782	at org.apache.spark.util.Utils$.setLogLevelIfNeeded(Utils.scala:2331)
4783	at org.apache.spark.SparkContext.setLogLevel(SparkContext.scala:400)
4784	at org.apache.spark.api.java.JavaSparkContext.setLogLevel(JavaSparkContext.scala:675)
4785	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
4786	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
4787	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
4788	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
4789	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
4790	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
4791	at py4j.Gateway.invoke(Gateway.java:282)
4792	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
4793	at py4j.commands.CallCommand.execute(CallCommand.java:79)
4794	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
4795	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
4796	at java.base/java.lang.Thread.run(Thread.java:840) 

Could this pr potentially trigger the aforementioned concurrent exceptions? Do you have time to confirm it? Thanks @HyukjinKwon

I have created https://issues.apache.org/jira/browse/SPARK-50434

[HyukjinKwon]

let me make a fix

[HyukjinKwon]

BTW, this PR doesn't trigger the exception. It was already there before so I made this fix and see if it fixes :-).

[dongjoon-hyun]

+1, LGTM.

[HyukjinKwon]

let me revert this - I think it becomes more flaky as @LuciferYang said. I am sorry for back and forth. I am taking a look for a proper fix