[SPARK-56833][TESTS] Add -XX:+EnableDynamicAgentLoading to test JVM options to suppress JEP 451 warnings

[dongjoon-hyun]

What changes were proposed in this pull request?

Add -XX:+EnableDynamicAgentLoading to the test-only JVM options in both SBT (project/SparkBuild.scala) and Maven (pom.xml).

Why are the changes needed?

Per JEP 451: Prepare to Disallow the Dynamic Loading of Agents, JDK 21+ warns when a Java agent is loaded dynamically (e.g. Mockito self-attaching ByteBuddy), and a future release will disallow it by default. Setting this flag on the test JVM silences the warning today and keeps tests working once the default flips. Production JVM options are intentionally untouched since Spark itself does not dynamically attach agents.

Currently, Apache Spark CIs show the following message at ERROR level.

• Java 21: https://github.com/apache/spark/actions/runs/25592460535/job/75172442863
• Java 25: https://github.com/apache/spark/actions/runs/25664584995/job/75333859273

[info] Test run started (JUnit Jupiter)
[info] Test org.apache.spark.network.TransportRequestHandlerSuite#handleMergedBlockMetaRequest() started
[error] Mockito is currently self-attaching to enable the inline-mock-maker. This will no longer work in future releases of the JDK. Please add Mockito as an agent to your build as described in Mockito's documentation: https://javadoc.io/doc/org.mockito/mockito-core/latest/org.mockito/org/mockito/Mockito.html#0.3
WARNING: A Java agent has been loaded dynamically (/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/net/bytebuddy/byte-buddy-agent/1.18.4/byte-buddy-agent-1.18.4.jar)
WARNING: If a serviceability tool is in use, please run with -XX:+EnableDynamicAgentLoading to hide this warning
WARNING: If a serviceability tool is not in use, please run with -Djdk.instrument.traceUsage for more information
WARNING: Dynamic loading of agents will be disallowed by default in a future release
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs and manual tests.

BEFORE

$ build/sbt "network-common/testOnly *.TransportRequestHandlerSuite" > /dev/null
Using SPARK_LOCAL_IP=localhost
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by scala.runtime.LazyVals$ (file:/Users/dongjoon/APACHE/spark-merge/build/sbt-launch-1.12.8.jar)
WARNING: Please consider reporting this to the maintainers of class scala.runtime.LazyVals$
WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: package sun.security.action not in java.base
WARNING: A Java agent has been loaded dynamically (/Users/dongjoon/.m2/repository/net/bytebuddy/byte-buddy-agent/1.18.4/byte-buddy-agent-1.18.4.jar)
WARNING: If a serviceability tool is in use, please run with -XX:+EnableDynamicAgentLoading to hide this warning
WARNING: If a serviceability tool is not in use, please run with -Djdk.instrument.traceUsage for more information
WARNING: Dynamic loading of agents will be disallowed by default in a future release
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::allocateMemory has been called by io.netty.util.internal.PlatformDependent0$2 (file:/Users/dongjoon/.m2/repository/io/netty/netty-common/4.2.12.Final/netty-common-4.2.12.Final.jar)
WARNING: Please consider reporting this to the maintainers of class io.netty.util.internal.PlatformDependent0$2
WARNING: sun.misc.Unsafe::allocateMemory will be removed in a future release

AFTER

$ build/sbt "network-common/testOnly *.TransportRequestHandlerSuite" > /dev/null
Using SPARK_LOCAL_IP=localhost
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by scala.runtime.LazyVals$ (file:/Users/dongjoon/APACHE/spark-merge/.claude/worktrees/youthful-sanderson-0fc91a/build/sbt-launch-1.12.8.jar)
WARNING: Please consider reporting this to the maintainers of class scala.runtime.LazyVals$
WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: package sun.security.action not in java.base
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::allocateMemory has been called by io.netty.util.internal.PlatformDependent0$2 (file:/Users/dongjoon/.m2/repository/io/netty/netty-common/4.2.12.Final/netty-common-4.2.12.Final.jar)
WARNING: Please consider reporting this to the maintainers of class io.netty.util.internal.PlatformDependent0$2
WARNING: sun.misc.Unsafe::allocateMemory will be removed in a future release

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 4.7 (1M context)

[dongjoon-hyun]

Could you review this test PR, @peter-toth ?

[dongjoon-hyun]

Thank you so much, @peter-toth !

[dongjoon-hyun]

Merged to master/4.x/4.2.