[SPARK-55497][BUILD][TESTS] Use jupyterTestFramework instead of TestFrameworks.JUnit

[sarutak]

What changes were proposed in this pull request?

This PR fixes an issue that TestFrameworks.JUnit is still used even though newer jupiter plugin (com.github.sbt.junit.sbt-jupiter-interface) is used.
As a result, -Dtests.include.tags and -Dtests.exclude.tags don't work.

$ build/sbt -Dtest.include.tags=org.apache.spark.tags.SlowHiveTest unsafe/test
...
[info] Run completed in 2 seconds, 877 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 2, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] No tests were executed.
[info] Passed: Total 97, Failed 0, Errors 0, Passed 97

Why are the changes needed?

Bug fix.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Confirmed -Dtests.exclude.tags works.

$ build/sbt -Dtest.include.tags=org.apache.spark.tags.SlowHiveTest unsafe/test
...
[info] Run completed in 738 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 2, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] No tests were executed.
[info] Passed: Total 0, Failed 0, Errors 0, Passed 0
[success] Total time: 2 s, completed 2026/02/12 19:33:53

There are no tagged .java tests so I didn't confirm if -Dtest.exclude.tags works.

Was this patch authored or co-authored using generative AI tooling?

No.

[pan3793]

how far should it be backported?

[sarutak]

@pan3793
This issue seems to affect branch-4.0 and newer branches.