[SPARK-57949][PYTHON][TESTS] Use shortest temp base for direct-worker UDS path to fix macOS AF_UNIX path too long

[HyukjinKwon]

What changes were proposed in this pull request?

TestDirectWorkerDispatcher.shortTempBase (in TestDirectWorkerHelpers.scala) chooses a
base directory for the direct-worker Unix-domain socket by taking the first writable
entry among $TMPDIR, $TEMP, $TMP, /tmp. This PR changes it to pick the shortest
usable base, measured by its real (symlink-resolved) path.

Why are the changes needed?

On macOS, $TMPDIR is a long per-user path like
/var/folders/k8/j7r3p6cx43xdqhzy2rmp6tqr0000gn/T/ (~49 chars). Combined with the
createTempDirectory random suffix and the w-<id>.sock leaf, the bound socket path
exceeds the 104-byte macOS sun_path limit, so PythonUDFWorkerSpecificationSuite
fails with:

OSError: AF_UNIX path too long
  at ...TestDirectWorkerDispatcher.throwWorkerExitedBeforeSocket(TestDirectWorkerHelpers.scala:207)

This keeps the scheduled Build / Maven (Scala 2.13, JDK 21, MacOS-26) lane red. Picking the
shortest base makes macOS use /tmp (which resolves to the short /private/tmp) instead of
the long $TMPDIR, keeping the socket path well within the limit. Linux is unaffected — its
per-user temp dirs are already short — and the choice is now deterministic (shortest wins)
rather than order-dependent.

Does this PR introduce any user-facing change?

No. Test-only.

How was this patch tested?

Ran the affected suites on both macos-14 and macos-15 GitHub Actions runners (arm64, the
same short-$TMPDIR layout as macos-26), where $TMPDIR was confirmed to be the long
/var/folders/... path (len=49):

• udf-worker-core: DirectWorkerDispatcherSuite + WorkerSessionSuite — 48 tests, 0 failures
• sql/core: PythonUDFWorkerSpecificationSuite — 1 test, 0 failures (previously
  AF_UNIX path too long)

Both runners green: https://github.com/HyukjinKwon/spark/actions/runs/28765870386

Before this change the same suite fails on macos-26:
https://github.com/apache/spark/actions/runs/28753698265

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code

[HyukjinKwon]

Merge Summary:

• merged into master 226ea56
• merged into branch-4.x fcb7a62

Posted by merge_spark_pr.py