[SPARK-58664][PYTHON] Factor out repeated paths in PySpark Classic's setup.py

[nchammas]

What changes were proposed in this pull request?

Factor out repeated paths into a single list of install paths that is used throughout the PySpark Classic install script.

Make the packaging cleanup a bit more tolerant to partial staging of dependencies by having it keep going if files are missing. I also changed the check to look at the actual filesystem object -- is it a symlink or not? -- rather than repeat the general check on OS capabilities.

I preserved the order in which the paths are installed and then cleaned up.

Why are the changes needed?

General code hygiene. We shouldn't have to manually touch multiple blocks of code that are supposed to always be in sync.

If a file is missing during cleanup, I think it's a marginal improvement to have the cleanup just move on to the next one. Checking the file's type directly during cleanup is also more robust than checking a general capability.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

dev/make-distribution.sh --pip
dev/run-pip-tests

Was this patch authored or co-authored using generative AI tooling?

I wrote this with assistance from GitHub Copilot.

[dongjoon-hyun]

LGTM. Nice cleanup — I verified that the install/cleanup ordering is preserved and that the removed variables have no remaining usages (JARS_TARGET/SCRIPTS_TARGET, which are still used elsewhere, are correctly kept). The cleanup change is a real robustness improvement: since it runs in the finally: block, the old code could mask the original exception with a FileNotFoundError when staging failed partway, while the new per-target islink check lets the root cause propagate.

A couple of minor, non-blocking notes:

• In the sdist-install scenario (in_spark=False with deps/ present), JARS_PATH remains an empty list [], which gets passed into InstallPath(source: str, ...). Harmless at runtime since INSTALL_PATHS is only iterated when in_spark is true, but it contradicts the declared type.
• Silently skipping missing targets also drops the signal that staging was partial; a one-line note to stderr when skipping could be worth considering.