[SPARK-56768][PYTHON][INFRA] Share SBT compile artifact across pyspark CI jobs

[zhengruifeng]

What changes were proposed in this pull request?

This PR adds a single shared precompile CI job that runs Spark's SBT build once and uploads the resulting target/ trees as a GitHub Actions artifact. The 8 pyspark matrix entries plus the optional pyspark-install entry now consume that artifact instead of re-running the same SBT build themselves. The job is named generically because the same artifact can be reused by sparkr, R, or documentation jobs in follow-ups.

Concretely:

• New precompile job in .github/workflows/build_and_test.yml runs the SBT build:

  ./build/sbt -Phadoop-3 -Pyarn -Pspark-ganglia-lgpl -Phadoop-cloud -Phive \
    -Pkubernetes -Pjvm-profiler -Pkinesis-asl -Phive-thriftserver \
    -Pdocker-integration-tests -Pvolcano \
    Test/package streaming-kinesis-asl-assembly/assembly connect/assembly assembly/package
  

  It tars every target/ directory (excluding ./build/ and ./.git/) with tar -cjf (bzip2), uploads as spark-compile-${{ github.run_id }} with retention-days: 1 so storage is reclaimed within 24h.
  The job's if: gate fires when any of pyspark, pyspark-pandas, or pyspark-install is true in the precondition output, so the artifact is always available for any matrix entry that needs it (including via inputs.jobs overrides used by scheduled / dispatched workflows).
• The pyspark matrix job adds precompile to needs:, downloads and extracts the artifact before running tests, and sets SKIP_BUILD: true in env.
• dev/run-tests.py now skips build_apache_spark and build_spark_assembly_sbt when SKIP_BUILD is set, matching the existing SKIP_UNIDOC / SKIP_MIMA pattern.

SBT invocations: before vs. after

Every pyspark matrix entry today drives dev/run-tests.py, which makes two SBT calls back-to-back (build_spark_sbt at dev/run-tests.py:647 then build_spark_assembly_sbt at dev/run-tests.py:656):

# build_spark_sbt
./build/sbt <11 profiles> Test/package streaming-kinesis-asl-assembly/assembly connect/assembly
# build_spark_assembly_sbt
./build/sbt <11 profiles> assembly/package

The 11 profiles, identical across all 8 entries: -Phadoop-3 -Pyarn -Phive -Phive-thriftserver -Pkubernetes -Phadoop-cloud -Pjvm-profiler -Pspark-ganglia-lgpl -Pkinesis-asl -Pdocker-integration-tests -Pvolcano.

After this PR, with SKIP_BUILD: true set on the matrix job, both calls are gated off; no SBT compile runs in the matrix entry at all. The new precompile job runs one SBT invocation that combines all four goals (safe because SKIP_MIMA=true in the pyspark job, so the original split for dev/mima is moot here):

	SBT compile invocations per pyspark matrix entry	Total SBT compile invocations across the matrix
Before	2	16
After	0	1 (in the new precompile job, all 4 goals combined)

The produced target/ is byte-equivalent — same goals, same profiles, same Scala/Java/Hadoop versions.

Why are the changes needed?

Each of the 8 pyspark matrix jobs runs the same ~13m27s SBT compile independently. Across a single CI run that's roughly 108m of redundant compile time, against a per-run total of ~700m. This change deduplicates that work.

Estimated savings, based on a recent run of Build and test:

	Per-run CI time
Redundant SBT compile today (8 pyspark matrix entries)	~108m
Add back: shared build + artifact transfer	~19m
Net CI compute saved per run	~89m (~13% of total)

Wall clock of the workflow is roughly unchanged. The build was previously parallel-hidden inside each matrix runner; sharing it serializes one ~13m build before the matrix, but the slowest matrix runner shrinks by the same amount, so the critical path is similar (within a few minutes).

Does this PR introduce any user-facing change?

No. CI infrastructure change only.

How was this patch tested?

The change is exercised by the CI run of this PR itself:

• If precompile succeeds and produces an artifact of reasonable size, the build phase works.
• If the pyspark matrix completes normally on top of the downloaded artifact, the artifact is sufficient and SKIP_BUILD is correctly skipping the local compile.

A few things to watch in the first run:

• Artifact size. Spark's combined target/ is roughly 1-3 GB raw; expect ~600 MB-1 GB after bzip2. The "Package compile output" step prints the size with ls -lh. If it ever gets close to GHA's 10 GB per-artifact cap we should slim the find pattern (e.g., exclude target/streams and intermediate scaladoc).
• bzip2 in the test images. The pyspark Docker images need bzip2 for the extract step. It is in the bzip2 package and present in every standard Ubuntu base image.

The doctests in dev/sparktestsupport/utils.py continue to pass; no logic in is-changed.py or the module graph was changed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.7)

[gaogaotiantian]

First of all, I think this is super useful.

This extra task however, takes an extra slot from out 20 concurrent job limit. I'm definitely not saying we should do this - we should definitely do it, without any question, but we also need to think about if it's worth it to give it a separate slot, or if we can combine this with some other prereq jobs (for example, when we detected that we need to build it, we just build).

Another observation is this

Task	download/upload	tar/untar
Compile	1m2s	4m52s
Use	25s	2m38s

which makes me wonder - maybe we should use a less aggressive compress algorithm? zstd or gzip? We are spending much more time to compress/decompress, than upload/download.