[SPARK-58002][INFRA] Refresh CI build images on a schedule

[nchammas]

What changes were proposed in this pull request?

Rebuild the various images used on CI on a schedule.

The schedule is set to once a day.

Why are the changes needed?

All of our CI build images are based on ubuntu:noble. They are only refreshed if we push changes to the Dockerfiles on master.

spark/.github/workflows/build_infra_images_cache.yml

Lines 31 to 48 in d386172

	push:
	branches:
	- 'master'
	- 'branch-*'
	paths:
	- 'dev/infra/Dockerfile'
	- 'dev/spark-test-image/docs/Dockerfile'
	- 'dev/spark-test-image/lint/Dockerfile'
	- 'dev/spark-test-image/sparkr/Dockerfile'
	- 'dev/spark-test-image/python-minimum/Dockerfile'
	- 'dev/spark-test-image/python-311/Dockerfile'
	- 'dev/spark-test-image/python-312/Dockerfile'
	- 'dev/spark-test-image/python-312-classic-only/Dockerfile'
	- 'dev/spark-test-image/python-312-pandas-3/Dockerfile'
	- 'dev/spark-test-image/python-313/Dockerfile'
	- 'dev/spark-test-image/python-314/Dockerfile'
	- 'dev/spark-test-image/python-314-nogil/Dockerfile'
	- '.github/workflows/build_infra_images_cache.yml'

On the other hand, however, ubuntu:noble is updated roughly once a month:

$ curl -s "https://hub.docker.com/v2/repositories/library/ubuntu/tags?name=noble&page_size=10" \
  | jq -r '.results[] | "\(.tag_last_pushed[:10])  \(.name)"'
2026-07-02  noble-20260610
2026-07-02  noble
2026-06-02  noble-20260509.1
2026-04-15  noble-20260410
2026-04-07  noble-20260324
2026-03-19  noble-20260217
2026-02-18  noble-20260210.1
2026-01-19  noble-20260113
2025-11-15  noble-20251013
2025-10-13  noble-20251001

This means that there are several updates to the base image that we have not incorporated. So when CI runs, it cannot pull from an existing cache and has to build the images from scratch. This takes about 50 minutes! With an up-to-date cache, this drops down to ~1 minute.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

I tested this with a change to build_and_test.yml on one of my branches, but I believe the correct fix is to ensure the that the base images are refreshed on a schedule.

cache-from: |
  type=registry,ref=ghcr.io/${{ github.repository_owner }}/spark/apache-spark-github-action-image-cache:${{ inputs.branch }}
  type=registry,ref=ghcr.io/apache/spark/apache-spark-github-action-image-cache:${{ inputs.branch }}
cache-to: type=registry,ref=ghcr.io/${{ github.repository_owner }}/spark/apache-spark-github-action-image-cache:${{ inputs.branch }},mode=max

Was this patch authored or co-authored using generative AI tooling?

No.

[nchammas]

cc @zhengruifeng and @dongjoon-hyun, since they have recently managed this infrastructure.

[gaogaotiantian]

I think the idea makes sense. We probably don't want actions from other repos to push to our docker cache, so we can't just add cache-to for our regular workflows (there will also be bandwidth/time cost). Checking the cache regularly would save us plenty of time.

This won't update the images for branches other than master, which is probably not a blocker. master is the most active CI branch anyway.

The workflow should be really fast if the cache is hit, which means even if we do it daily, we won't lose 1 hr every day just to build the docker images (correct me if I'm wrong here because we probably need to readjust the frequency).

[nchammas]

Amusingly, we just so happened to push a recent commit that triggered a refresh of the images, so the builds are now fast again. 0fea5ee

Once ubuntu:noble is refreshed, we're going to go back to the ~50 minute image builds on every PR. We could pin ubuntu:noble to a specific version, which would also solve the problem, but I assume part of the reason we're using it is to stay up-to-date with the latest security patches.

In any case, is there anyone else you'd like to get a review from, @gaogaotiantian, or is this PR good to go?

[zhengruifeng]

I think it is a good idea, but I feel it might be too often, maybe a weekly refresh is enough?

I asked AI to analyze the frequency:

Actual run history on master (from the GitHub Actions API)

108 runs over the last ~13 months (2025-06-03 → 2026-07-07; older runs are aged out by GitHub's ~400-day API retention). All 108 were push-triggered — confirming no scheduled refresh.

┌─────────────────────────────────┬───────────┐
│             Metric              │   Value   │
├─────────────────────────────────┼───────────┤
│ Avg runs/month                  │ ~8.3      │
├─────────────────────────────────┼───────────┤
│ Median gap between runs         │ ~1 day    │
├─────────────────────────────────┼───────────┤
│ Mean gap                        │ ~3.7 days │
...

[nchammas]

I'm fine with any frequency (the PR actually started with a weekly schedule). But I think it's nice to have the predictability of a daily refresh vs. depending on updates to the Dockerfiles to trigger the workflow.

With a weekly refresh schedule, if we somehow happen to have a long period of time where no updates are made to the images, that means potentially accepting the ~50 minute image builds on every change to every PR for up to a week.

Another, more sophisticated, approach we could take is to persist the version of ubuntu:noble we last built against, run the job hourly, but only rebuild images if we see the version has changed. But that requires persisting the last seen version somewhere, perhaps in a repo variable.

And to reiterate an earlier suggestion, we could eliminate the need for regular refreshes and solve the problem of surprise rebuilds being needed on every PR by simply pinning all of our uses of ubuntu:noble to a specific version. Is it that important that we always be running the latest version of noble? If not, pinning the version would be the best solution.

[zhengruifeng]

WDYT? @HyukjinKwon @dongjoon-hyun

[gaogaotiantian]

I think it's fine to run it daily because it would be super cheap if we don't need to update the cache. It basically costs us nothing. If we run it weekly it means we could lose a lot of minutes during the week just to build the image.

[HyukjinKwon]

yeah im ok 2

[zhengruifeng]

@gaogaotiantian which branches should we merge this PR? are those CI already decoupled by branch?