remove delete-temp-images job

[jameslamb]

Follow-up to #708.

Proposes completely removing the delete-temp-images job, in favor of relying on the scheduled nightly cleanup at https://github.com/rapidsai/workflows/blob/main/.github/workflows/cleanup_staging.yaml.

Notes for Reviewers

Details

CI here writes images to the rapidsai/staging repo on DockerHub, then later copies them to individual user-facing repos.
To avoid those temporary CI artifacts piling up in the rapidsai/staging repo, pull requests and branch builds run a job called delete-temp-images which does what it sounds like.

In exchange for more aggressive cleaning, this job introduces significant complexity for development here. Most notably, we've observed several instances where that job deletes images before all CI jobs needing them have completed successfully, leading to all of CI needing to be re-run.

Significant effort has been put into trying to avoid that, and we've found it's been difficult to get it right:

some attempts:

• enable jobs retry & multi-arch builds of succeeding docker image builds #702
• ensure test jobs always run on PRs #708

a recent example:

• Drop CUDA 12.2 Docker images #696 (comment)

Ok so how will we clean up?

The workflow at https://github.com/rapidsai/workflows/blob/main/.github/workflows/cleanup_staging.yaml.

It runs once a day and deletes anything from rapidsai/staging that's more than 30 days old.

Benefits of these changes

As described in #708 (comment) ...

CI here will work as it does in other RAPIDS repos.... if any jobs fail for retryable reasons (like network issues), you can safely click "re-run failed jobs" and make incremental progress towards all builds passing.

Also reduces the need to maintain code that has to keep up with the DockerHub API in two places (by deleting ci/delete-temp-images.sh here).

[bdice]

Cool. This is what I hoped #708 would help with.

[jameslamb]

cuspatial notebook failures 😬

Error during notebook tests!
Errors during cuspatial/Taxi_Dropoff_Reverse_Geocoding.ipynb
Errors during cuspatial/trajectory_clustering.ipynb

(build link)

Those are both tested in cuspatial CI (the are not excluded in this list), so that means cuspatial CI will be blocked. Will open an issue there and link it here.

[bdice]

@jameslamb I am checking cuSpatial's CI with this PR: rapidsai/cuspatial#1441 (comment)

[jameslamb]

Great, thanks for pushing this forward @bdice . Looks like CI there did fail on these same notebooks (build link), and that rapidsai/cuspatial#1442 is up to hopefully fix it.

[mroeschke]

rapidsai/cuspatial#1442 has been merged so hopefully a rerun of this PR should pass the CI now

[jameslamb]

Thanks as always @mroeschke !

I just restarted all CI jobs here (assuming we need to do rebuilds to pull in the newest cuspatial).

[bdice]

@jameslamb There was a solver error so I dropped Python 3.9 to make CI pass. See b6e37e3.

[jameslamb]

Ok sure, works for me. Thank you!

[jakirkham]

Thanks James! 🙏

Also thanks everyone who reviewed and pushed fixes here and elsewhere! 🙏

[jameslamb]

Given the set of approvals and passing jobs, I think this is ready to merge. Thanks yall.

[jameslamb]

/merge

[jakirkham]

Thanks James and everyone for reviewing! 🙏