New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ARROW-16901: [R][CI] Prune R nightly builds #13453
ARROW-16901: [R][CI] Prune R nightly builds #13453
Conversation
@raulcd could you give this a first pass? |
|
0ad8d38
to
f29cccf
Compare
Thanks for the review @kou ! While integrating the changes I was already annoyed to do everything twice so I refactored the syncing as a local action. This way it can be easily reused by other components to upload their nightly builds (e.g. java). Here is an example run in a test repo |
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1
This PR adds pruning to the nightly R upload, 14 versions will be kept by default.
I have removed the
burnett01/rsync-deployments
actions because the use of docker for this was unnecessary and the action can only upload to a remote. This new manual version also utilizes host key checking for which I createdsecrets.NIGHTLIES_RSYNC_HOST_KEY
(which should contain the result ofssh-keyscan -H nightlies.apache.org 2> /dev/null
and needs to be added to apache/arrow before this can run). This way we are no longer depending on the action and it's associated Dockerfile (drinternet/rsync
).We might want to refactor this into a local action for use with all nightly upload jobs.
The pruning is not super efficient as we download the whole nightly repository (on cache miss). This could be avoided for the libarrow files, they could possibly be deleted via ssh instead but we need to download all R packages as
tools::write_PACKAGES
needs access to each archive.