Jenkins CI capture / backup - #105
Merged
Merged
Conversation
We had another incident over the last few days where: * CI jobs had disappeared * All worker credentials were invalidated * Some workers had completely disappeared Restoring it was a pain. Some items were restored with help from ASF infra but they ended up in a different place (path), others were restored that were from more than 5 years ago (referencing 2.3 release branches). Other bits and pieces had to be recreated and validated by hand. Here we try to capture a bit more now to hopefully be able to restore it quicker. The job definition XML files are now included along with a few more yaml config files besides just `jenkins.yaml`. It's still not a single backup/restore kind of a default but it's a bit more thorough than before. This will eventually becomes stale but it's better to have something at least than nothing at all.
big-r81
approved these changes
Jul 27, 2026
big-r81
left a comment
big-r81
left a comment
Contributor
There was a problem hiding this comment.
Thx Nick for collecting the pieces together
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We had another incident over the last few days where:
Restoring it was a pain. Some items were restored with help from ASF infra but they ended up in a different place (path), others were restored that were from more than 5 years ago (referencing 2.3 release branches). Other bits and pieces had to be recreated and validated by hand. Here we try to capture a bit more now to hopefully be able to restore it quicker. The job definition XML files are now included along with a few more yaml config files besides just
jenkins.yaml. It's still not a single backup/restore kind of a default but it's a bit more thorough than before.This will eventually becomes stale but it's better to have something at least than nothing at all.