ci: Fix buildcache sync so we can always run protected-publish #38866
+166
−46
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
spackbot-app
bot
added
commands
core
scottwittenburg
force-pushed
the
improve-buildcache-sync
branch
2 times, most recently
from
bb2fdf4
to
0d8de58
Compare
|
Here's a simulated |
Provide option to only sync buildcache entries that are properly signed. Also, gracefully ignore any entries from the manifest that did not get created created in the pipeline. Add tests of new and previously untested code paths.
scottwittenburg
force-pushed
the
improve-buildcache-sync
branch
from
0d8de58
to
92d811f
Compare
kwryankrattiger
previously approved these changes
Jul 14, 2023
This was referenced Jul 20, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR makes some improvements to the
spack buildcache synccommand, configures theprotected-publishjob to trust the public reputational signing key (and ensure sync'd binaries have been signed with it), and sets theprotected-publishjob to runalwaysinstead of onlyon_success.Motivation
Spack pipelines rely on this command in the
protected-publishjob, a final pipeline job (scheduled in protected pipelines only) which runs after all stack-specific child pipelines have completed. The goal of the job is to copy any specs rebuilt in the pipeline from stack-specific mirrors to the root.Until now, this job only ran when all the child pipelines succeeded, but this is causing the top-level mirror to be missing a lot of specs. By making the
spack buildcache synccommand more resilient to certain kinds of errors, we will be able to configure theprotected-publishjob to run always.The essential modifications are the following: The command should only sync buildcache entries which are signed with a trusted key. This way, if any
sign-pkgsjobs fail, we don't end up with binaries at the top-level which are still signed with the intermediate signing key. Also, if any of the rebuild jobs in a pipeline fail to produce the files listed in the manifest, the command should gracefully ignore those, and continue copying valid, properly signed binaries on a best-effort basis.Specific improvements to the
spack buildcache synccommand--only-verifiedoption to only copy buildcache entries whose signature can be verifiedlocal_pathto thecopy_buildcache_filemethod, as it didn't have the intended effect but added complication.spec.yamlfiles from the list of metadata files to look for.spec.json.sigfile, and only if that fails, try a.spec.jsonfile--only-verifiedand the signature could not be verified), then skip copying the archive file--only-verfiedoption (as well as the--manifest-globoption which was previously untested)