protected runner jobs at uo: trust e4s-uo signing key

[eugeneswalker]

Excluding these changes from the aarch64 stacks as we do not plan to provide aarch64 runners for the protected pool.

@kotfic @scottwittenburg

[scottwittenburg]

The last time I was thinking about this, I thought we were only having UO signing to support the power builds, so #32710 had a bit to support it, but only for that stack.

Based on this PR and the conversation I saw in slack today though, it seems like the plan is to have UO do builds for all stacks?

[eugeneswalker]

... it seems like the plan is to have UO do builds for all stacks?

Well, the thing is, we need at least a couple of our X86_64 machines in the protected runner pool in order to start running GPU tests using our GPUs. We would like to leverage the existing spack test integration in CI to try testing some of the packages like kokkos +cuda on the GPU-enabled runners we have.

Maybe we could flesh this out more in one of the CI meetings if there are some aspects we need to consider prior to merge? Bottom line I think we need to have at least a couple of our X86_64 GPU instances in the protected runner pool, as well as our power runners to support (a) spack test of GPU-requiring packages as part of CI and (b) E4S power pipeline. What do you think?

[scottwittenburg]

I think we can move ahead with letting UO pick up any protected jobs (instead of restricting it to just power builds). But before we merge this, we'll need to coordinate a PR on spack/spack-infrastructture to provision the UO public key to aws protected runners, and make sure we add that secret wherever it belongs.

[eugeneswalker]

@scottwittenburg I updated this PR to address your comments. Can you review? Is this OK?

[scottwittenburg]

Looks good, thank you! I think this could be merged now, although we can't make the change to actually let UO runners pick up protected jobs until we get the UO public key mounted onto our runners.

[eugeneswalker]

Looks good, thank you! I think this could be merged now, although we can't make the change to actually let UO runners pick up protected jobs until we get the UO public key mounted onto our runners.

Sounds great. Thanks for the quick review.

I'm thinking it might be best to keep this one open until the other work is done, in case we realize we could tweak this to be better. I'm not saying that will happen, but by leaving this open we keep our options open. And since merging now won't have an effect until the other work is done, there is no cost to leaving it open. What do you think?

[scottwittenburg]

I'm thinking it might be best to keep this one open until the other work is done

Yeah, that makes sense.

[kotfic]

K8S infrastructure has been updated to include the e4s.gpg public key which means dependencies built on UO infrastructure should now verify in AWS. I believe once this is rebased on develop it should be good to go.

[eugeneswalker]

Just re-based this on develop. Hoping we get a green CI pipeline here and we can proceed with merge.

[eugeneswalker]

Looks like this is all green and ready to go. @scottwittenburg What do you think?

[scottwittenburg]

Just waiting for some protected build jobs to run so I can validate the e4s key mounted on the image, then I'll approve again and merge. Thanks for the reminder!

[scottwittenburg]

I tested trusting the key that gets mounted in the protected runners now (thanks @kotfic), this is what I see:

gpg: key 954DB486ABE5E7AA: public key "E4S - University of Oregon - Spack 01" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: inserting ownertrust of 6

So the key is properly formatted and I'll merge this now.

[scottwittenburg]

Thanks @eugeneswalker and @kotfic!