CI: 2023 08 11

Attendees

We updated gitlab.spack.io to use AWS's managed ElastiCache rather than an in-cluster redis deployment.
We merged an improvement (spack-infra PR #446) to how we schedule pipeline pods on nodes. For example, x86_64_v2 can now be scheduled on an existing x86_64_v4 node with available capacity.
Next week we will apply another minor version upgrade gitlab.spack.io.

We're hoping to finalize review and merge Spack PR #38598 soon. This will provide more fine-grained timing data from the build jobs in our GitLab CI pipelines.
We're working on ingesting this timing data already. We're setting up a webhook to receive the data from GitLab, and designing a Postgres database to store it in.

Ryan has a working implementation of a script to crawl recent GitLab CI pipelines & figure out what hashes have been used recently. Unfortunately, the performance is currently too slow to be practical. Next step is to try parallelizing the script to speed it up.

We're investigating the use of CloudFront to speed up downloads from our public buildcaches. We are designing experiments to measure the impact that this change would have.
- For now, we are hopeful that stale/invalid caches will not have much of an impact on the performance of our pipelines.
As part of this effort, we are updating the spack ci module to improve how mirrors are handled. In particular, this will make it easier to specify and use separate push vs. fetch URLs for a given mirror.
We are also investigating the use of OpenID Connect to authenticate our GitLab CI runners with their buildcaches' S3 buckets. This will allow users to retry their jobs after 12 hours, and it will allow us to remove some custom credential rotation infrastructure.

We implemented an alternative to using the file command on Windows. This helps enable buildcaches on Windows with one fewer dependency.
We investigated creating a Spack package for gpg4win to provide gnupg functionality on Windows. A downside of using this package is that its install step requires admin access. We are going to look into adding gpg4win to Spack's bootstrap mirror to sidestep this requirement.
John & Scott are going to brainstorm on what to do about Spack CI's current use of Makefiles and what we can do about that for Windows.

Continue investigating the use of CloudFront to speedup CI and public buildcache usage.
Keep progressing towards the following goals:
- Timing Dashboards
- Gitlab CI runners for Windows
- Automatic buildcache pruning