CI: 2023 08 04

Attendees

This week we updated gitlab.spack.io to use S3 rather than minio to store job logs & artifacts.
Our staging cluster has been similarly upgraded to use ElastiCache rather than redis. We plan to make this switch in production soon.

Work continues on our effort to record more specific timing data from spack install (Spack PR #38598)
We're hoping to publish some proof-of-concept dashboards using this new data soon!

We are investigating why our new scripts in spack-infra PR #415 are erroneously returning specs that should not actually be parsed yet.

We plan to deploy a fix for copy-only pipelines for the ml-darwin stack early next week (Spack PR #38514)
We spent some time this week working with UO to diagnose issues with our Mac runners where they would fail to import their GPG keys
We investigated a spike in no-binaries-for-spec errors that occurred this week. We tracked this down to a failing credential rotation script getting retried too often, causing these credentials to get rotated too frequently.
We are also investigating using CloudFront rather than raw S3 for downloading from our buildcaches in CI. Our plan is to use our new timing data capabilities to determine how big of an improvement this switch would make before proceeding further with the implementation.

This is getting closer! We are currently porting packages to Windows (file, gnupg) that are needed to enable use of buildcaches for this OS.

Wrap up timing statistics work and generate some first-draft dashboards
Use timing data to estimate the impact CloudFront would have on buildcache downloads from our CI pipelines.
Keep striving towards the goal of Windows Gitlab CI runners
- Create a bootstrap buildcache for Windows with required dependencies: gpg, file, etc.
Improve error messages for the "access denied" case when attempting to download dependencies from the buildcache. Ideally we should be able to categorize these errors as something more specific than no-binaries-for-spec.