Skip to content
Dan LaManna edited this page Sep 16, 2023 · 2 revisions

Attendees

  • Aashish Chaudhary
  • Alec Scott
  • Jacob Nesbitt
  • John Parent
  • Massimiliano Culpo
  • Dan LaManna
  • Mike VanDenburgh
  • Ryan Krattiger
  • Scott Wittenburg
  • Todd Gamblin
  • Zack Galbreath

CI updates and improvements

  • We are gradually rolling out OpenID Connect authentication for our GitLab CI runners. Once this process is complete we will be able to remove our custom credential rotation logic, and users will be able to retry failed jobs after the current 12 hour expiration window.
  • We are trying to better understand and classify no-binary-for-spec errors.
  • In pursuit of this goal, we opened Spack PR #40045 to mark a job as failed when it encounters an error while pushing to the binary mirror. The current behavior is for the subsequent job to fail when it attempts to pull this nonexistent job from the mirror.
  • We are overhauling our binary mirrors are used within the spack ci module PR #39939. We are also taking this opportunity to deprecate and/or removing unused ci mirror features (like support for artifacts buildcache).

Metrics & Dashboarding

  • Our new analytics database is up and collecting data from GitLab CI jobs.
  • Using the analytics database, we published some initial results comparing S3 vs. CloudFront download speeds in our CI pipelines. We plan to continue this investigation, but so far it looks like it's worth pursuing CloudFront for increased pipeline efficiency.
  • We created a new "Job by Runner" view in Metabase that makes it easier to search through GitLab CI jobs.
  • Our investigation into Sentry continues. This week we examined GitLab's integration with Sentry. It looks useful for GitLab error/incident tracking.

Priorities

  • Help alleviate any pain points in the release process prior to SC.
  • Publish an overview page describing our CI system.
  • Create a presentation describing our preliminary results comparsing S3 vs. CloudFront download speeds for our GitLab CI pipelines.
Clone this wiki locally