CI: 2023 06 16
Zack Galbreath edited this page Jun 16, 2023
·
1 revision
- Aashish Chaudhary
- Alec Scott
- Dan LaManna
- Jacob Nesbitt
- John Parent
- Massimiliano Culpo
- Mike VanDenburgh
- Ryan Krattiger
- Scott Wittenburg
- Tamara Grimmett
- Zack Galbreath
- We practiced upgrading GitLab in our staging cluster and the whole process went pretty smoothly.
- We performed additional experiments on our upgraded staging GitLab and noticed that the auto-cancel redundant pipelines feature was sometimes not working. This feature is important for our workflow because two pipelines running simultaneously on the same branch can cause race conditions as they both read and write from their binary mirror. For this reason, we postponed the scheduled upgrade of our production GitLab instance.
- We repeated this experiment on our production GitLab and it demonstrated the same behavior. Therefore, our plan is to proceed with the upgrade early next week. We will also continue to investigate the problem with automatic pipeline cancellation with the goal of being able to detect and work around this issue.
- Access to prometheus.spack.io has been restored via GitHub OAuth. Due to an upstream bug, this is currently restricted to an explicit AllowList of users.
- We began investigating the Grafana dashboards that are provided by GitLab.
- We are working on gathering more fine-grained timing statistics from our GitLab CI build jobs. Our goal is to record separate timing information for download, unarchive, relocate, build from source, etc.
- We are working on using S3 bucket policies to restrict access to non-public pcluster binaries.
- Once that's done we can delete the custom buildcache pruning steps from our pcluster pipeline definitions.
- We've built an image with all the required development tools preinstalled.
- We continue to iron out issues and make progress towards the goal of Windows cloud runners.
- Complete the upgrade of gitlab.spack.io
- Pursue more fine-grained timing data for GitLab CI build jobs
- Deploy GitLab CI runners for Windows
- Consider approaches to work around the race conditions caused by the unreliability of pipeline auto-cancellation:
- Separate write destination per push
- GitLab resource groups to prevent jobs running simultaneously (per hash)
- cache.spack.io improvements:
- Show packages for snapshot tags
- On the spack-specific view, display instructions on how to register that stack's mirror