CI: 2023 07 07

Attendees

• Aashish Chaudhary
• Dan LaManna
• John Parent
• Massimiliano Culpo
• Ryan Krattiger
• Scott Wittenburg
• Todd Gamblin
• Zack Galbreath

Timing Statistics

• Ryan is making good progress towards our goal of capturing more fine-grained timing statistics. We're hoping that PR #38598 will be merged soon.
• Scott pointed out that some operations are not currently captured by timers (such as waiting for locks). We're hoping to record this "uncategorized" time spent during spack install so we can know how much time is not captured by our more fine-grained timers.
• Once we have better timing data, we will start focusing on creating new dashboards using this data. Some ideas include:
  • a pie chart showing how much time is spent on the different build phases:
    • fetch
    • build from source
    • install from buildcache
    • cache creation (spack buildcache create)
    • signing
    • upload to buildcache
  • top N packages that take the longest to build from source
  • top N packages that take the longest to install their dependencies from buildcache
  • average time spent downloading on different instance types (checking for inefficient instance types)

Other topics

• Ryan plans to manually run his script to prune the develop buildcache today, and every two weeks until this process is automated.
• cache.spack.io should soon include specs from our weekly develop snapshot buildcaches.
• Scott is investigating strategies to run the protected-publish job more reliably. This would help solve the problem of a spec's dependencies sometimes not being found in the buildcache.

Priorities

• Upgrade EKS and karpenter
• Finish timing data PR and start working on subsequent dashboards
• Update the sync script to merge topics branches against their base branch instead of assuming that it is always develop (necessary for release branch PRs).
• Migrate remaining gp2 volumes to gp3
• Update gitlab.spack.io to use S3 and ElasticCache rather than minio and redis.

Future work

• For relocation, stop spidering so much and instead publish & use fix-up tables from our buildcaches.
• Perform a study of how well different packages respond to building with higher -j numbers.