Stop running all CI legs on every checkin

[MichaelSimons]

We should be good stewards of the amount of compute consumed by our CI. Currently each checkin is consuming 8+ hours of compute. We don't get a large amount of value from running each distro leg on every commit. We should consider creating a new CI pipeline that is run on a less frequent schedule - e.g. once per weekday. The CI run on every commit should contain the legs that have the highest ROI - e.g. online/offline, amd64/arm32, stage 1/2

[ellahathaway]

Optimization of CI Pipeline Strategies

The Problem

Currently, each check-in is consuming 8+ hours of computational resources. To optimize our processes, it's necessary to explore options that would allow the CI pipeline to operates on a less frequent schedule, perhaps once per weekday. It's proposed that the CI runs triggered by every commit should selectively include the distribution legs that yield the highest ROI. These could involve factors such as online/offline status, architecture (amd64/arm32), and the stage of the process (stage 1/2).

End Goal

Our primary objective is to streamline our computational resource utilization while keeping a positive user experience at the forefront.

Solutions

Option 1: Multiple Pipelines

One approach is to establish two distinct CI pipelines.

Heavy Pipeline:

• Schedule: Once per weekday (12 am)
• Tasks: Computationally intensive processes
• Distribution Legs: All legs
• Pros:
  • Utilizes the full potential of computational resources during the dedicated run.
  • Comprehensive testing helps identify deep-seated issues that might not surface in 'lite' runs.

Lite Pipeline:

• Schedule: More frequent (every check-in)
• Tasks: Less computation-intensive tasks
• Distribution Legs: Highest ROI legs (online/offline, amd/arm32, stage 1/2)
• Pros:
  • Improved user experience due to clearer run status and distinct purposes of the pipelines.
  • Faster feedback loop as 'lite' runs occur more frequently.

Cons:

• More pipelines to monitor, potentially causing user confusion.
• Users will need to track yet another pipeline alongside the existing ones.
• Managing two separate pipelines might increase administrative overhead.

Option 2: Single Pipeline

Alternatively, the current single pipeline can be retained, but its runs can be categorized by schedule.

Full Runs:

• Schedule: Once per weekday (12 am)
• Tasks: Comprehensive testing, resource-intensive computations
• Distribution Legs: All legs
• Pros:
  • Detailed testing helps ensure the stability of the entire system.
  • Dedicated 'full' runs provide a holistic view of the system's health.

Lite Runs:

• Schedule: More frequent (every check-in)
• Tasks: Less computation-intensive tasks
• Distribution Legs: Highest ROI legs (online/offline, amd/arm32, stage 1/2)
• Pros:
  • Centralized monitoring in a single pipeline simplifies user experience.
  • Users do not need to navigate through multiple pipelines, reducing potential confusion.

Cons:

• Determining whether a run was 'full' or 'lite' could be challenging, potentially leading to confusion.
• The current pipeline's interface might become cluttered due to the inclusion of additional information.
• Managing different run types within the same pipeline might require more sophisticated tracking mechanisms.

In either approach, the decision hinges on striking a balance between computational efficiency and user-friendliness. I am personally in favor of Option 1 (multiple pipelines), but I'd like to hear other opinions on the options & if anyone has thoughts on additional options not mentioned above.