Consider grouping CodSpeed benchmark packages to reduce CI overhead

[mattsu2020]

Proposal: group benchmark packages to reduce duplicated CI overhead

The current Benchmarks workflow runs CodSpeed benchmarks with a matrix over both type and package:

type: [simulation, memory]
package:
  - uu_base64
  - uu_cksum
  - ...

This results in:

2 benchmark modes × 33 packages = 66 jobs

Each job runs on a separate GitHub-hosted runner, so the following setup work is repeated for every package:

checkout
Rust toolchain setup
rust-cache restore/save
sccache setup
locale installation
cargo-codspeed installation
compilation of shared workspace dependencies

Although rust-cache and sccache help, the current package-level matrix still causes significant duplicated setup and build overhead because each matrix entry runs in a separate VM.

Suggested improvement

Instead of running one job per package, we could group multiple packages into a smaller number of package groups.

For example:

strategy:
  fail-fast: false
  max-parallel: 4
  matrix:
    type: [simulation, memory]
    group:
      - text
      - file
      - numeric
      - misc

Then each job can run multiple packages sequentially within the same runner:

case "${{ matrix.group }}" in
  text)
    packages=(uu_cat uu_cut uu_fold uu_join uu_nl uu_sort uu_uniq uu_wc)
    ;;
  file)
    packages=(uu_cp uu_dd uu_df uu_du uu_ls uu_mv uu_rm)
    ;;
  numeric)
    packages=(uu_cksum uu_numfmt uu_seq uu_factor)
    ;;
  misc)
    packages=(uu_base64 uu_echo uu_expand uu_hostname uu_shuf uu_split uu_tee uu_timeout uu_tsort uu_unexpand uu_date uu_csplit uu_true uu_false)
    ;;
esac

The build/run steps could then loop over the packages in the selected group:

for package in "${packages[@]}"; do
  echo "Building ${{ matrix.type }} benchmarks for ${package}"
  cargo codspeed build -m "${{ matrix.type }}" -p "${package}"
done

for package in "${packages[@]}"; do
  echo "Running ${{ matrix.type }} benchmarks for ${package}"
  cargo codspeed run -m "${{ matrix.type }}" -p "${package}"
done

[oech3]

Does this not affect to benchmark's result?