Skip to content

ci: parallelize Burrito target builds across a matrix to cut release time from 12+ min to <5 #66

Description

@bougyman

Problem

The burrito job in .github/workflows/main.yaml builds all three Burrito
targets (macos_aarch64, linux_x86_64, windows_x86_64 -
mix.exs's releases.lc.burrito.targets) serially, inside one
MIX_ENV=prod mix release lc invocation on a single runner - Burrito builds
each target one after another internally. The whole release pipeline
(validate -> burrito -> container) currently takes 12+ minutes end to
end.

Goal

Split the three target builds into a matrix so they build concurrently,
aiming for the release pipeline to complete in under 5 minutes.

Decisions

  1. Split today's single burrito job into two:
    • burrito-build: a 3-way matrix (target: [macos_aarch64, linux_x86_64, windows_x86_64]), each leg running MIX_ENV=prod BURRITO_TARGET=${{ matrix.target }} mix release lc - the same
      BURRITO_TARGET env var the container job already uses today for its
      own single-target (linux_x86_64) build.
    • burrito-package: gathers all three built binaries and does everything
      that needs them all together - packaging into per-platform tarballs,
      checksums, the app SBOM, gh release create, and marking the release PR
      "autorelease: tagged". Logic unchanged from today's burrito job, just
      moved here.
  2. p7zip (only needed for the Windows target's build, per the existing step
    comment) becomes conditional: if: matrix.target == 'windows_x86_64' -
    skips that install on the other two legs.
  3. "Verify the required target actually built" moves into each matrix leg,
    checking only its own target. linux_x86_64/macos_aarch64 stay
    required (fail their leg hard if missing, matching today's exact target
    list); windows_x86_64 stays non-required. A failed required leg fails
    the matrix (default fail-fast: true), which correctly blocks
    burrito-package (a job whose needs: entry failed/was skipped is itself
    skipped by default - no extra if needed).
  4. Passing binaries between jobs: each matrix leg uploads its one binary via
    actions/upload-artifact@v7 (matching the version this workflow already
    pins for container-sbom) under a per-target artifact name (e.g.
    burrito-${{ matrix.target }}). burrito-package downloads all of them
    with actions/download-artifact@v7's pattern/merge-multiple: true
    inputs into one burrito_out/ directory - verified against the action's
    own README that pattern + merge-multiple: true merges every matching
    artifact into a single directory (available since v4, still present in
    the current v7/v8 releases).
  5. burrito-package needs its own setup-beam + mix deps.get (so
    erlef/mix_sbom's dependency graph is accurate) but not setup-zig or
    p7zip - it never builds a release itself, only assembles what the
    matrix already built.
  6. container's needs: [burrito] becomes needs: [burrito-package] (same
    tag_name output, just sourced from the renamed job).

Expected outcome

Three target builds run concurrently instead of serially. If each leg's
Zig cross-compile + Erlang-runtime-fetch currently takes a meaningful chunk
of the ~12 minutes, parallelizing them should cut several minutes off the
critical path.

Open questions / risks

  • Per-target build time hasn't been measured individually yet - if one
    target (e.g. macos_aarch64, cross-compiling to Darwin) is much slower
    than the others, the matrix helps but the pipeline's total wall-clock
    (validate -> burrito-build -> burrito-package -> container, still
    sequential across those stages) may not land under 5 minutes on the first
    try.
  • Three legs each re-running checkout/setup-beam/setup-zig adds total
    runner-minutes (not wall-clock) versus one job doing it once - a
    reasonable tradeoff for a pipeline that only runs on release-PR merges.

Sequencing

  1. Split burrito into burrito-build (matrix) + burrito-package, wire
    the artifact upload/download between them.
  2. Update container's needs to point at burrito-package.
  3. Verify end-to-end via workflow_dispatch (or a real release-PR merge),
    and confirm actual total wall-clock against the <5 minute target.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions