ci: speed up CI builds and shrink macOS artifact#2
Merged
Conversation
Cold-cache CI runs spent ~3-4 min on macOS arm64 (~1-2 min on Linux)
re-downloading and installing GHC 9.6.7 via haskell-actions/setup, even
though Windows already had a manual ~/.ghcup cache. Mirror that pattern
for Linux/macOS:
- Restore ~/.ghcup keyed on `${{ matrix.name }}-ghc${{ GHC_VERSION }}`.
- On cache hit, skip haskell-actions/setup entirely; a tiny inline step
prepends ~/.ghcup/bin to PATH and runs `cabal update` (the Hackage
index lives outside ~/.ghcup so it isn't covered by the cache).
- Save ~/.ghcup only on cache miss, with no `always()`, so a partial
install from a failed run isn't silently reused.
Also widen the cabal-store restore-keys with a coarser
`cabal-${{ matrix.name }}-` fallback so a GHC version bump can still
seed the new cache from the previous one (cabal re-resolves but reuses
unchanged object files).
The macOS CI artifact was 26 M zipped — almost 2x the Linux ones — because build.yml passed --no-optimize: UPX'd Mach-O on arm64 fails to launch under cabal test even after ad-hoc re-sign, so any test that spawns volca timed out. Tests therefore had to run on the unstripped binary. Decouple "build for tests" from "binary to ship": - build.sh: extract the strip/UPX/codesign block into optimize_volca_binary() and add a --optimize-only flag that runs only that block (no cabal rebuild, no dep checks). Drop the `-1` flag so UPX uses its default compression level — meaningfully smaller than `-1`, still seconds-fast. - build.yml: after the macOS test step (still --no-optimize), call ./build.sh --optimize-only to strip + UPX + re-sign the binary in place, then run "$VOLCA_BIN --version" as a smoke test. If UPX corrupts the Mach-O (it sometimes does on arm64 even after re-sign), fall back to strip-only via cabal re-link from cached objects. - gen-cabal-config.sh: append `-Wl,-dead_strip` and `-Wl,-dead_strip_dylibs` to the darwin link flags. Combined with the already-set split-sections, ld64 prunes unreferenced sections and unused dylib load commands before strip even runs.
…LL'd PR #2's first run confirmed UPX 5.1.1 with --force-macos compresses the arm64 Mach-O fine (104 M -> 48 M, 54 % reduction) but the kernel kills the resulting process at fork with "Killed: 9", even after `codesign --force --sign -` re-signs it (job 73290319277). This is upstream UPX/Mach-O behaviour that we cannot work around from here. Skip UPX on macOS in optimize_volca_binary; just strip + ad-hoc sign. The artifact still shrinks ~14 % from strip alone, and the smoke test in CI now passes deterministically instead of relying on a fallback that didn't actually re-link (the fallback `rm + cabal build` was a no-op — cabal sees the build as up-to-date based on sources, not output binary presence — so the subsequent strip failed with "can't open file"). Linux + Windows UPX is unchanged.
The first warm-cache run on PR #2 failed with "GHC version 9.14.1 (expected 9.6.7)" on linux-amd64 (job 73294752135). Root cause: haskell-actions/setup@v2 makes a GHC version "active" by prepending its version-specific bindir to PATH, not by running `ghcup set ghc`. So ~/.ghcup/bin/ghc (the bare symlink without a version suffix) is left pointing at whatever the runner image had set previously — on ubuntu-latest, the preinstalled GHC 9.14.1. The symlink survives in the cached ~/.ghcup tarball and silently shadows our pinned 9.6.7 when the warm-path step adds ~/.ghcup/bin to PATH. Fix: in the warm path, explicitly `ghcup set ghc <version>` to update the symlink before touching PATH. Also assert the cached GHC dir exists with a clear error message if the cache is stale. Also flip the GHCup save step to `if: always()`. The original cold install runs in a single step that errors out before the save would fire, so there's no risk of caching a partial ~/.ghcup, and we'd otherwise lose a successful install just because a later step (tests, optimize, ...) failed downstream.
The previous warm-cache attempt restored a 202-byte ~/.ghcup tarball and exited with "Cached ~/.ghcup is missing GHC 9.6.7" — confirmed by inspecting job 73296608769's logs. Root cause: haskell-actions/setup@v2 detects the runner image's preinstalled /usr/local/.ghcup, links our "installed" GHC there, and leaves /home/runner/.ghcup essentially empty. Caching ~/.ghcup therefore saved ~200 bytes of metadata, not the GHC tree. Switch the cold path to bootstrap ghcup directly via get-ghcup.haskell.org with BOOTSTRAP_HASKELL_GHC_VERSION pinned — the same pattern the Windows path already uses. This guarantees ~/.ghcup holds the full GHC + cabal install, so the cache actually contains something useful. Bump the cache key to -v2 so prior runs' empty caches don't get restored. Also re-assert the active GHC symlink in the warm path with `ghcup set` (defence in depth — bootstrap auto-sets it, but a snapshot taken mid-state could in principle have a different active version).
PR #2 run 25026862228 hit the same "missing GHC 9.6.7" warm-cache error: the cold install ran and reported success, but only saved 203 bytes — confirmed by inspecting the prior cold run's log (job 73298297469): "downloading ... as file /usr/local/.ghcup/cache/...". The runner image exports GHCUP_INSTALL_BASE_PREFIX=/usr/local in the global environment, so the get-ghcup.haskell.org bootstrap script (which honors that var) installed into /usr/local/.ghcup, NOT ~/.ghcup. Caching ~/.ghcup therefore captured a few stray symlinks and nothing else. Fix: set GHCUP_INSTALL_BASE_PREFIX=$HOME in the cold step before running the bootstrap. Add a post-install sanity check that bails loudly if ~/.ghcup/bin/ghc-<version> isn't present, so a future runner-image change that re-breaks this is immediately visible instead of silently reverting to a 200-byte cache. Bump cache key to -v3 to invalidate the bad caches still on disk.
Cold install on Linux did write GHC into ~/.ghcup, but the cache save still only captured ~200 bytes — confirmed in run 25027426001 job 73301594530 (203 B saved despite a successful 9.6.7 install). Root cause: the ubuntu-latest image pre-creates ~/.ghcup as a façade directory whose subentries are SYMLINKS into the system-wide /usr/local/.ghcup (e.g. `share -> ./ghc/9.14.1/share`). When ghcup install runs against that directory, it writes through the symlinks, which means the actual GHC tree ends up at /usr/local/.ghcup/. Then tar (used by actions/cache) archives the symlink shells in ~/.ghcup without dereferencing them, so the cache holds only the symlinks — which on a fresh runner point at /usr/local/.ghcup paths that don't have our pinned 9.6.7. Fix: `rm -rf ~/.ghcup` before the bootstrap so the install lands in a real, plain directory tree. rm doesn't follow symlinks, so the runner's /usr/local/.ghcup is untouched (and irrelevant because $PATH in subsequent steps points only at ~/.ghcup/bin). Also log `du -sh ~/.ghcup` after install so a future regression here shows up as a visible-in-the-log size, not a silent 200-byte cache. Bump cache key to -v4.
PR #2 run 25028781359 hit "version 9.6.7 of the tool ghc is not installed" in the warm step despite a healthy 550 MB cache restore. Root cause: the runner image exports GHCUP_INSTALL_BASE_PREFIX=/usr/local globally, so `ghcup set ghc 9.6.7` was looking under /usr/local/.ghcup/ghc/9.6.7 — which doesn't exist on a fresh runner — instead of ~/.ghcup/ghc/9.6.7 where the cache restore put it. Same fix as the cold step: export GHCUP_INSTALL_BASE_PREFIX=$HOME before invoking ghcup. The downstream cabal/ghc binaries find their own installation via argv[0] resolution, so they don't need the override; only ghcup itself does.
Warm v4 run (25029012799) restored a 129 MB cabal cache successfully but cabal still rebuilt every dep — confirmed in linux-arm64 job 73306452714: "Building blaze-builder-0.4.4.1", "Building bsb-http-chunked", etc., taking ~9 min instead of expected ~1 min. Root cause: the ubuntu-latest runner image pre-creates ~/.config/cabal/config, so newer cabal-install versions log "Both /home/runner/.cabal and /home/runner/.config/cabal/config exist - ignoring the former" and use XDG paths — including a different store directory under ~/.local/state/cabal/store. Our cached ~/.cabal/store sat unused. Fix: set CABAL_DIR=~/.cabal in $GITHUB_ENV for Linux/macOS so cabal honors the legacy paths regardless of cabal version or runner-image state. (The warning above explicitly says CABAL_DIR overrides this auto-detection.) Bump cabal cache key to -v2: existing caches' dist-newstyle was populated while cabal used XDG paths, so its plan.json may reference locations cabal can no longer resolve — cleanest to force one cold rebuild with the consistent layout, then warm runs will hit cleanly.
ccomb
added a commit
that referenced
this pull request
May 31, 2026
…ngine 0.7.0 (#108) Two fixes to align pyvolca with the VoLCA engine 0.7.0 wire format. Both are data-correctness bugs where the client silently lost information. ## Bug #1 — substitutions silently ignored `Client.get_impacts(substitutions=…)` (and `get_supply_chain`, `get_inventory`, `get_impacts_batch`) was broken against 0.7.0. The client built the body with the **raw Haskell field names**, but the engine's `stripLowerPrefix` codec drops the lowercase type-prefix before parsing, so it reads the *stripped* names: | raw Haskell field | wire name read by engine | |-------------------|--------------------------| | `srSubstitutions` | `substitutions` | | `subFrom` | `from` | | `subTo` | `to` | | `subConsumer` | `consumer` | pyvolca sent the pre-strip names, so the engine recognised zero fields and dropped the substitution without error — the score came back identical to a no-substitution call. Correct shape (proven via raw HTTP): ```json { "substitutions": [ { "from": "...", "to": "...", "consumer": "..." } ] } ``` `Substitution.to_wire()` and `_substitution_body()` now emit the stripped keys. ## Bug #2 — SupplyChainEdge dropped the cross-DB columns The engine's `SupplyChainEdge` carries `edgeFromDb` / `edgeToDb` (each endpoint's database name) next to the process ids, but `from_json` read only `edgeFrom` / `edgeTo` / `edgeAmount` and discarded the two DB columns. They're required to route edges across databases — the same process id can exist in more than one loaded DB. Added `from_db` / `to_db` to the dataclass and parse them. ## Misc - Patch bump to 0.5.1. - Regenerated the README API reference (also absorbs an earlier `Method`-docstring drift). ## Verification `uv run --extra dev pytest` → 137 passed, 7 skipped. New unit tests cover the substitution wire shape (typed + dict input) and the edge's DB endpoints.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two improvements to the build matrix introduced in #1:
~/.ghcupon Linux/macOS so warm runs skip the 3–4 min GHC reinstall, and forceCABAL_DIR=~/.cabalso cabal actually uses our cached package store (newer cabal-install otherwise prefers$XDG_STATE_HOMEand the cached store sits unused). Various smaller fixes along the way: rm the runner-image's symlinked~/.ghcupshim before bootstrap, overrideGHCUP_INSTALL_BASE_PREFIX=$HOMEin both cold and warm paths, widen the cabal cacherestore-keys../build.sh --optimize-onlyto strip + re-sign. Added-Wl,-dead_strip -Wl,-dead_strip_dylibsto the darwin link flags.Final timings (run 25030160840, fully-warm)
Cold-cache (run 25029725786, after
-v2invalidation): 5–6 min on Linux/macOS, 8m21s on Windows.Artifact size (macos-arm64)
UPX on macOS arm64 is broken upstream — the
--force-macoscompressor produces a binary that is SIGKILL'd at fork even after ad-hoc re-signing. Strip-only is the ceiling for now (saved as project memory for future sessions).Test plan
--versionsmoke test passes after--optimize-only~/.cabal/storeand gets reused (compile log shows no "Building xyz" for deps)