Commit 1493ca6
ci: Custom GitHub Runners for Nix Builds (#1745)
* chores(ci): opted-out nix-fast-build
* chores(ci): set up nix-github-actions
* chores(ci): comment out vestigal testinfra-ami-build
* chores(ci): use custom github runners
* chores(ci): use nix-eval-jobs and sets AWS creds to /etc/nix/aws
* feat(ci): do not build if already cached
* chore: improve reproducibility of postgresql builds
Do not create postgresql package that depends on the runtime evaluation context as
that harms reproducibility.
* chore: remove nix-github-actions dependency
* feat(ci): split nix build workflow into separate extensions and checks jobs
Split the monolithic nix-build job into two workflows:
one for building PostgreSQL extensions and another for checks.
Building extensions can be resource-intensive and time-consuming,
so isolating them allows for better resource allocation and parallelism.
Once they are built, the checks job can run tests and validations
on the already built extensions.
* feat(ci): extract nix build setup into reusable action and split builds by architecture
Extract AWS credential setup and nix build steps into a composite action
to reduce duplication. Split extension builds into separate jobs per
architecture (aarch64-linux, aarch64-darwin, x86_64-linux) and update
matrix generation to group packages by system.
* fix(ci): do not hide cached builds
* Revert "fix(ci): do not hide cached builds"
This reverts commit 9d41b1e.
* chore: Temporarily disable x86_64-linux builds
We don't have a self hosted runner for x86_64-linux at the moment
* fix: sort packages and filter out cached ones
* fix: do not skip checks-matrix if dependencies are skipped
* fix: do not return debug fields in GitHub Actions matrix output
* debugging
* fix(ci): use !cancelled() instead of always() for dependent job conditions
Replace always() with !cancelled() to prevent jobs from running when the workflow
is cancelled. Add explicit success/skip conditions for dependent jobs to ensure
proper workflow execution flow.
* fix(ci): stop chaining aws roles
We couldn't set duration to 18000 seconds when chaining roles.
* ci: run nixos test on aarch64-linux
* fix(ci): disable eval-cache and accept-flake-config
* ci: split checks build jobs by system architecture
Refactor GitHub Actions workflow to run build checks in parallel across different
architectures (aarch64-linux, aarch64-darwin) with separate job matrices.
* fix(ci): use correct architecture name in aarch64-linux builds
* fix(ci): do not try to build already cached checks
* fix(ci): simplify GitHub Actions workflow for Nix builds
Create a single nix-eval job to determine packages to build, removing
redundant extension and check matrices.
* Revert "chores(ci): comment out vestigal testinfra-ami-build"
This reverts commit e2db368.
* feat(ci): eval on blacksmith-32vcpu-ubuntu-2404
* feat: add ephemeral Nix install action for GitHub runners
* refactor(ci): extract nix eval into reusable workflow
* feat: enable x86_64-linux builds in CI
* feat: add PostgreSQL version to GitHub Actions job names
When building a postgres extension, the build matrix may include
multiple time the same extension for different PostgreSQL versions.
This change makes it easier to identify which job corresponds to which PostgreSQL
version in the workflow runs.
* fix: disable treefmt flake check
treefmt is already included in the pre-commit hooks check.
* feat: run actionlint on new GitHub Actions workflows
* chore: improve github matrix script type annotations
* feat: optimize CI runner selection based on package size
Dynamically assign larger runners (32vcpu) for Rust and PostGIS extensions
while using smaller runners (8vcpu) for standard packages.
* chore: fix package meta maintainers format
* chore: create a nix package for generating GitHub Actions matrix
Add pytest tests for the package
Add nix-eval-jobs in path for the package
* fix: configure runner according to the matrix job
The matrix job returns the type of runner, so we can configure the nix
installation step accordingly.
* Update nix-eval-jobs
Our changes were merged upstream, so we can now track the original
repository again.
* refactor(ci): standardize nix installation and disable cache push by default
- Replace DeterminateSystems/nix-installer-action with custom nix-install-ephemeral action across all workflows
- Change default push-to-cache from 'true' to 'false' to prevent unnecessary nix/aws configurations
- Explicitly enable push-to-cache only for nix-build and nix-eval workflows where caching is beneficial
* feat: use big-parallel to identify large packages
* fix(ci): ensure x86_64-linux build is considered in testinfra and test workflows
* fix: nix devShell inclusion condition
* fix(ci): eval should fail if github-matrix run fails
* fix(ci): remove redundant build psql bundle step
* fix: reduce ARM runner size from 8vcpu to 4vcpu for ephemeral builds
We might not need the full 8vcpu for aarch64-linux builds, so this
change reduces the runner size to 4vcpu to wait less for available
blacksmith runners.
* feat: do not return empty matrices if no package has to be built
* feat: fail pipeline if nix evaluation fails
* Update nix/ext/pgvector.nix
Co-authored-by: samrose <samuel@supabase.io>
* fix: add skip job only for systems that don't have any job
* fix(github-matrix): handle evaluation errors without deadlock
Fix github-matrix that would hang when nix-eval-jobs encountered errors due to subprocess pipe deadlock - stderr buffer would fill while reading stdout.
This change ensure that evaluation errors are visible and the workflow fails properly while still showing which packages succeeded.
* feat(github-matrix): integrate github-action-utils for better error visibility
Integrates github-action-utils library to improve error and warning
visibility in GitHub Actions UI through workflow command annotations.
* feat(github-matrix): group evaluation errors by message
Refactor error handling to collect and group evaluation errors similar to warnings. Errors with the same message are now displayed together with a list of affected attributes.
* fix(github-matrix): improve multiline error display in GitHub Actions
Extract core error messages and format them better for GitHub Actions
annotations.
* fix(ci): skip run-testinfra and run-tests when nix-eval fails
Add nix-eval to needs dependencies and check its result in conditional expressions to prevent downstream test jobs from running when evaluation fails.
* chore(github-matrix): update message when there are no build for a system
* fix(github-matrix): backward compatibility for Result access
We are running an older version of the 'result' library that uses
'_value' instead of 'ok_value' to access the successful result of a
computation.
* refactor: migrate from packages to legacyPackages for PostgreSQL extensions
To be able to build extensions versions packages separately in CI, we
need to expose them in a nested structure. It is currently not possible
to do so with the flattened packages structure, as the individual
extension packages are not directly accessible.
In this change, we replace the flattened package structure with nested
legacyPackages to improve discoverability of individual extension
packages.
* refactor(nix): remove "-all" suffix from extension package names
Simplify extension package naming by removing the redundant "-all" suffix that was appended to pname attributes.
* refactor(ci): split nix build jobs into separate packages and checks workflows
To make sure we only build what is necessary, we start building packages first,
then run checks once all packages are built successfully.
* feat: use 8 vCPU runner for aarch64 builds
Use the same 8 vCPU runner for aarch64 builds as used for x86_64 builds to improve build performance.
* feat(ci): test blacksmith stickydisk for eval
---------
Co-authored-by: Jean-François Roche <jfroche@pyxel.be>
Co-authored-by: samrose <samuel@supabase.io>1 parent a431c10 commit 1493ca6
File tree
60 files changed
+1281
-184
lines changed- .github
- actions/nix-install-self-hosted
- workflows
- nix
- docs
- ext
- pg_cron
- pg_graphql
- pg_jsonschema
- pgmq
- pgroonga
- plv8
- tests
- wrappers
- packages
- github-matrix
- tests
- postgresql
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
60 files changed
+1281
-184
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
18 | | - | |
19 | | - | |
20 | | - | |
21 | 17 | | |
22 | | - | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
23 | 139 | | |
24 | 140 | | |
25 | | - | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
| 141 | + | |
| 142 | + | |
35 | 143 | | |
36 | 144 | | |
37 | | - | |
38 | | - | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
39 | 150 | | |
40 | | - | |
| 151 | + | |
41 | 152 | | |
42 | 153 | | |
43 | 154 | | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
75 | 180 | | |
76 | | - | |
77 | | - | |
78 | | - | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
79 | 187 | | |
80 | 188 | | |
81 | | - | |
82 | | - | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
83 | 199 | | |
84 | 200 | | |
85 | 201 | | |
86 | 202 | | |
87 | 203 | | |
88 | | - | |
89 | | - | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
90 | 214 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
0 commit comments