Follow-up to #351 and #352, raised because jdatcmd noted on #381 that this keeps
recurring and deserves its own issue.
#352 did its job. The step no longer hangs forever, it fails at the bound. What is left
is the cause of the stall, and the standing attribution for it is wrong.
The failures are a timeout wall, not slow fetches
Every Install PostgreSQL N ... step across the last 80 runs, counting all attempts:
| conclusion |
n |
min |
p50 |
p90 |
max |
| success |
352 |
13s |
25s |
36s |
137s |
| failure |
6 |
606s |
608s |
608s |
608s |
The distribution is bimodal with nothing in between. The slowest success is 137s, which
is 4.4x under the 600s bound. No success sits anywhere near the cap.
That rules out a margin problem. The step either finishes in about half a minute or it
never finishes. Exit 124 is the timeout 600 on the apt-get install line firing.
It is not apt.postgresql.org
From the failing job in run 30936410271, the three PGDG packages all arrived in under
0.2 seconds:
17:58:45.685 Get:12 https://apt.postgresql.org/... postgresql-client-18 [2095 kB]
17:58:45.747 Get:13 https://apt.postgresql.org/... postgresql-18 [7555 kB]
17:58:45.800 Get:14 https://apt.postgresql.org/... postgresql-server-dev-18 [1397 kB]
The stall is on the Ubuntu mirror, fetching the LLVM toolchain:
17:58:46.182 Get:2 http://azure.archive.ubuntu.com/... libclang-cpp19 [14.3 MB]
18:05:37.672 Get:3 http://azure.archive.ubuntu.com/... libclang-common-19-dev [743 kB]
18:05:59.563 Get:4 http://azure.archive.ubuntu.com/... llvm-19-linker-tools [1314 kB]
18:06:37.852 Get:5 http://azure.archive.ubuntu.com/... libclang1-19 [8272 kB]
18:08:45.299 ##[error]Process completed with exit code 124.
apt queues per host, so those gaps are that host's throughput. Roughly 24 MB moved in
the 599 seconds available, on the order of 30 kB/s.
Where the 90 MB comes from
The step already passes --no-install-recommends and asks only for
postgresql-N postgresql-server-dev-N liblz4-dev libzstd-dev zlib1g-dev. apt still
resolves 13 packages and reports Need to get 102 MB.
The log separates the two categories, which settles where they come from:
The following additional packages will be installed:
clang-19 libclang-common-19-dev libclang-cpp19 libclang1-19 liburing2
llvm-19 llvm-19-dev llvm-19-linker-tools llvm-19-runtime llvm-19-tools
postgresql-client-18
Recommended packages:
libclang-rt-19-dev postgresql-18-jit
The LLVM packages are listed as additional, not recommended, so they are hard
dependencies of postgresql-server-dev-N and --no-install-recommends cannot drop
them. The three PostgreSQL debs are 11 MB of the 102 MB. The other 91 MB is a JIT
toolchain, pulled from the mirror that is throttling, to build an extension whose CI
never uses it.
Two things that will not fix it
Retries. Acquire::Retries=5 only fires on a failed transfer. A mirror trickling at
30 kB/s is succeeding, slowly, so retries never trigger.
Rerunning. In run 30852966199 the job failed at 21:03:14, was rerun, and the rerun
failed the same way at 21:14:28. Both attempts are in the table above.
Occurrences
Six in the last 80 runs, four of them on 2026-08-04, across both PG 17 and PG 18 and
across build and suites:
| started |
job |
run |
| 2026-08-03T21:03:14Z |
build (PG 17, x86_64) |
30852966199 |
| 2026-08-03T21:14:28Z |
build (PG 17, x86_64), rerun |
30852966199 |
| 2026-08-04T02:41:32Z |
suites (PG 18) |
30872468390 |
| 2026-08-04T03:50:45Z |
build (PG 17, x86_64) |
30875881302 |
| 2026-08-04T14:18:22Z |
build (PG 18, x86_64) |
30918243160 |
| 2026-08-04T17:58:36Z |
build (PG 18, x86_64) |
30936410271 |
The last one is #381, the 1.0-alpha benchmark correction.
Options
I have not measured these, so this is the part to argue about rather than take.
- Cache the debs. Key on PG major plus runner image. Removes the download on a hit.
Largest win, and it also cuts the median 25s.
- Do not install the toolchain. Needs a way to satisfy
postgresql-server-dev-N
without its LLVM dependency, so it is the least straightforward option despite being
the one that addresses the actual waste.
- Raise the bound. Cheapest, and the worst of the three. At 30 kB/s the fetch needs
roughly 50 minutes, so this trades a red X for a job that ties up a runner.
Happy to take this, and to measure option 1 before proposing it.
Follow-up to #351 and #352, raised because jdatcmd noted on #381 that this keeps
recurring and deserves its own issue.
#352 did its job. The step no longer hangs forever, it fails at the bound. What is left
is the cause of the stall, and the standing attribution for it is wrong.
The failures are a timeout wall, not slow fetches
Every
Install PostgreSQL N ...step across the last 80 runs, counting all attempts:The distribution is bimodal with nothing in between. The slowest success is 137s, which
is 4.4x under the 600s bound. No success sits anywhere near the cap.
That rules out a margin problem. The step either finishes in about half a minute or it
never finishes. Exit 124 is the
timeout 600on theapt-get installline firing.It is not apt.postgresql.org
From the failing job in run
30936410271, the three PGDG packages all arrived in under0.2 seconds:
The stall is on the Ubuntu mirror, fetching the LLVM toolchain:
apt queues per host, so those gaps are that host's throughput. Roughly 24 MB moved in
the 599 seconds available, on the order of 30 kB/s.
Where the 90 MB comes from
The step already passes
--no-install-recommendsand asks only forpostgresql-N postgresql-server-dev-N liblz4-dev libzstd-dev zlib1g-dev. apt stillresolves 13 packages and reports
Need to get 102 MB.The log separates the two categories, which settles where they come from:
The LLVM packages are listed as additional, not recommended, so they are hard
dependencies of
postgresql-server-dev-Nand--no-install-recommendscannot dropthem. The three PostgreSQL debs are 11 MB of the 102 MB. The other 91 MB is a JIT
toolchain, pulled from the mirror that is throttling, to build an extension whose CI
never uses it.
Two things that will not fix it
Retries.
Acquire::Retries=5only fires on a failed transfer. A mirror trickling at30 kB/s is succeeding, slowly, so retries never trigger.
Rerunning. In run
30852966199the job failed at 21:03:14, was rerun, and the rerunfailed the same way at 21:14:28. Both attempts are in the table above.
Occurrences
Six in the last 80 runs, four of them on 2026-08-04, across both PG 17 and PG 18 and
across
buildandsuites:The last one is #381, the 1.0-alpha benchmark correction.
Options
I have not measured these, so this is the part to argue about rather than take.
Largest win, and it also cuts the median 25s.
postgresql-server-dev-Nwithout its LLVM dependency, so it is the least straightforward option despite being
the one that addresses the actual waste.
roughly 50 minutes, so this trades a red X for a job that ties up a runner.
Happy to take this, and to measure option 1 before proposing it.