When the cluster a suite is standing up refuses to start, lib.sh prints eight
identical lines and then a verdict that names none of the eight causes:
-- start attempt 1 failed; retrying on a fresh port
...
-- start attempt 8 failed; retrying on a fresh port
FATAL: no cluster of our own on port 15208 after 8 attempts
(refusing to run against a cluster this suite does not own)
The actual reason is in $PGC_WORKDIR/server.log, which pg_ctl -l has been
writing all along:
FATAL: could not load library ".../pgcolumnar.so": undefined symbol: get_relation_info_hook
LOG: database system is shut down
The workdir is removed when the suite exits, so by the time the operator reads
the verdict the evidence is gone. I had to run the suite in the background and
snapshot the workdir mid-run to see that line at all — six probes to reach a
cause that had been written to disk on the first attempt.
Why #519 does not cover this
#519 added first-fatal reporting, but in the summary path (lib.sh:726-751),
which the start failure never reaches — lib.sh:273 reports and exits first.
And the pattern would not match even if it ran:
grep -nE 'AddressSanitizer|UndefinedBehaviorSanitizer|runtime error:|terminated by signal|PANIC:'
FATAL: could not load library is none of those. A library that will not load
is exactly the kind of first fatal event #518 argued for surfacing, so this is
the same defect in a path the fix did not reach.
The message is also wrong about the cause
(refusing to run against a cluster this suite does not own)
That is one reason a start can fail, and it was not this one. Nothing was
squatting the port; our own postmaster died on startup, eight times, on eight
different ports. The parenthetical asserts a diagnosis the code has not
established, which sends the reader to look for a port collision that is not
there. lib.sh:246-252 already distinguishes the two cases internally — it knows
whether it found another cluster's data directory on the port — but the final
message collapses them.
Fix
On the failure path at lib.sh:273, before exiting:
- print the last ~20 lines of
$PGC_LOGFILE, or the first FATAL: in it;
- broaden the first-fatal pattern to include
FATAL: and could not load library, so the summary path catches this class too;
- only claim the ownership diagnosis when
pgc_cluster_datadir actually returned
another cluster's directory. Otherwise say the postmaster failed to start and
point at the log.
Eight identical retry lines that discard the cause are worse than one line that
names it. The retry is right; the reporting is what needs to change.
Found while diagnosing the sibling issue about stale objects surviving the
matrix's per-major copy.
When the cluster a suite is standing up refuses to start,
lib.shprints eightidentical lines and then a verdict that names none of the eight causes:
The actual reason is in
$PGC_WORKDIR/server.log, whichpg_ctl -lhas beenwriting all along:
The workdir is removed when the suite exits, so by the time the operator reads
the verdict the evidence is gone. I had to run the suite in the background and
snapshot the workdir mid-run to see that line at all — six probes to reach a
cause that had been written to disk on the first attempt.
Why #519 does not cover this
#519 added first-fatal reporting, but in the summary path (
lib.sh:726-751),which the start failure never reaches —
lib.sh:273reports and exits first.And the pattern would not match even if it ran:
grep -nE 'AddressSanitizer|UndefinedBehaviorSanitizer|runtime error:|terminated by signal|PANIC:'FATAL: could not load libraryis none of those. A library that will not loadis exactly the kind of first fatal event #518 argued for surfacing, so this is
the same defect in a path the fix did not reach.
The message is also wrong about the cause
That is one reason a start can fail, and it was not this one. Nothing was
squatting the port; our own postmaster died on startup, eight times, on eight
different ports. The parenthetical asserts a diagnosis the code has not
established, which sends the reader to look for a port collision that is not
there.
lib.sh:246-252already distinguishes the two cases internally — it knowswhether it found another cluster's data directory on the port — but the final
message collapses them.
Fix
On the failure path at
lib.sh:273, before exiting:$PGC_LOGFILE, or the firstFATAL:in it;FATAL:andcould not load library, so the summary path catches this class too;pgc_cluster_datadiractually returnedanother cluster's directory. Otherwise say the postmaster failed to start and
point at the log.
Eight identical retry lines that discard the cause are worse than one line that
names it. The retry is right; the reporting is what needs to change.
Found while diagnosing the sibling issue about stale objects surviving the
matrix's per-major copy.