Skip to content

A cluster that will not start reports eight identical retries and discards the reason, which #519's first-fatal reporting does not reach #537

Description

@ChronicallyJD

When the cluster a suite is standing up refuses to start, lib.sh prints eight
identical lines and then a verdict that names none of the eight causes:

-- start attempt 1 failed; retrying on a fresh port
...
-- start attempt 8 failed; retrying on a fresh port
FATAL: no cluster of our own on port 15208 after 8 attempts
       (refusing to run against a cluster this suite does not own)

The actual reason is in $PGC_WORKDIR/server.log, which pg_ctl -l has been
writing all along:

FATAL:  could not load library ".../pgcolumnar.so": undefined symbol: get_relation_info_hook
LOG:  database system is shut down

The workdir is removed when the suite exits, so by the time the operator reads
the verdict the evidence is gone. I had to run the suite in the background and
snapshot the workdir mid-run to see that line at all — six probes to reach a
cause that had been written to disk on the first attempt.

Why #519 does not cover this

#519 added first-fatal reporting, but in the summary path (lib.sh:726-751),
which the start failure never reaches — lib.sh:273 reports and exits first.

And the pattern would not match even if it ran:

grep -nE 'AddressSanitizer|UndefinedBehaviorSanitizer|runtime error:|terminated by signal|PANIC:'

FATAL: could not load library is none of those. A library that will not load
is exactly the kind of first fatal event #518 argued for surfacing, so this is
the same defect in a path the fix did not reach.

The message is also wrong about the cause

(refusing to run against a cluster this suite does not own)

That is one reason a start can fail, and it was not this one. Nothing was
squatting the port; our own postmaster died on startup, eight times, on eight
different ports. The parenthetical asserts a diagnosis the code has not
established, which sends the reader to look for a port collision that is not
there. lib.sh:246-252 already distinguishes the two cases internally — it knows
whether it found another cluster's data directory on the port — but the final
message collapses them.

Fix

On the failure path at lib.sh:273, before exiting:

  • print the last ~20 lines of $PGC_LOGFILE, or the first FATAL: in it;
  • broaden the first-fatal pattern to include FATAL: and could not load library, so the summary path catches this class too;
  • only claim the ownership diagnosis when pgc_cluster_datadir actually returned
    another cluster's directory. Otherwise say the postmaster failed to start and
    point at the log.

Eight identical retry lines that discard the cause are worse than one line that
names it. The retry is right; the reporting is what needs to change.

Found while diagnosing the sibling issue about stale objects surviving the
matrix's per-major copy.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions