fix(multigres): stop base config data_directory from overriding the pooler data dir - #2344
Merged
Merged
Conversation
…r data dir The Multigres pooler failed to bootstrap: the transient PostgreSQL that pgctld starts during InitDataDir died at startup with FATAL: data directory "/var/lib/postgresql/data" has invalid permissions so it never became ready, the first-backup bootstrap retried forever, and the pooler socket never appeared. pgctld runs initdb at <pooler-dir>/pg_data and pins the data dir via -D, but the operator appends `include = '/etc/postgresql/postgresql.conf'` onto the generated config (POSTGRES_INITDB_EXTRA_CONF). In the layered image that base config carries an active data_directory = '/var/lib/postgresql/data', and since the include is last it wins, redirecting PostgreSQL off the freshly initdb'd data dir onto the base image's empty, wrong-permission directory. The pre-layered image shipped data_directory commented, which is why it worked. Comment it out in the layered image too (alongside the existing wal-g include removal), and correct the now-wrong comment claiming the base config is never loaded. postgresql.conf.j2 is left untouched, so the base supabase image and the VM/ansible deploys (which run `postgres -D /etc/postgresql` and rely on data_directory) are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mkindahl
marked this pull request as draft
August 5, 2026 15:39
mkindahl
marked this pull request as ready for review
August 5, 2026 16:58
soedirgo
approved these changes
Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Multigres pooler never bootstraps. pgctld's transient PostgreSQL (started during
InitDataDir) dies at startup and never becomes ready, so the first-backup bootstrap retries forever (observed 791× over ~8h in one pod) and the pooler socket never appears — surfacing downstream asMonitorPostgres: failed to create first backupandfailed to connect to Unix socket /var/lib/pooler/pg_sockets/.s.PGSQL.5432.The pgctld logs only show the swallowed
transient PostgreSQL did not become ready after 30 seconds. The real reason is in$PGDATA/setup.log:Cause
pgctld runs
initdbat<pooler-dir>/pg_dataand pins the data directory via-D/PGDATA. But the operator appendsinclude = '/etc/postgresql/postgresql.conf'onto pgctld's generated config (viaPOSTGRES_INITDB_EXTRA_CONF). In the layered image that base config has an activedata_directory = '/var/lib/postgresql/data', and because theincludeis last it wins — redirecting PostgreSQL off the freshly-initdb'd cluster onto the base image's empty, wrong-permission/var/lib/postgresql/data, which fails startup.The pre-layered (non-layered) image shipped
data_directorycommented out, which is why it worked; the layered rewrite left it active. A comment inDockerfile-multigreseven asserted the base config "is never loaded" — that assumption is wrong (it is loaded, via the operator's include), which is the root of the regression.Fix
Comment out
data_directoryin the layered image's copy of/etc/postgresql/postgresql.conf(via the samesedstep that already removes the dangling wal-ginclude), so the base-configincludebecomes harmless regardless of whether it arrives via flag or env var. The stale "never loaded" comment is corrected.postgresql.conf.j2is intentionally left untouched: it is shared by the base supabase image and the VM/ansible deployments, which runpostgres -D /etc/postgresqland genuinely rely ondata_directoryto locate the data — commenting it there would break them. The fix belongs in the Multigres layer, where pgctld pins the data dir itself.Validation
include.postgres -C data_directoryresolves to/var/lib/postgresql/datadespitePGDATAbeing elsewhere; an empty extra-conf (no include) inits cleanly.includestill present anddata_directoryresolves to the correct pooler data dir.🤖 Generated with Claude Code