Streaming replication not set up....

[MazenEzzeddine]

Hi,

Streaming replication is not setup and the logs are showing missing files? Any hint? below are the logs for the master and the slave....

Master 👍

2020-04-06 11:09:38.597 GMT [1] LOG: skipping missing configuration file "/etc/replica.conf"
2020-04-06 11:09:38.597 GMT [1] LOG: skipping missing configuration file "/etc/replica.conf"
2020-04-06 11:09:38.606 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2020-04-06 11:09:38.606 UTC [1] LOG: listening on IPv6 address "::", port 5432
2020-04-06 11:09:38.624 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-04-06 11:09:38.693 UTC [22] LOG: database system was interrupted; last known up at 2020-04-06 11:09:36 UTC
2020-04-06 11:09:39.737 UTC [22] LOG: database system was not properly shut down; automatic recovery in progress
2020-04-06 11:09:39.742 UTC [22] LOG: invalid record length at 0/1633A18: wanted 24, got 0
2020-04-06 11:09:39.742 UTC [22] LOG: redo is not required
2020-04-06 11:09:39.793 UTC [1] LOG: database system is ready to accept connections

==============================================================
Slave:

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data/pgdata ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
syncing data to disk ... ok

Success. You can now start the database server using:

pg_ctl -D /var/lib/postgresql/data/pgdata -l logfile start

waiting for server to start....2020-04-06 11:10:33.135 UTC [52] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-04-06 11:10:33.195 UTC [53] LOG: database system was shut down at 2020-04-06 11:10:31 UTC
2020-04-06 11:10:33.212 UTC [52] LOG: database system is ready to accept connections
done
server started

/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

2020-04-06 11:10:33.317 UTC [52] LOG: received fast shutdown request
waiting for server to shut down....2020-04-06 11:10:33.319 UTC [52] LOG: aborting any active transactions
2020-04-06 11:10:33.323 UTC [52] LOG: worker process: logical replication launcher (PID 59) exited with exit code 1
2020-04-06 11:10:33.323 UTC [54] LOG: shutting down
2020-04-06 11:10:33.402 UTC [52] LOG: database system is shut down
done
server stopped

PostgreSQL init process complete; ready for start up.

2020-04-06 11:10:33.431 GMT [1] LOG: skipping missing configuration file "/etc/master.conf"
2020-04-06 11:10:33.432 GMT [1] LOG: skipping missing configuration file "/etc/master.conf"
2020-04-06 11:10:33.437 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2020-04-06 11:10:33.437 UTC [1] LOG: listening on IPv6 address "::", port 5432
2020-04-06 11:10:33.460 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-04-06 11:10:33.508 UTC [61] LOG: database system was shut down at 2020-04-06 11:10:33 UTC
2020-04-06 11:10:33.524 UTC [1] LOG: database system is ready to accept connections....

Thanks...

[gs-offcl]

I am also facing same issue.... The issue coming when I reboot docker daemon...

Any help on below?

2020-04-19 19:24:23.222 GMT [1] LOG: skipping missing configuration file "/etc/replica.conf"
2020-04-19 19:24:23.222 GMT [1] LOG: skipping missing configuration file "/etc/replica.conf"
2020-04-19 19:24:23.279 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2020-04-19 19:24:23.279 UTC [1] LOG: listening on IPv6 address "::", port 5432
2020-04-19 19:24:23.294 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-04-19 19:24:23.635 UTC [139] LOG: database system was interrupted; last known up at 2020-04-19 19:05:46 UTC
2020-04-19 19:24:24.576 UTC [146] FATAL: the database system is starting up
2020-04-19 19:24:29.569 UTC [153] FATAL: the database system is starting up
2020-04-19 19:24:29.789 UTC [160] FATAL: the database system is starting up
2020-04-19 19:24:34.570 UTC [167] FATAL: the database system is starting up
2020-04-19 19:24:39.590 UTC [174] FATAL: the database system is starting up
2020-04-19 19:24:39.797 UTC [181] FATAL: the database system is starting up
2020-04-19 19:24:44.580 UTC [188] FATAL: the database system is starting up
2020-04-19 19:24:49.564 UTC [195] FATAL: the database system is starting up
2020-04-19 19:24:49.791 UTC [202] FATAL: the database system is starting up
2020-04-19 19:24:49.796 UTC [1] LOG: received smart shutdown request
2020-04-19 19:24:54.576 UTC [209] FATAL: the database system is shutting down
2020-04-19 19:24:59.576 UTC [216] FATAL: the database system is shutting down

[arianitu]

@MazenEzzeddine @suresh-offcl

The skipping file error is normal since these are the 2 config variables that include them:

# Add settings for extensions here
include_if_exists = 'master.conf'
include_if_exists = 'replica.conf'

In master, you'll see that replica.conf is skipped and master.conf is skipped on the replica.

Can I get a few details? How is the replica server not working, is it not running? From the logs, I just see that your database is being shutdown and was requested to be shutdown.

[gs-offcl]

For me, the issue is db getting shutdown when I restart docker daemon on host machine where postgres is running as container... we are evaluating this setup in our local env's and we restart docker daemon's frequently...

[gs-offcl]

The issue seems to be coming especially when deleting existing deployments and recreating it by claiming existing PVs... Pl. check this scenario...

please check this link -

https://stackoverflow.com/questions/24442623/fatal-the-database-system-is-shutting-down-while-creating-connection-to-postg

[arianitu]

@gs-offcl Can you provide a reproduction case, please paste your deployment specs? The link you posted looks like the user was running two Postgres instances in one data directory which is a user error.

Are you potentially mounting both your replica and master to the same data directory?

[gs-offcl]

No.... master and replica instances have separate data directories created .. the deployments yamls are used as is with few configurations changes like namespace, storage class etc.

Can you please check below scenarios??

1. Delete existing stateful deployments and recreate it again by claiming PVs

2. Try restarting docker daemon and kubelet..