Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

too many forks #711

Open
ahoiahoi opened this issue Sep 10, 2021 · 1 comment
Open

too many forks #711

ahoiahoi opened this issue Sep 10, 2021 · 1 comment
Assignees
@ibarwick
Labels
question Question(s) about repmgr usage resolution-proposed Resolution proposed for issue, awaiting feedback

Comments

@ahoiahoi
Copy link

ahoiahoi commented Sep 10, 2021 •
edited
Loading

Greetings

We have a 2 node setup. When the primary node goes offline (eg stop postgres & repmgr), the standby would promote itself.
The setup used to work - I am not aware of any changes.
The problem I see is that the promote command is forked until the max connections (set to 100) of the database are exhausted and accepts no more connections;

[..]
[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [ERROR] connection to database failed
[2021-09-10 16:57:56] [DETAIL]
FATAL:  sorry, too many clients already
[..]

When running "ps -ef" immediately when the promote is executed, there are indeed many "standby promote" commands running

Versions:
$ rpm -qa|egrep 'postgres|repmgr'
postgresql11-11.8-1PGDG.rhel7.x86_64
repmgr11-5.2.1-1.el7.x86_64
postgresql11-contrib-11.8-1PGDG.rhel7.x86_64
postgresql11-libs-11.8-1PGDG.rhel7.x86_64
postgresql11-server-11.8-1PGDG.rhel7.x86_64

Thanks in advance

repmgr.conf 
node_id=                                2   # A unique integer greater than zero
node_name=                              node-b         # An arbitrary (but unique) string; we recommend
conninfo=                               'host=10.222.67.87 user=repmgr dbname=repmgr'               # Database connection information as a conninfo string.
data_directory=                         '/var/lib/pgsql/11/data'        # The node's data directory. This is needed by repmgr
replication_user=                       postgres      # User to make replication connections with, if not set
location=                               default    # An arbitrary string defining the location of the node; this
use_replication_slots=                  1       # whether to use physical replication slots
log_level=                              INFO            # Log level: possible values are DEBUG, INFO, NOTICE,
log_facility=                           STDERR                 # Logging facility: possible values are STDERR, or for
log_file=                               '/var/log/repmgr/repmgr.log'                  # STDERR can be redirected to an arbitrary file
log_status_interval=                    300      # interval (in seconds) for repmgrd to log a status message
pg_bindir=                              '/usr/pgsql-11/bin'        # Path to PostgreSQL binary directory (location
repmgr_bindir=                          '/usr/pgsql-11/bin'        # Path to repmgr binary directory (location of the repmgr
pg_basebackup_options=                  '--wal-method=fetch'            # Options to append to "pg_basebackup"
ssh_options=                            '-q -o ConnectTimeout=10'       # Options to append to "ssh"
failover=                               automatic                   # one of 'automatic', 'manual'.
priority=                               100                   # indicates a preferred priority for promoting nodes;
connection_check_type=                  ping                                    # How to check availability of the upstream node; valid options:
promote_command=                        '/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file'          # command repmgrd executes when promoting a new primary; use something like:
follow_command=                         '/usr/pgsql-11/bin/repmgr standby follow -f /var/lib/pgsql/11/repmgr.conf --upstream-node-id=%n --log-to-file'           # command repmgrd executes when instructing a standby to follow a new primary;
monitoring_history=                     'True'               # Whether to write monitoring data to the "montoring_history" table
repmgrd_pid_file=                       '/var/lib/pgsql/11'               # Path of PID file to use for repmgrd; if not set, a PID file will
service_start_command =                 'systemctl start postgresql-11'
service_stop_command =                  'systemctl stop postgresql-11'
service_restart_command =               'systemctl restart postgresql-11'
service_reload_command =                'systemctl reload postgresql-11'
service_promote_command =               '/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file'           # This parameter is intended for systems which provide a
repmgrd_service_start_command =         'systemctl start repmgr'
repmgrd_service_stop_command =          'systemctl stop repmgr'
full log 
[2021-09-10 16:57:05] [WARNING] unable to ping "host=10.222.67.86 user=repmgr dbname=repmgr"
[2021-09-10 16:57:05] [DETAIL] PQping() returned "PQPING_REJECT"
[2021-09-10 16:57:05] [WARNING] unable to connect to upstream node "node-a" (ID: 1)
[2021-09-10 16:57:05] [INFO] checking state of node "node-a" (ID: 1), 1 of 6 attempts
[2021-09-10 16:57:05] [WARNING] unable to ping "user=repmgr dbname=repmgr host=10.222.67.86 connect_timeout=2 fallback_application_name=repmgr"
[2021-09-10 16:57:05] [DETAIL] PQping() returned "PQPING_REJECT"
[2021-09-10 16:57:05] [INFO] sleeping up to 10 seconds until next reconnection attempt
[2021-09-10 16:57:15] [INFO] checking state of node "node-a" (ID: 1), 2 of 6 attempts
[2021-09-10 16:57:15] [WARNING] unable to ping "user=repmgr dbname=repmgr host=10.222.67.86 connect_timeout=2 fallback_application_name=repmgr"
[2021-09-10 16:57:15] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:57:15] [INFO] sleeping up to 10 seconds until next reconnection attempt
[2021-09-10 16:57:25] [INFO] checking state of node "node-a" (ID: 1), 3 of 6 attempts
[2021-09-10 16:57:25] [WARNING] unable to ping "user=repmgr dbname=repmgr host=10.222.67.86 connect_timeout=2 fallback_application_name=repmgr"
[2021-09-10 16:57:25] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:57:25] [INFO] sleeping up to 10 seconds until next reconnection attempt
[2021-09-10 16:57:35] [INFO] checking state of node "node-a" (ID: 1), 4 of 6 attempts
[2021-09-10 16:57:35] [WARNING] unable to ping "user=repmgr dbname=repmgr host=10.222.67.86 connect_timeout=2 fallback_application_name=repmgr"
[2021-09-10 16:57:35] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:57:35] [INFO] sleeping up to 10 seconds until next reconnection attempt
[2021-09-10 16:57:45] [INFO] checking state of node "node-a" (ID: 1), 5 of 6 attempts
[2021-09-10 16:57:45] [WARNING] unable to ping "user=repmgr dbname=repmgr host=10.222.67.86 connect_timeout=2 fallback_application_name=repmgr"
[2021-09-10 16:57:45] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:57:45] [INFO] sleeping up to 10 seconds until next reconnection attempt
[2021-09-10 16:57:55] [INFO] checking state of node "node-a" (ID: 1), 6 of 6 attempts
[2021-09-10 16:57:55] [WARNING] unable to ping "user=repmgr dbname=repmgr host=10.222.67.86 connect_timeout=2 fallback_application_name=repmgr"
[2021-09-10 16:57:55] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:57:55] [WARNING] unable to reconnect to node "node-a" (ID: 1) after 6 attempts
[2021-09-10 16:57:55] [INFO] 0 active sibling nodes registered
[2021-09-10 16:57:55] [INFO] 2 total nodes registered
[2021-09-10 16:57:55] [INFO] primary node  "node-a" (ID: 1) and this node have the same location ("default")
[2021-09-10 16:57:55] [INFO] no other sibling nodes - we win by default
[2021-09-10 16:57:55] [NOTICE] this node is the only available candidate and will now promote itself
[2021-09-10 16:57:55] [INFO] promote_command is:
  "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:55] [NOTICE] promoting standby to primary
[2021-09-10 16:57:55] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:55] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [NOTICE] promoting standby to primary
[2021-09-10 16:57:56] [DETAIL] promoting server "node-b" (ID: 2) using "/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file"
[2021-09-10 16:57:56] [NOTICE] redirecting logging output to "/var/log/repmgr/repmgr.log"

[2021-09-10 16:57:56] [ERROR] connection to database failed
[2021-09-10 16:57:56] [DETAIL]
FATAL:  sorry, too many clients already

[2021-09-10 16:57:56] [DETAIL] attempted to connect using:
  user=repmgr dbname=repmgr host=10.222.67.87 connect_timeout=2 fallback_application_name=repmgr options=-csearch_path=
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] unable to promote server from standby to primary
[2021-09-10 16:57:56] [ERROR] promote command failed
[2021-09-10 16:57:56] [DETAIL] promote command exited with error code 8
[2021-09-10 16:57:56] [INFO] checking if original primary node has reappeared
[2021-09-10 16:57:56] [INFO] searching for primary node
[2021-09-10 16:57:56] [INFO] checking if node 1 is primary
[2021-09-10 16:57:56] [ERROR] connection to database failed
[2021-09-10 16:57:56] [DETAIL]
could not connect to server: Connection refused
        Is the server running on host "10.222.67.86" and accepting
        TCP/IP connections on port 5432?

[2021-09-10 16:57:56] [DETAIL] attempted to connect using:
  user=repmgr dbname=repmgr host=10.222.67.86 connect_timeout=2 fallback_application_name=repmgr options=-csearch_path=
[2021-09-10 16:57:56] [INFO] checking if node 2 is primary
[2021-09-10 16:57:56] [WARNING] unable to ping "host=10.222.67.86 user=repmgr dbname=repmgr"
[2021-09-10 16:57:56] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:57:56] [ERROR] unable to execute get_primary_current_lsn()
[2021-09-10 16:57:56] [DETAIL]
FATAL:  terminating connection due to administrator command
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

[2021-09-10 16:57:56] [WARNING] unable to retrieve primary's current LSN
[2021-09-10 16:57:58] [WARNING] unable to ping "host=10.222.67.86 user=repmgr dbname=repmgr"
[2021-09-10 16:57:58] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:57:58] [WARNING] unable to ping "host=10.222.67.86 user=repmgr dbname=repmgr"
[2021-09-10 16:57:58] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:57:58] [WARNING] monitoring_history requested but primary connection not available
[2021-09-10 16:58:00] [WARNING] unable to ping "host=10.222.67.86 user=repmgr dbname=repmgr"
[2021-09-10 16:58:00] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:58:00] [WARNING] unable to ping "host=10.222.67.86 user=repmgr dbname=repmgr"
[2021-09-10 16:58:00] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2021-09-10 16:58:00] [WARNING] monitoring_history requested but primary connection not available
[2021-09-10 16:58:02] [WARNING] unable to ping "host=10.222.67.86 user=repmgr dbname=repmgr"
[2021-09-10 16:58:02] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
@ahoiahoi ahoiahoi changed the title too mand forks too many forks Sep 10, 2021
@ibarwick ibarwick self-assigned this Jan 4, 2022
@ibarwick
Copy link
Collaborator

ibarwick commented Jan 4, 2022

You have this entry in your repmgr.conf file:

service_promote_command = '/usr/pgsql-11/bin/repmgr standby promote -f /var/lib/pgsql/11/repmgr.conf --log-to-file'

The full comment in the sample repmgr.conf file (which is partially included in your repmgr.conf file) is:

#service_promote_command = ''		# This parameter is intended for systems which provide a
					# package-level promote command, such as Debian's
					# "pg_ctlcluster". *IMPORTANT*: it is *not* a substitute
					# for "promote_command"; do not use "repmgr standby promote"
					# (or a script which executes "repmgr standby promote") here.

so what's happening is that repmgr standby promote is executing the command defined in service_promote_command, which is repmgr standby promote, which will continue executing itself until resources are exhausted.

So just remove the service_promote_command.

@ibarwick ibarwick added question Question(s) about repmgr usage resolution-proposed Resolution proposed for issue, awaiting feedback labels Jan 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ibarwick ibarwick

Labels
question Question(s) about repmgr usage resolution-proposed Resolution proposed for issue, awaiting feedback
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ibarwick @ahoiahoi