Primary & Replicas continuously restarting

[vaigau6g]

Please ensure you do the following when reporting a bug:

• Provide a concise description of what the bug is.
  Primary & Replicas continuously restarting in postgresql operator on Openshift 4.10
• Provide information about your environment.
  Openshift - 4.10
  Openshift Data Foundation - odf-operator.v4.10.10
  Crunchydata postgresql Operator - 5.3.0
  pgo-version: 5.3.0
  postgresVersion: 14
• Provide clear steps to reproduce the bug.

1. On Openshift 4.10 with ODF 4.10.10 installed with local storage cluster configured, install Crunchydata postgresql Operator - 5.3.0
2. Create cluster with replicas
3. Start performance testing / or add lots of transactions
4. Continuously restarting primary and replicas

• Attach applicable logs. Please do not attach screenshots showing logs unless you are unable to copy and paste the log data.
  mdsp-psql-lpc-primary-8csb-0-database.log

Overview

Add a concise description of what the bug is.

Environment

Please provide the following details:

• Platform: (Kubernetes, OpenShift, Rancher, GKE, EKS, AKS etc.)
• Platform Version: (e.g. 1.20.3, 4.7.0)
• PGO Image Tag: (e.g. ubi8-5.3.0-0)
• Postgres Version (e.g. 14)
• Storage: (e.g. hostpath, nfs, or the name of your storage class)

Steps to Reproduce

REPRO

Provide steps to get to the error condition:

1. Run ...
2. Do ...
3. Try ...

EXPECTED

1. Provide the behavior that you expected.

ACTUAL

1. Describe what actually happens

Logs

Please provided appropriate log output or any configuration files that may help troubleshoot the issue. DO NOT include sensitive information, such as passwords.

Additional Information

Please provide any additional information that may be helpful.

[vaigau6g]

Exception in primary/replicas pods =======

2023-04-09 14:34:01,752 WARNING: Exception happened during processing of request from ::ffff:10.128.12.1:48184
2023-04-09 14:34:01,754 WARNING: Traceback (most recent call last):
File "/usr/lib64/python3.6/socketserver.py", line 320, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib64/python3.6/socketserver.py", line 669, in process_request
t.start()
File "/usr/lib64/python3.6/threading.py", line 867, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
2023-04-09 14:34:01,755 WARNING: Exception happened during processing of request from ::ffff:10.128.12.1:48182
2023-04-09 14:34:01,755 WARNING: Traceback (most recent call last):
File "/usr/lib64/python3.6/socketserver.py", line 320, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib64/python3.6/socketserver.py", line 669, in process_request
t.start()
File "/usr/lib64/python3.6/threading.py", line 867, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
2023-04-09 14:34:10,034 INFO: no action. I am (mdsp-psql-lpc-primary-8csb-0), the leader with the lock
2023-04-09 14:34:20,031 INFO: no action. I am (mdsp-psql-lpc-primary-8csb-0), the leader with the lock
2023-04-09 14:34:21,753 WARNING: Exception happened during processing of request from ::ffff:10.128.12.1:59482
2023-04-09 14:34:21,754 WARNING: Traceback (most recent call last):
File "/usr/lib64/python3.6/socketserver.py", line 320, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib64/python3.6/socketserver.py", line 669, in process_request
t.start()
File "/usr/lib64/python3.6/threading.py", line 867, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

[vaigau6g]

pg_hba configuration ============================

patronictl show-config
loop_wait: 10
postgresql:
parameters:
archive_command: pgbackrest --stanza=db archive-push "%p"
archive_mode: 'on'
archive_timeout: 60s
default_pool_size: '100'
jit: 'off'
max_connections: '2500'
password_encryption: scram-sha-256
restore_command: pgbackrest --stanza=db archive-get %f "%p"
shared_buffers: 4GB
shared_preload_libraries: pgaudit
ssl: 'on'
ssl_ca_file: /pgconf/tls/ca.crt
ssl_cert_file: /pgconf/tls/tls.crt
ssl_key_file: /pgconf/tls/tls.key
synchronous_commit: 'on'
unix_socket_directories: /tmp/postgres
wal_level: logical
pg_hba:

• local all "postgres" peer
• hostssl replication "_crunchyrepl" all cert
• hostssl "postgres" "_crunchyrepl" all cert
• host all "_crunchyrepl" all reject
• hostssl all "_crunchypgbouncer" all scram-sha-256
• host all "_crunchypgbouncer" all reject
• local all all trust
• host all all all md5
• hostssl all all all md5
  use_pg_rewind: true
  use_slots: false
  synchronous_commit: 'on'
  synchronous_mode: true
  ttl: 30

[vaigau6g]

issue is regarding application requests in psql 
getting below error on psql pod and restarting continuously

Exception happened during processing of request from ::ffff:10.128.10.1:43576
1192023-04-11 06:58:03,453 WARNING: Traceback (most recent call last):
120File "/usr/lib64/python3.6/socketserver.py", line 320, in _handle_request_noblock
121self.process_request(request, client_address)
122File "/usr/lib64/python3.6/socketserver.py", line 669, in process_request
123t.start()
124File "/usr/lib64/python3.6/threading.py", line 867, in start
125_start_new_thread(self._bootstrap, ())
126RuntimeError: can't start new thread

[vaigau6g]

postgres user credentials working with connection string

[vaigau6g]

Is it due to restoring the older version database to the newer version pgo cluster ?
Previous version before upgrade
pgo version = 4.7.3
postgresql version = 13

[andrewlecuyer]

Considering this issue is for an older version of CPK that is no longer actively maintained via the Crunchy Developer Program, I am proceeding with closing (see the Supported Plaforms page for additional information about supported versions of CPK).

For information about upgrading from CPK v4 to v5, please see the upgrade guide:

https://access.crunchydata.com/documentation/postgres-operator/latest/upgrade/v4tov5

And if you still require support for CPK v4.7.3, I recommend recaching out to info@crunchydata.com to discuss your requirements/needs further.