Replica not starting with pgpool or pgbouncer

[domcar]

Describe the bug
If I create a cluster with flag pgpool or pgbouncer the replica cannot connect to DB

To Reproduce
./pgo-mac.3.5.1 create cluster tre --pgpool=true --password=ciaociao --replica-count=1

Screenshots

postgres-operator-pgo-6f9886f76d-7t757   3/3       Running   0          2h
tre-5b6df6764d-zjxc8                     2/2       Running   0          13m
tre-cxem-76bff775b7-ck6bc                1/2       Running   0          13m

Logs from the replica k logs tre-cxem-76bff775b7-ck6bc -c database

Mon Mar 18 14:47:56 UTC 2019 INFO: Setting PGROOT to /usr/pgsql-11.
Mon Mar 18 14:47:56 UTC 2019 INFO: The PGUSER secret exists.
Mon Mar 18 14:47:57 UTC 2019 INFO: The PGROOT secret exists.
Mon Mar 18 14:47:57 UTC 2019 INFO: The PGPRIMARY secret exists.
Mon Mar 18 14:47:57 UTC 2019 INFO: Cleaning up the old postmaster.pid file..
Mon Mar 18 14:47:57 UTC 2019 INFO: User ID is set to uid=26(postgres) gid=26(postgres) groups=26(postgres).
Mon Mar 18 14:47:57 UTC 2019 INFO: Setting LOG_MIN_DURATION_STATEMENT to 60000.
Mon Mar 18 14:47:57 UTC 2019 INFO: Setting LOG_STATEMENT to none.
Mon Mar 18 14:47:57 UTC 2019 INFO: Working on replica..
Mon Mar 18 14:47:57 UTC 2019 INFO: Initializing the replica.
Mon Mar 18 14:47:57 UTC 2019 INFO: Waiting to allow the primary database time to successfully start before performing the initial backup..
tre:5432 - no response
tre:5432 - no response
tre:5432 - no response
tre:5432 - no response
tre:5432 - no response
tre:5432 - no response
tre:5432 - no response

Please tell us about your environment:

• Operating System: MacOS
• Where is this running ( Local, Cloud Provider): on GCP
• Storage being used (NFS, Hostpath, Gluster, etc): GCE
• Container Image Tag: centos7-3.5.1
• PostgreSQL Version: centos7-11.2-2.3.1
• Platform (Docker, Kubernetes, OpenShift): K8S
• Platform Version: k8s 1.11

[Sayrus]

Might be linked to #637.
Tagging with pgbouncer does disable service-name tagging as it will trigger newpod.ObjectMeta.Labels[util.LABEL_PGBOUNCER] == "true" in controller/podcontroller.go and return before tagging the service-name.
I will change my issue title.

I am currently using this patch and will try my luck with a PR as soon as I have a better way to do it:

diff --git a/controller/podcontroller.go b/controller/podcontroller.go
index b2a2d214..a1cc1057 100644
--- a/controller/podcontroller.go
+++ b/controller/podcontroller.go
@@ -254,13 +254,15 @@ func isPostgresPod(newpod *apiv1.Pod) bool {
                log.Debugf("pgo-backrest-repo pod added [%s]", newpod.Name)
                return false
        }
-       if newpod.ObjectMeta.Labels[util.LABEL_PGPOOL] == "true" {
-               log.Debugf("pgpool pod added [%s]", newpod.Name)
-               return false
-       }
-       if newpod.ObjectMeta.Labels[util.LABEL_PGBOUNCER] == "true" {
-               log.Debugf("pgbouncer pod added [%s]", newpod.Name)
-               return false
+       if _, ok := newpod.ObjectMeta.Labels[util.LABEL_PRIMARY]; !ok {
+               if newpod.ObjectMeta.Labels[util.LABEL_PGPOOL] == "true" {
+                       log.Debugf("pgpool pod added [%s]", newpod.Name)
+                       return false
+               }
+               if newpod.ObjectMeta.Labels[util.LABEL_PGBOUNCER] == "true" {
+                       log.Debugf("pgbouncer pod added [%s]", newpod.Name)
+                       return false
+               }
        }
        return true
 }

Note that it does not changes getType in apiserver/clusterservice/clusterimpl.go which means you won't be able to retrieve your pods correctly with the PGO executable:

	pod : test-cluster-7d85d96489-b8lwp (Running) on node-6 (1/1) (pgbouncer)
	pvc : test-cluster
	pod : test-cluster-dqyh-899bc5f84-jdtt4 (Running) on node-6 (1/1) (pgbouncer)
	pvc : test-cluster-dqyh
	pod : test-cluster-pgbouncer-5456b87cd7-jzhsw (Running) on node-6 (1/1) (pgbouncer)
	pod : test-cluster-skcb-5bb5f9c5b8-cvzzj (Running) on node-5 (1/1) (pgbouncer)
	pvc : test-cluster-skcb

All pods are detected as pgbouncer as they have the crunchy-pgbouncer: true label.

You can try this patch with CO_IMAGE_PREFIX=esayrus. It currently only has postgres-operator so I am building and pushing the others.
Please tell me if this does fix the problem and if pg-bouncer works correctly. (I only tested that it deploys all green on my cluster)

[domcar]

Thanks @Sayrus I will have a look into it, by the way it is possible to start the cluster without pgpool/pgbouncer and add them only later with a pgo create pgpool test-cluster and it appears to be working fine

[Sayrus]

Thanks @domcar I tried this for pgpool too.
While this works on an existing cluster, you won't be able to scale your cluster without manually deleting the label as new deployment will have the crunchy-pgbouncer label.

[domcar]

@Sayrus that's true, if I scale the replica it will be labeled as bouncer

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.