standby-cluster operator : panic: runtime error: invalid memory address or nil pointer dereference

[thangamani-arun]

• Which image of the operator are you using? e.g. registry.opensource.zalan.do/acid/postgres-operator:v1.7.1
• Where do you run it - cloud or metal? Kubernetes or OpenShift? K8S
• Are you running Postgres Operator in production? yes
• Type of issue? Bug report

Some general remarks when posting a bug report:

There is a Active cluster, then deployed standby cluster in a different Namespace and the streaming replication is working as expected, but the operator keeps restarting after crash. The operator and clusters has been deployed in a isolated-namespace

Pod status

NAME                                                       READY   STATUS             RESTARTS   AGE
automation-gitlab-postgres-0                               1/1     Running            0          32h
gitlab-postgres-operator-7f8ccbc4b8-spvbm                  0/1     CrashLoopBackOff   270        32h
logical-backup-automation-gitlab-postgres-27620800-wz894   0/1     Completed          0          13m
logical-backup-automation-gitlab-postgres-27620805-g82hx   0/1     Completed          0          8m16s
logical-backup-automation-gitlab-postgres-27620810-jlhn2   0/1     Completed          0          3m16s

Operator Pod Logs

time="2022-07-08T02:44:24Z" level=info msg="Spilo operator v1.7.1\n"
time="2022-07-08T02:44:24Z" level=error msg="could not create customResourceDefinition \"operatorconfigurations.acid.zalan.do\": CustomResourceDefinition.apiextensions.k8s.io \"operatorconfigurations.acid.zalan.do\" is invalid: spec.versions[0].schema.openAPIV3Schema: Required value: schemas are required" pkg=controller
time="2022-07-08T02:44:28Z" level=warning msg="in the operator configuration, the pod service account name postgres-pod-gitlab-postgres-dc2 does not match the name postgres-pod given in the account definition; using the former for consistency" pkg=controller
time="2022-07-08T02:44:28Z" level=info msg="Parse role bindings" pkg=controller
time="2022-07-08T02:44:28Z" level=info msg="successfully parsed" pkg=controller
time="2022-07-08T02:44:28Z" level=info msg="Listenting to the specific namespace \"gitlab-postgres-dc2\"" pkg=controller
time="2022-07-08T02:44:28Z" level=info msg="customResourceDefinition \"postgresqls.acid.zalan.do\" is already registered and will only be updated" pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="{" pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ReadyWaitInterval\": 3000000000," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ReadyWaitTimeout\": 30000000000," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ResyncPeriod\": 300000000000," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"RepairPeriod\": 300000000000," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnableCRDValidation\": true," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ResourceCheckInterval\": 3000000000," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ResourceCheckTimeout\": 120000000000," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"PodLabelWaitTimeout\": 600000000000," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"PodDeletionWaitTimeout\": 600000000000," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"SpiloRunAsUser\": null," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"SpiloRunAsGroup\": null," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"SpiloFSGroup\": null," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"PodPriorityClassName\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ClusterDomain\": \"cluster.local\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"SpiloPrivileged\": false," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"SpiloAllowPrivilegeEscalation\": true," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"AdditionalPodCapabilities\": null," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ClusterLabels\": {" pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="      \"application\": \"spilo\"" pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   }," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"InheritedLabels\": null," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"InheritedAnnotations\": null," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"DownscalerAnnotations\": null," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ClusterNameLabel\": \"cluster-name\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"DeleteAnnotationDateKey\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"DeleteAnnotationNameKey\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"PodRoleLabel\": \"spilo-role\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"PodToleration\": null," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"DefaultCPURequest\": \"100m\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"DefaultMemoryRequest\": \"100Mi\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"DefaultCPULimit\": \"1\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"DefaultMemoryLimit\": \"500Mi\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"MinCPULimit\": \"250m\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"MinMemoryLimit\": \"250Mi\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"PodEnvironmentConfigMap\": \"gitlab-postgres-dc2/postgres-pod-config\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"PodEnvironmentSecret\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"NodeReadinessLabel\": null," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"MaxInstances\": -1," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"MinInstances\": -1," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ShmVolume\": true," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"SecretNameTemplate\": \"{username}.{cluster}.credentials.{tprkind}.{tprgroup}\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"PamRoleName\": \"zalandos\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"PamConfiguration\": \"https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"TeamsAPIUrl\": \"https://teams.example.com/api/\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"OAuthTokenSecretName\": \"gitlab-postgres-dc2/gitlab-postgres-operator\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"InfrastructureRolesSecretName\": \"/\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"InfrastructureRoles\": null," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"InfrastructureRolesDefs\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"SuperUsername\": \"postgres\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ReplicationUsername\": \"standby\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ScalyrAPIKey\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ScalyrImage\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ScalyrServerURL\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ScalyrCPURequest\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ScalyrMemoryRequest\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ScalyrCPULimit\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ScalyrMemoryLimit\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"LogicalBackupSchedule\": \"*/5 * * * *\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"LogicalBackupDockerImage\": \"registry.opensource.zalan.do/acid/logical-backup:v1.7.1\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"LogicalBackupProvider\": \"s3\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"LogicalBackupS3Bucket\": \"postgres-backups\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"LogicalBackupS3Region\": \"default\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"LogicalBackupS3Endpoint\": \"https://xx.xx.xx.xx\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"LogicalBackupS3AccessKeyID\": \"ABABABABABABBAAB\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"LogicalBackupS3SecretAccessKey\": \"MASKED/REMOVED\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"LogicalBackupS3SSE\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"LogicalBackupGoogleApplicationCredentials\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"LogicalBackupJobPrefix\": \"logical-backup-\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"NumberOfInstances\": 2," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"Schema\": \"pooler\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"User\": \"pooler\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"Image\": \"registry.opensource.zalan.do/acid/pgbouncer:master-19\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"Mode\": \"transaction\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"MaxDBConnections\": 60," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ConnectionPoolerDefaultCPURequest\": \"500m\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ConnectionPoolerDefaultMemoryRequest\": \"100Mi\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ConnectionPoolerDefaultCPULimit\": \"1\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ConnectionPoolerDefaultMemoryLimit\": \"100Mi\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"WatchedNamespace\": \"gitlab-postgres-dc2\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"KubernetesUseConfigMaps\": false," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EtcdHost\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"DockerImage\": \"registry.opensource.zalan.do/acid/spilo-14:2.1-p3\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"SidecarImages\": null," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"SidecarContainers\": null," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"PodServiceAccountName\": \"postgres-pod-gitlab-postgres-dc2\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"PodServiceAccountDefinition\": \"{\\\"apiVersion\\\":\\\"v1\\\",\\\"kind\\\":\\\"ServiceAccount\\\",\\\"metadata\\\":{\\\"name\\\":\\\"postgres-pod\\\"}}\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"PodServiceAccountRoleBindingDefinition\": \"{\\\"apiVersion\\\":\\\"rbac.authorization.k8s.io/v1\\\",\\\"kind\\\":\\\"RoleBinding\\\",\\\"metadata\\\":{\\\"name\\\":\\\"postgres-pod-gitlab-postgres-dc2\\\"},\\\"roleRef\\\":{\\\"apiGroup\\\":\\\"rbac.authorization.k8s.io\\\",\\\"kind\\\":\\\"ClusterRole\\\",\\\"name\\\":\\\"postgres-pod-gitlab-postgres-dc2\\\"},\\\"subjects\\\":[{\\\"kind\\\":\\\"ServiceAccount\\\",\\\"name\\\":\\\"postgres-pod-gitlab-postgres-dc2\\\"}]}\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"MasterPodMoveTimeout\": 1200000000000," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"DbHostedZone\": \"db.example.com\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"AWSRegion\": \"eu-central-1\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"WALES3Bucket\": \"postgres-geo-standby\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"LogS3Bucket\": \"postgres-logs\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"KubeIAMRole\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"WALGSBucket\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"GCPCredentials\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"WALAZStorageAccount\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"AdditionalSecretMount\": \"\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"AdditionalSecretMountPath\": \"/meta/credentials\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnableEBSGp3Migration\": false," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnableEBSGp3MigrationMaxSize\": 1000," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"DebugLogging\": false," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnableDBAccess\": true," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnableTeamsAPI\": false," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnableTeamSuperuser\": false," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"TeamAdminRole\": \"admin\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"RoleDeletionSuffix\": \"_deleted\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnableTeamMemberDeprecation\": false," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnableAdminRoleForUsers\": true," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnablePostgresTeamCRD\": false," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnablePostgresTeamCRDSuperusers\": false," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnableMasterLoadBalancer\": false," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnableReplicaLoadBalancer\": false," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"CustomServiceAnnotations\": null," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"CustomPodAnnotations\": null," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnablePodAntiAffinity\": false," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"PodAntiAffinityTopologyKey\": \"kubernetes.io/hostname\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"StorageResizeMode\": \"pvc\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnableLoadBalancer\": null," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ExternalTrafficPolicy\": \"Cluster\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"MasterDNSNameFormat\": \"{cluster}.{team}.{hostedzone}\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ReplicaDNSNameFormat\": \"{cluster}-repl.{team}.{hostedzone}\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"PDBNameFormat\": \"postgres-{cluster}-pdb\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnablePodDisruptionBudget\": true," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnableInitContainers\": true," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnableSidecars\": true," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"Workers\": 8," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"APIPort\": 8080," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"RingLogLines\": 100," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ClusterHistoryEntries\": 1000," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"TeamAPIRoleConfiguration\": {" pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="      \"log_statement\": \"all\"" pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   }," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"PodTerminateGracePeriod\": 30000000000," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"PodManagementPolicy\": \"ordered_ready\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"ProtectedRoles\": [" pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="      \"admin\"" pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   ]," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"PostgresSuperuserTeams\": [" pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="      \"postgres_superusers\"" pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   ]," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"SetMemoryRequestToLimit\": false," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnableLazySpiloUpgrade\": false," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnableCrossNamespaceSecret\": false," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnablePgVersionEnvVar\": true," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"EnableSpiloWalPathCompat\": true," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"MajorVersionUpgradeMode\": \"off\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"MinimalMajorVersion\": \"9.6\"," pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="   \"TargetMajorVersion\": \"14\"" pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="}" pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="SYNC event has been queued" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=controller worker=0
time="2022-07-08T02:44:33Z" level=info msg="there are 1 clusters running" pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="started working in background" pkg=controller
time="2022-07-08T02:44:33Z" level=info msg="listening on :8080" pkg=apiserver
time="2022-07-08T02:44:33Z" level=info msg="ADD event has been queued" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=controller worker=0
time="2022-07-08T02:44:33Z" level=info msg="syncing of the cluster started" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=controller worker=0
time="2022-07-08T02:44:33Z" level=info msg="syncing secrets" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=cluster
time="2022-07-08T02:44:34Z" level=info msg="volume claims do not require changes" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=cluster
time="2022-07-08T02:44:34Z" level=info msg="Standby from S3 bucket using custom parsed S3WalPath from the manifest s3://postgres-geo-standby/spilo/automation-gitlab-postgres/e4e66ba9-ab1c-40ba-8077-70129ad6664b/wal/14/ " cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=cluster
W0708 02:44:46.894287       1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0708 02:44:46.912270       1 warnings.go:70] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
time="2022-07-08T02:44:46Z" level=info msg="Mount additional volumes: []" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=cluster
time="2022-07-08T02:44:46Z" level=info msg="Installing lookup function" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=cluster
time="2022-07-08T02:44:46Z" level=info msg="install pooler lookup function into database 'dc1_db'" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=cluster
time="2022-07-08T02:44:47Z" level=error msg="could not execute after retries \n\t\tCREATE SCHEMA IF NOT EXISTS pooler;\n\n\t\tCREATE OR REPLACE FUNCTION pooler.user_lookup(\n\t\t\tin i_username text, out uname text, out phash text)\n\t\tRETURNS record AS $$\n\t\tBEGIN\n\t\t\tSELECT usename, passwd FROM pg_catalog.pg_shadow\n\t\t\tWHERE usename = i_username INTO uname, phash;\n\t\t\tRETURN;\n\t\tEND;\n\t\t$$ LANGUAGE plpgsql SECURITY DEFINER;\n\n\t\tREVOKE ALL ON FUNCTION pooler.user_lookup(text)\n\t\t\tFROM public, pooler;\n\t\tGRANT EXECUTE ON FUNCTION pooler.user_lookup(text)\n\t\t\tTO pooler;\n\t\tGRANT USAGE ON SCHEMA pooler TO pooler;\n\t: pq: cannot execute CREATE SCHEMA in a read-only transaction" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=cluster
time="2022-07-08T02:44:47Z" level=info msg="install pooler lookup function into database 'postgres'" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=cluster
time="2022-07-08T02:44:47Z" level=error msg="could not execute after retries \n\t\tCREATE SCHEMA IF NOT EXISTS pooler;\n\n\t\tCREATE OR REPLACE FUNCTION pooler.user_lookup(\n\t\t\tin i_username text, out uname text, out phash text)\n\t\tRETURNS record AS $$\n\t\tBEGIN\n\t\t\tSELECT usename, passwd FROM pg_catalog.pg_shadow\n\t\t\tWHERE usename = i_username INTO uname, phash;\n\t\t\tRETURN;\n\t\tEND;\n\t\t$$ LANGUAGE plpgsql SECURITY DEFINER;\n\n\t\tREVOKE ALL ON FUNCTION pooler.user_lookup(text)\n\t\t\tFROM public, pooler;\n\t\tGRANT EXECUTE ON FUNCTION pooler.user_lookup(text)\n\t\t\tTO pooler;\n\t\tGRANT USAGE ON SCHEMA pooler TO pooler;\n\t: pq: cannot execute CREATE SCHEMA in a read-only transaction" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=cluster
time="2022-07-08T02:44:47Z" level=info msg="install pooler lookup function into database 'gitlabhq_production'" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=cluster
time="2022-07-08T02:44:47Z" level=error msg="could not execute after retries \n\t\tCREATE SCHEMA IF NOT EXISTS pooler;\n\n\t\tCREATE OR REPLACE FUNCTION pooler.user_lookup(\n\t\t\tin i_username text, out uname text, out phash text)\n\t\tRETURNS record AS $$\n\t\tBEGIN\n\t\t\tSELECT usename, passwd FROM pg_catalog.pg_shadow\n\t\t\tWHERE usename = i_username INTO uname, phash;\n\t\t\tRETURN;\n\t\tEND;\n\t\t$$ LANGUAGE plpgsql SECURITY DEFINER;\n\n\t\tREVOKE ALL ON FUNCTION pooler.user_lookup(text)\n\t\t\tFROM public, pooler;\n\t\tGRANT EXECUTE ON FUNCTION pooler.user_lookup(text)\n\t\t\tTO pooler;\n\t\tGRANT USAGE ON SCHEMA pooler TO pooler;\n\t: pq: cannot execute CREATE SCHEMA in a read-only transaction" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=cluster
time="2022-07-08T02:44:47Z" level=warning msg="error while syncing cluster state: could not install pooler lookup function in every specified databases" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=cluster
time="2022-07-08T02:44:47Z" level=error msg="could not sync cluster: could not sync connection pooler: could not install pooler lookup function in every specified databases" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=controller worker=0
time="2022-07-08T02:44:47Z" level=info msg="recieved add event for already existing Postgres cluster" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=controller worker=0
time="2022-07-08T02:49:33Z" level=info msg="SYNC event has been queued" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=controller worker=0
time="2022-07-08T02:49:33Z" level=info msg="there are 1 clusters running" pkg=controller
time="2022-07-08T02:49:33Z" level=info msg="syncing of the cluster started" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=controller worker=0
time="2022-07-08T02:49:33Z" level=info msg="syncing secrets" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=cluster
time="2022-07-08T02:49:33Z" level=info msg="volume claims do not require changes" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=cluster
time="2022-07-08T02:49:34Z" level=info msg="Standby from S3 bucket using custom parsed S3WalPath from the manifest s3://postgres-geo-standby/spilo/automation-gitlab-postgres/e4e66ba9-ab1c-40ba-8077-70129ad6664b/wal/14/ " cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=cluster
W0708 02:49:46.487607       1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0708 02:49:46.498644       1 warnings.go:70] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
time="2022-07-08T02:49:46Z" level=info msg="Mount additional volumes: []" cluster-name=gitlab-postgres-dc2/automation-gitlab-postgres pkg=cluster
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x1a4d62e]

goroutine 114 [running]:
github.com/zalando/postgres-operator/pkg/cluster.(*Cluster).syncConnectionPooler(0xc00012c000, 0xc0010dd398, 0xc001400800, 0xc0010dd368, 0x1, 0x0, 0x0, 0x0, 0x0)
        /workspace/pkg/cluster/connection_pooler.go:715 +0xd6e
github.com/zalando/postgres-operator/pkg/cluster.(*Cluster).Sync(0xc00012c000, 0xc001400800, 0x0, 0x0)
        /workspace/pkg/cluster/sync.go:128 +0x7e9
github.com/zalando/postgres-operator/pkg/controller.(*Controller).processEvent(0xc00052c000, 0xc0aa04af43a7f7ae, 0x47daa2aa68, 0x30e4920, 0xc00104a1e0, 0x24, 0x1fdd42e, 0x4, 0x0, 0xc001400800, ...)
        /workspace/pkg/controller/postgresql.go:310 +0xc75
github.com/zalando/postgres-operator/pkg/controller.(*Controller).processClusterEventsQueue(0xc00052c000, 0x0, 0xc000114f00, 0xc000138000)
        /workspace/pkg/controller/postgresql.go:344 +0x278
created by github.com/zalando/postgres-operator/pkg/controller.(*Controller).Run
        /workspace/pkg/controller/controller.go:440 +0x8a

[thangamani-arun]

@FxKu any comments or feedback?

[FxKu]

@thangamani-arun this nil pointer was fixed with #1701. Please, update the operator to the newest version.
Another issue that is already fixed is that a standby cluster could not have connection pooler enabled in the manifest. Because the operator then tries to install the pooler schema which fails as a standby cluster is read-only. You can see the error in your logs could not install pooler lookup function.

[thangamani-arun]

@FxKu: Thanks for your comments, after upgrade to 1.8.1 there are no such failures.