using postgres user for pgbouncer/pooler #892

haroonb · 2020-03-30T11:17:26Z

Hello

I'm trying to use the new integrated pgbouncer feature and running into some trouble. The pods for the pgbouncer are not being created. As far as I can tell from the documentation the configuration files are setup correctly.

I would assume with this configuration the postgres-operator would create {cluster-name}-pooler like documented, but only the cluster is created.

## configmap
...
data:
  # additional_secret_mount: "some-secret-name"
  # additional_secret_mount_path: "/some/dir"
  api_port: "8080"
  aws_region: eu-central-1
  cluster_domain: cluster.local
  cluster_history_entries: "1000"
  cluster_labels: application:spilo
  cluster_name_label: cluster-name
  connection_pool_default_cpu_limit: "1"
  connection_pool_default_cpu_request: "500m"
  connection_pool_default_memory_limit: "250Mi"
  connection_pool_default_memory_request: "100Mi"
  connection_pool_image: "registry.opensource.zalan.do/acid/pgbouncer:master-5"
  connection_pool_max_db_connections: "100"
  connection_pool_mode: "transaction"
  connection_pool_number_of_instances: "4"
  connection_pool_schema: "postgres"
  connection_pool_user: "postgres"
  #  custom_service_annotations: "keyx:valuez,keya:valuea"
  # custom_pod_annotations: "keya:valuea,keyb:valueb"
...

apiVersion: "acid.zalan.do/v1"
kind: postgresql
metadata:
  name: acid-postgres-cluster
spec:
  enableConnectionPool: "true"
  connectionPool:
    number_of_instances: "4"
    mode: "transaction"
    schema: "postgres"
    user: "postgres"
    resources:
      requests:
        cpu: 500m
        memory: 100Mi
      limits:
        cpu: "1"
        memory: 250Mi
  dockerImage: registry.opensource.zalan.do/acid/spilo-12:1.6-p2
  teamId: "acid"
  volume:
    size: 10Gi
    storageClass: rook-ceph-block
  numberOfInstances: 3
  enableMasterLoadBalancer: false
  enableReplicaLoadBalancer: false
  allowedSourceRanges:  # load balancers' source ranges for both master and replica services
  - 127.0.0.1/32
  postgresql:
    version: "12"
    parameters: # Expert section
      tcp_keepalives_count: "5"
      tcp_keepalives_interval: "60"
      tcp_keepalives_idle: "60"
      max_connections: "300"
      shared_buffers: "4GB"
      log_statement: "all"
  enableShmVolume: true
  resources:
    requests:
      cpu: 10m
      memory: 100Mi
    limits:
      cpu: 5000m
      memory: 5000Mi
  patroni:
    initdb:
      encoding: "UTF8"
      locale: "en_US.UTF-8"
      data-checksums: "true"
    pg_hba:
    - hostssl all all 0.0.0.0/0 md5
    - host    all all 0.0.0.0/0 md5
    ttl: 30
    loop_wait: &loop_wait 10
    retry_timeout: 10
    maximum_lag_on_failover: 33554432
  initContainers:
  - name: date
    image: busybox
    command: [ "/bin/date" ]
  tls:
    secretName: ""  # should correspond to a Kubernetes Secret resource to load
    certificateFile: "tls.crt"
    privateKeyFile: "tls.key"
    caFile: ""  # optionally configure Postgres with a CA certificate

The text was updated successfully, but these errors were encountered:

FxKu · 2020-03-30T13:10:45Z

I guess, you're using the v1.4.0 image, right? We only update the operator image in the deployment manifests (and helm chart) during releases, which means in order to use the new features you either have to build it yourself, use the latest tag or grab the tag from the travis pipeline. Note, that we will change the naming of the connection pool settings to "...pooler" in one of the next commits.

FxKu · 2020-04-01T09:42:08Z

the renaming got merged. To try it out use the freshly build image registry.opensource.zalan.do/acid/postgres-operator:v1.4.0-20-gb43b22d-dirty

haroonb · 2020-04-01T09:47:39Z

Thanks for the message I was testing stuff with the old build and running into a few issues, I'll pull the latest repo and see, if my issues persist.

Update 1: The operator doesn't seem to listen to "...pooler: true" yet, but still to "...pool: true. The operator, also doesn't seem to accept my values to create 3 pgbouncers or change the resources.

Update 2: switching postgres-operator tag from latest to 20-gb43b22d-dirty seems to do the trick, but the generated poolers go into a crashLoopBackoff with the message "2020-04-01 12:37:55.278 UTC [1] ERROR invalid value "" for parameter pool_mode in configuration (/etc/pgbouncer/pgbouncer.ini:5)". I tried to remove the entry from the cluster.yaml and set it to default in the configmap, but it didn't seem to help.
The operator thew this error message, don't know if its helpful "postgres-operator-6b8966ffd7-d4j7p postgres-operator panic: runtime error: invalid memory address or nil pointer dereference"

haroonb · 2020-04-03T10:38:23Z

I'm running into another issue with the pooler feature. As I understand from the documentation connection_pool_schema: and connection_pool_user: is responsible for the pooler to choose the correct secret/user to pass and the pooler. I want the pooler to choose the already created postgres.acid-test-cluster.credentials, Which has the postgres user and password in it.

If I only spcify connection_pool_schema: postgres the operator creates new credentials with the default settings.

If I only spcify connection_pool_user: postgres the operator creates no credentials and no poolers are created.

If I specify connection_pool_user: postgres2 and connection_pool_schema: postgres3 the operator creates postgres3.acid-test-cluster.credentials with the username: postgres3

As far as I understand from the documentation this not intended or am I understanding incorrectly?

FxKu · 2020-04-03T12:54:15Z

We forgot the reflect the renaming in our pgBouncer image 😞 . Will fix that now.

The operator creates a lookup function for credentials to be used by pgBouncer. With the ..._schema, you can tell in which schema it should be installed. And with .._user which DB role shall be used. You can reuse postgres for both. By default, a new dedicated role pooler is created.

FxKu · 2020-04-03T15:00:20Z

Fixed. New image is: registry.opensource.zalan.do/acid/postgres-operator:v1.4.0-22-g64389b8-dirty. Sorry for letting you run into these issues.

vitobotta · 2020-04-04T17:29:42Z

Hi, I was reading about pgbouncer and just saw here that it's now supported in the operator. I'm on 1.4.0. What are the steps to use it now? Is there an estimated release date for the version that will include this? Thanks!

haroonb · 2020-04-06T08:37:40Z

@FxKu I think this issue may need to be reopened. I tried to get the latest image up and running, but I'm still running into the same issues as before. I'll append the log from the postgres-operator, maybe it will help find the solution.

The logs is from the scenario, where I set connection_pooler_schema: "postgres" and comment # connection_pooler_user: "postgres"

From our discussion I would assume this configuration would result in the pooler using the existing secret/postgres.acid-postgres-cluster.credentials to pass to the pooler. The result is, that the operator creates a new secret/pooler.acid-postgres-cluster.credentials and passes this to the pooler. I can verify it, by looking at the /etc/pgbouncer/auth_file.txt in one of the containers, it contains "pooler" "xxxxxxxxxxxxxxxxx".

I would assume, this is not the intended behaviour.

+ postgres-operator-7857d8956b-zd892 › postgres-operator
postgres-operator-7857d8956b-zd892 postgres-operator 2020/04/06 07:52:26 Fully qualified configmap name: default/postgres-operator
postgres-operator-7857d8956b-zd892 postgres-operator 2020/04/06 07:52:26 Spilo operator v1.4.0-22-g64389b8-dirty
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:26Z" level=info msg="Parse role bindings" pkg=controller
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:26Z" level=info msg="successfully parsed" pkg=controller
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:26Z" level=info msg="Listening to all namespaces" pkg=controller
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:26Z" level=info msg="customResourceDefinition \"postgresqls.acid.zalan.do\" has been registered" pkg=controller
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:30Z" level=info msg="config: {\n\t\"ReadyWaitInterval\": 3000000000,\n\t\"ReadyWaitTimeout\": 30000000000,\n\t\"ResyncPeriod\": 1800000000000,\n\t\"RepairPeriod\": 300000000000,\n\t\"EnableCRDValidation\": true,\n\t\"ResourceCheckInterval\": 3000000000,\n\t\"ResourceCheckTimeout\": 600000000000,\n\t\"PodLabelWaitTimeout\": 600000000000,\n\t\"PodDeletionWaitTimeout\": 600000000000,\n\t\"SpiloFSGroup\": null,\n\t\"PodPriorityClassName\": \"\",\n\t\"ClusterDomain\": \"cluster.local\",\n\t\"SpiloPrivileged\": false,\n\t\"ClusterLabels\": {\n\t\t\"application\": \"spilo\"\n\t},\n\t\"InheritedLabels\": null,\n\t\"ClusterNameLabel\": \"cluster-name\",\n\t\"PodRoleLabel\": \"spilo-role\",\n\t\"PodToleration\": null,\n\t\"DefaultCPURequest\": \"100m\",\n\t\"DefaultMemoryRequest\": \"100Mi\",\n\t\"DefaultCPULimit\": \"1\",\n\t\"DefaultMemoryLimit\": \"500Mi\",\n\t\"MinCPULimit\": \"250m\",\n\t\"MinMemoryLimit\": \"250Mi\",\n\t\"PodEnvironmentConfigMap\": \"/\",\n\t\"NodeReadinessLabel\": null,\n\t\"MaxInstances\": -1,\n\t\"MinInstances\": -1,\n\t\"ShmVolume\": true,\n\t\"SecretNameTemplate\": \"{username}.{cluster}.credentials\",\n\t\"PamRoleName\": \"zalandos\",\n\t\"PamConfiguration\": \"https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees\",\n\t\"TeamsAPIUrl\": \"https://teams.example.com/api/\",\n\t\"OAuthTokenSecretName\": \"default/postgresql-operator\",\n\t\"InfrastructureRolesSecretName\": \"/\",\n\t\"SuperUsername\": \"postgres\",\n\t\"ReplicationUsername\": \"standby\",\n\t\"ScalyrAPIKey\": \"\",\n\t\"ScalyrImage\": \"\",\n\t\"ScalyrServerURL\": \"https://upload.eu.scalyr.com\",\n\t\"ScalyrCPURequest\": \"100m\",\n\t\"ScalyrMemoryRequest\": \"50Mi\",\n\t\"ScalyrCPULimit\": \"1\",\n\t\"ScalyrMemoryLimit\": \"500Mi\",\n\t\"LogicalBackupSchedule\": \"30 00 * * *\",\n\t\"LogicalBackupDockerImage\": \"registry.opensource.zalan.do/acid/logical-backup\",\n\t\"LogicalBackupS3Bucket\": \"\",\n\t\"LogicalBackupS3Region\": \"\",\n\t\"LogicalBackupS3Endpoint\": \"\",\n\t\"LogicalBackupS3AccessKeyID\": \"\",\n\t\"LogicalBackupS3SecretAccessKey\": \"\",\n\t\"LogicalBackupS3SSE\": \"AES256\",\n\t\"NumberOfInstances\": 2,\n\t\"Schema\": \"postgres\",\n\t\"User\": \"pooler\",\n\t\"Image\": \"registry.opensource.zalan.do/acid/pgbouncer:master-6\",\n\t\"Mode\": \"transaction\",\n\t\"MaxDBConnections\": 100,\n\t\"ConnectionPoolerDefaultCPURequest\": \"500m\",\n\t\"ConnectionPoolerDefaultMemoryRequest\": \"100Mi\",\n\t\"ConnectionPoolerDefaultCPULimit\": \"1\",\n\t\"ConnectionPoolerDefaultMemoryLimit\": \"100Mi\",\n\t\"WatchedNamespace\": \"\",\n\t\"KubernetesUseConfigMaps\": false,\n\t\"EtcdHost\": \"\",\n\t\"DockerImage\": \"registry.opensource.zalan.do/acid/spilo-12:1.6-p2\",\n\t\"Sidecars\": null,\n\t\"PodServiceAccountName\": \"postgres-pod\",\n\t\"PodServiceAccountDefinition\": \"\\n\\t\\t{\\n\\t\\t\\t\\\"apiVersion\\\": \\\"v1\\\",\\n\\t\\t\\t\\\"kind\\\": \\\"ServiceAccount\\\",\\n\\t\\t\\t\\\"metadata\\\": {\\n\\t\\t\\t\\t\\\"name\\\": \\\"postgres-pod\\\"\\n\\t\\t\\t}\\n\\t\\t}\",\n\t\"PodServiceAccountRoleBindingDefinition\": \"\\n\\t\\t{\\n\\t\\t\\t\\\"apiVersion\\\": \\\"rbac.authorization.k8s.io/v1\\\",\\n\\t\\t\\t\\\"kind\\\": \\\"RoleBinding\\\",\\n\\t\\t\\t\\\"metadata\\\": {\\n\\t\\t\\t\\t   \\\"name\\\": \\\"postgres-pod\\\"\\n\\t\\t\\t},\\n\\t\\t\\t\\\"roleRef\\\": {\\n\\t\\t\\t\\t\\\"apiGroup\\\": \\\"rbac.authorization.k8s.io\\\",\\n\\t\\t\\t\\t\\\"kind\\\": \\\"ClusterRole\\\",\\n\\t\\t\\t\\t\\\"name\\\": \\\"postgres-pod\\\"\\n\\t\\t\\t},\\n\\t\\t\\t\\\"subjects\\\": [\\n\\t\\t\\t\\t{\\n\\t\\t\\t\\t\\t\\\"kind\\\": \\\"ServiceAccount\\\",\\n\\t\\t\\t\\t\\t\\\"name\\\": \\\"postgres-pod\\\"\\n\\t\\t\\t\\t}\\n\\t\\t\\t]\\n\\t\\t}\",\n\t\"MasterPodMoveTimeout\": 1200000000000,\n\t\"DbHostedZone\": \"db.example.com\",\n\t\"AWSRegion\": \"eu-central-1\",\n\t\"WALES3Bucket\": \"\",\n\t\"LogS3Bucket\": \"\",\n\t\"KubeIAMRole\": \"\",\n\t\"AdditionalSecretMount\": \"\",\n\t\"AdditionalSecretMountPath\": \"/meta/credentials\",\n\t\"DebugLogging\": true,\n\t\"EnableDBAccess\": true,\n\t\"EnableTeamsAPI\": false,\n\t\"EnableTeamSuperuser\": false,\n\t\"TeamAdminRole\": \"admin\",\n\t\"EnableAdminRoleForUsers\": true,\n\t\"EnableMasterLoadBalancer\": false,\n\t\"EnableReplicaLoadBalancer\": false,\n\t\"CustomServiceAnnotations\": null,\n\t\"CustomPodAnnotations\": null,\n\t\"EnablePodAntiAffinity\": false,\n\t\"PodAntiAffinityTopologyKey\": \"kubernetes.io/hostname\",\n\t\"EnableLoadBalancer\": null,\n\t\"MasterDNSNameFormat\": \"{cluster}.{team}.{hostedzone}\",\n\t\"ReplicaDNSNameFormat\": \"{cluster}-repl.{team}.{hostedzone}\",\n\t\"PDBNameFormat\": \"postgres-{cluster}-pdb\",\n\t\"EnablePodDisruptionBudget\": true,\n\t\"EnableInitContainers\": true,\n\t\"EnableSidecars\": true,\n\t\"Workers\": 4,\n\t\"APIPort\": 8080,\n\t\"RingLogLines\": 100,\n\t\"ClusterHistoryEntries\": 1000,\n\t\"TeamAPIRoleConfiguration\": {\n\t\t\"log_statement\": \"all\"\n\t},\n\t\"PodTerminateGracePeriod\": 300000000000,\n\t\"PodManagementPolicy\": \"ordered_ready\",\n\t\"ProtectedRoles\": [\n\t\t\"admin\"\n\t],\n\t\"PostgresSuperuserTeams\": null,\n\t\"SetMemoryRequestToLimit\": false\n}" pkg=controller
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:30Z" level=debug msg="acquiring initial list of clusters" pkg=controller
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:30Z" level=info msg="no clusters running" pkg=controller
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:30Z" level=info msg="started working in background" pkg=controller
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:30Z" level=info msg="listening on :8080" pkg=apiserver
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:30Z" level=debug msg="new node has been added: \"/devops-m1\" ()" pkg=controller
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:30Z" level=debug msg="new node has been added: \"/devops-w1\" ()" pkg=controller
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:30Z" level=debug msg="new node has been added: \"/devops-w2\" ()" pkg=controller
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:30Z" level=debug msg="new node has been added: \"/devops-w3\" ()" pkg=controller
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:35Z" level=info msg="\"ADD\" event has been queued" cluster-name=default/acid-postgres-cluster pkg=controller worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:35Z" level=info msg="creation of the cluster started" cluster-name=default/acid-postgres-cluster pkg=controller worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:35Z" level=info msg="endpoint \"default/acid-postgres-cluster\" has been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:35Z" level=info msg="master service \"default/acid-postgres-cluster\" has been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:35Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:35Z" level=info msg="replica service \"default/acid-postgres-cluster-repl\" has been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:35Z" level=debug msg="team API is disabled, returning empty list of members for team \"acid\"" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:35Z" level=info msg="users have been initialized" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:35Z" level=debug msg="created new secret \"default/pooler.acid-postgres-cluster.credentials\", uid: \"3663e439-12e0-4b80-8a09-55807eb4d1c2\"" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:35Z" level=debug msg="created new secret \"default/postgres.acid-postgres-cluster.credentials\", uid: \"93af71b2-2d92-48d5-be9c-e524db186dbf\"" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:35Z" level=debug msg="created new secret \"default/standby.acid-postgres-cluster.credentials\", uid: \"f16bbb48-6259-4ba5-a884-376391ae09b7\"" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:35Z" level=info msg="secrets have been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:35Z" level=info msg="pod disruption budget \"default/postgres-acid-postgres-cluster-pdb\" has been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:35Z" level=debug msg="Generating Spilo container, environment variables: [{SCOPE acid-postgres-cluster nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-postgres-cluster.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-postgres-cluster.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{\"bin_dir\":\"/usr/lib/postgresql/12/bin\",\"parameters\":{\"log_statement\":\"all\",\"shared_buffers\":\"4GB\",\"tcp_keepalives_count\":\"5\",\"tcp_keepalives_idle\":\"60\",\"tcp_keepalives_interval\":\"60\"},\"pg_hba\":[\"hostssl all all 0.0.0.0/0 md5\",\"host    all all 0.0.0.0/0 md5\"]},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"},\"data-checksums\",{\"encoding\":\"UTF8\"},{\"locale\":\"en_US.UTF-8\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{\"ttl\":30,\"loop_wait\":10,\"retry_timeout\":10,\"maximum_lag_on_failover\":33554432,\"postgresql\":{\"parameters\":{\"max_connections\":\"300\"}}}}} nil} {DCS_ENABLE_KUBERNETES_API true nil}]" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:35Z" level=debug msg="created new statefulset \"default/acid-postgres-cluster\", uid: \"c1b299c8-383e-4d44-ac10-e20fb99101f2\"" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:35Z" level=info msg="statefulset \"default/acid-postgres-cluster\" has been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:52:35Z" level=info msg="waiting for the cluster being ready" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:05Z" level=debug msg="Waiting for 3 pods to become ready" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="pods are ready" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="Create roles" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=debug msg="closing database connection" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="users have been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=debug msg="closing database connection" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="databases have been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="found pod disruption budget: \"default/postgres-acid-postgres-cluster-pdb\" (uid: \"e323ea22-7d5d-43c1-844e-3a1f82dde913\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="found statefulset: \"default/acid-postgres-cluster\" (uid: \"c1b299c8-383e-4d44-ac10-e20fb99101f2\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="found secret: \"default/pooler.acid-postgres-cluster.credentials\" (uid: \"3663e439-12e0-4b80-8a09-55807eb4d1c2\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="found secret: \"default/postgres.acid-postgres-cluster.credentials\" (uid: \"93af71b2-2d92-48d5-be9c-e524db186dbf\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="found secret: \"default/standby.acid-postgres-cluster.credentials\" (uid: \"f16bbb48-6259-4ba5-a884-376391ae09b7\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="found master endpoint: \"default/acid-postgres-cluster\" (uid: \"8a434859-cadd-454d-84f4-e10ea7acbc43\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="found master service: \"default/acid-postgres-cluster\" (uid: \"dbf2a088-8914-49b5-b0be-5c9ff42654b2\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="found replica service: \"default/acid-postgres-cluster-repl\" (uid: \"529bb80c-70cd-4840-99e9-00409dbc5865\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="found pod: \"default/acid-postgres-cluster-0\" (uid: \"933e53f0-1fdf-419f-9e26-1f17c29b5ccb\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="found pod: \"default/acid-postgres-cluster-1\" (uid: \"a061799e-b510-43bd-a575-442a2f5e5c11\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="found pod: \"default/acid-postgres-cluster-2\" (uid: \"d5f2ce9a-a0db-4693-8967-0959c59af11a\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="found PVC: \"default/pgdata-acid-postgres-cluster-0\" (uid: \"c4bf4d0c-71a2-47de-aa4d-0c5b8672cd40\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="found PVC: \"default/pgdata-acid-postgres-cluster-1\" (uid: \"8fffd715-7065-4a60-bee2-c14081746092\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="found PVC: \"default/pgdata-acid-postgres-cluster-2\" (uid: \"0a5001c9-7d36-444f-b83d-fc8e86550525\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="Installing lookup function" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="Install pooler lookup function into postgres" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=info msg="pooler lookup function installed into postgres" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator time="2020-04-06T07:53:44Z" level=debug msg="closing database connection" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-zd892 postgres-operator panic: runtime error: invalid memory address or nil pointer dereference
postgres-operator-7857d8956b-zd892 postgres-operator [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x16a995e]
postgres-operator-7857d8956b-zd892 postgres-operator
postgres-operator-7857d8956b-zd892 postgres-operator goroutine 65 [running]:
postgres-operator-7857d8956b-zd892 postgres-operator github.com/zalando/postgres-operator/pkg/cluster.(*Cluster).installLookupFunction(0xc0007c2c00, 0xc00018f9a0, 0x8, 0x179b13e, 0x6, 0x0, 0x0)
postgres-operator-7857d8956b-zd892 postgres-operator    /workspace/pkg/cluster/database.go:362 +0x9fe
postgres-operator-7857d8956b-zd892 postgres-operator github.com/zalando/postgres-operator/pkg/cluster.(*Cluster).createConnectionPooler(0xc0007c2c00, 0xc000ce3220, 0x0, 0x28, 0x0)
postgres-operator-7857d8956b-zd892 postgres-operator    /workspace/pkg/cluster/resources.go:118 +0xd2
postgres-operator-7857d8956b-zd892 postgres-operator github.com/zalando/postgres-operator/pkg/cluster.(*Cluster).Create(0xc0007c2c00, 0x0, 0x0)
postgres-operator-7857d8956b-zd892 postgres-operator    /workspace/pkg/cluster/cluster.go:351 +0xcdc
postgres-operator-7857d8956b-zd892 postgres-operator github.com/zalando/postgres-operator/pkg/controller.(*Controller).processEvent(0xc00013e000, 0xbf9ad5d0ca666207, 0x20dad6c47, 0x2c95280, 0xc0009872f0, 0x24, 0x1b5bc81, 0x3, 0x0, 0xc0009a2800, ...)
postgres-operator-7857d8956b-zd892 postgres-operator    /workspace/pkg/controller/postgresql.go:236 +0x72b
postgres-operator-7857d8956b-zd892 postgres-operator github.com/zalando/postgres-operator/pkg/controller.(*Controller).processClusterEventsQueue(0xc00013e000, 0x0, 0xc000190c00, 0xc000048040)
postgres-operator-7857d8956b-zd892 postgres-operator    /workspace/pkg/controller/postgresql.go:338 +0x260
postgres-operator-7857d8956b-zd892 postgres-operator created by github.com/zalando/postgres-operator/pkg/controller.(*Controller).Run
postgres-operator-7857d8956b-zd892 postgres-operator    /workspace/pkg/controller/controller.go:359 +0x8a

FxKu · 2020-04-06T16:28:16Z

@haroonb when you do not specify the user (e.g. when it's commented) a new user and secret will be created and that user is called pooler by an internal default. Or have you configured it to be postgres in the configmap? If you want to reuse the postgres user simply remove the #.

Nevertheless, the go error doesn't look good. Will have a look.

Edit: I saw that you are talking about connection_pooler_user which is in fact the configuration field. Have you tried to set "postgres" there?

haroonb · 2020-04-07T06:57:26Z

I tried to use postgres as schema and user, if use that setting no poolers are created. I'll append the logs of two other scenarios I ran to test this issue, I ran a lot more test, but I think these two will demonstrate the issue. I hope I could provide useful information.

configmap values:
connection_pooler_schema: "postgres1"
connection_pooler_user: "postgres2"

created secret:
secret/postgres2.acid-postgres-cluster.credentials

admin user in pgbouncer.ini:
admin_users = postgres2


postgres-operator log:
+ postgres-operator-7857d8956b-p69mk › postgres-operator
postgres-operator-7857d8956b-p69mk postgres-operator 2020/04/07 06:26:30 Spilo operator v1.4.0-22-g64389b8-dirty
postgres-operator-7857d8956b-p69mk postgres-operator 2020/04/07 06:26:30 Fully qualified configmap name: default/postgres-operator
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:30Z" level=info msg="Parse role bindings" pkg=controller
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:30Z" level=info msg="successfully parsed" pkg=controller
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:30Z" level=info msg="Listening to all namespaces" pkg=controller
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:30Z" level=info msg="customResourceDefinition \"postgresqls.acid.zalan.do\" has been registered" pkg=controller
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:34Z" level=info msg="config: {\n\t\"ReadyWaitInterval\": 3000000000,\n\t\"ReadyWaitTimeout\": 30000000000,\n\t\"ResyncPeriod\": 1800000000000,\n\t\"RepairPeriod\": 300000000000,\n\t\"EnableCRDValidation\": true,\n\t\"ResourceCheckInterval\": 3000000000,\n\t\"ResourceCheckTimeout\": 600000000000,\n\t\"PodLabelWaitTimeou
t\": 600000000000,\n\t\"PodDeletionWaitTimeout\": 600000000000,\n\t\"SpiloFSGroup\": null,\n\t\"PodPriorityClassName\": \"\",\n\t\"ClusterDomain\": \"cluster.local\",\n\t\"SpiloPrivileged\": false,\n\t\"ClusterLabels\": {\n\t\t\"application\": \"spilo\"\n\t},\n\t\"InheritedLabels\": null,\n\t\"ClusterNameLabel\": \"cluster-name\",\n\t\"PodRoleLabel\": \"spilo-role\",\n\t\"PodToleration\": null
,\n\t\"DefaultCPURequest\": \"100m\",\n\t\"DefaultMemoryRequest\": \"100Mi\",\n\t\"DefaultCPULimit\": \"1\",\n\t\"DefaultMemoryLimit\": \"500Mi\",\n\t\"MinCPULimit\": \"250m\",\n\t\"MinMemoryLimit\": \"250Mi\",\n\t\"PodEnvironmentConfigMap\": \"/\",\n\t\"NodeReadinessLabel\": null,\n\t\"MaxInstances\": -1,\n\t\"MinInstances\": -1,\n\t\"ShmVolume\": true,\n\t\"SecretNameTemplate\": \"{username}
.{cluster}.credentials\",\n\t\"PamRoleName\": \"zalandos\",\n\t\"PamConfiguration\": \"https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees\",\n\t\"TeamsAPIUrl\": \"https://teams.example.com/api/\",\n\t\"OAuthTokenSecretName\": \"default/postgresql-operator\",\n\t\"InfrastructureRolesSecretName\": \"/\",\n\t\"SuperUsername\": \"postgres\",\n\t\"ReplicationUsername\": \"standby\",\n\t\"ScalyrAPIKey\": \"\",\n\t\"ScalyrImage\": \"\",\n\t\"ScalyrServerURL\": \"https://upload.eu.scalyr.com\",\n\t\"ScalyrCPURequest\": \"100m\",\n\t\"ScalyrMemoryRequest\": \"50Mi\",\n\t\"ScalyrCPULimit\": \"1\",\n\t\"ScalyrMemoryLimit\": \"500Mi\",\n\t\"LogicalBackupSchedule\": \"30 00 * * *\",\n\t\"LogicalBackupDockerImage\": \"registry.opensource.zalan.do/acid/logical-backup\",\n\t\"LogicalBackupS3Bucket\": \"\",\n\t\"LogicalBackupS3Region\": \"\",\n\t\"LogicalBackupS3Endpoint\": \"\",\n\t\"LogicalBackupS3AccessKeyID\": \"\",\n\t\"LogicalBackupS3SecretAccessKey\": \"\",\n\t\"LogicalBackupS3SSE\": \"AES256\",\n\t\"NumberOfInstances\": 2,\n\t\"Schema\": \"postgres1\",\n\t\"User\": \"postgres2\",\n\t\"Image\": \"registry.opensource.zalan.do/acid/pgbouncer:master-6\",\n\t\"Mode\": \"transaction\",\n\t\"MaxDBConnections\": 100,\n\t\"ConnectionPoolerDefaultCPURequest\": \"500m\",\n\t\"ConnectionPoolerDefaultMemoryRequest\": \"100Mi\",\n\t\"ConnectionPoolerDefaultCPULimit\": \"1\",\n\t\"ConnectionPoolerDefaultMemoryLimit\": \"100Mi\",\n\t\"WatchedNamespace\": \"\",\n\t\"KubernetesUseConfigMaps\": false,\n\t\"EtcdHost\": \"\",\n\t\"DockerImage\": \"registry.opensource.zalan.do/acid/spilo-12:1.6-p2\",\n\t\"Sidecars\": null,\n\t\"PodServiceAccountName\": \"postgres-pod\",\n\t\"PodServiceAccountDefinition\": \"\\n\\t\\t{\\n\\t\\t\\t\\\"apiVersion\\\": \\\"v1\\\",\\n\\t\\t\\t\\\"kind\\\": \\\"ServiceAccount\\\",\\n\\t\\t\\t\\\"metadata\\\": {\\n\\t\\t\\t\\t\\\"name\\\": \\\"postgres-pod\\\"\\n\\t\\t\\t}\\n\\t\\t}\",\n\t\"PodServiceAccountRoleBindingDefinition\": \"\\n\\t\\t{\\n\\t\\t\\t\\\"apiVersion\\\": \\\"rbac.authorization.k8s.io/v1\\\",\\n\\t\\t\\t\\\"kind\\\": \\\"RoleBinding\\\",\\n\\t\\t\\t\\\"metadata\\\": {\\n\\t\\t\\t\\t   \\\"name\\\": \\\"postgres-pod\\\"\\n\\t\\t\\t},\\n\\t\\t\\t\\\"roleRef\\\": {\\n\\t\\t\\t\\t\\\"apiGroup\\\": \\\"rbac.authorization.k8s.io\\\",\\n\\t\\t\\t\\t\\\"kind\\\": \\\"ClusterRole\\\",\\n\\t\\t\\t\\t\\\"name\\\": \\\"postgres-pod\\\"\\n\\t\\t\\t},\\n\\t\\t\\t\\\"subjects\\\": [\\n\\t\\t\\t\\t{\\n\\t\\t\\t\\t\\t\\\"kind\\\": \\\"ServiceAccount\\\",\\n\\t\\t\\t\\t\\t\\\"name\\\": \\\"postgres-pod\\\"\\n\\t\\t\\t\\t}\\n\\t\\t\\t]\\n\\t\\t}\",\n\t\"MasterPodMoveTimeout\": 1200000000000,\n\t\"DbHostedZone\": \"db.example.com\",\n\t\"AWSRegion\": \"eu-central-1\",\n\t\"WALES3Bucket\": \"\",\n\t\"LogS3Bucket\": \"\",\n\t\"KubeIAMRole\": \"\",\n\t\"AdditionalSecretMount\": \"\",\n\t\"AdditionalSecretMountPath\": \"/meta/credentials\",\n\t\"DebugLogging\": true,\n\t\"EnableDBAccess\": true,\n\t\"EnableTeamsAPI\": false,\n\t\"EnableTeamSuperuser\": false,\n\t\"TeamAdminRole\": \"admin\",\n\t\"EnableAdminRoleForUsers\": true,\n\t\"EnableMasterLoadBalancer\": false,\n\t\"EnableReplicaLoadBalancer\": false,\n\t\"CustomServiceAnnotations\": null,\n\t\"CustomPodAnnotations\": null,\n\t\"EnablePodAntiAffinity\": false,\n\t\"PodAntiAffinityTopologyKey\": \"kubernetes.io/hostname\",\n\t\"EnableLoadBalancer\": null,\n\t\"MasterDNSNameFormat\": \"{cluster}.{team}.{hostedzone}\",\n\t\"ReplicaDNSNameFormat\": \"{cluster}-repl.{team}.{hostedzone}\",\n\t\"PDBNameFormat\": \"postgres-{cluster}-pdb\",\n\t\"EnablePodDisruptionBudget\": true,\n\t\"EnableInitContainers\": true,\n\t\"EnableSidecars\": true,\n\t\"Workers\": 4,\n\t\"APIPort\": 8080,\n\t\"RingLogLines\": 100,\n\t\"ClusterHistoryEntries\": 1000,\n\t\"TeamAPIRoleConfiguration\": {\n\t\t\"log_statement\": \"all\"\n\t},\n\t\"PodTerminateGracePeriod\": 300000000000,\n\t\"PodManagementPolicy\": \"ordered_ready\",\n\t\"ProtectedRoles\": [\n\t\t\"admin\"\n\t],\n\t\"PostgresSuperuserTeams\": null,\n\t\"SetMemoryRequestToLimit\": false\n}" pkg=controller
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:34Z" level=debug msg="acquiring initial list of clusters" pkg=controller
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:34Z" level=info msg="no clusters running" pkg=controller
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:34Z" level=info msg="started working in background" pkg=controller
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:34Z" level=info msg="listening on :8080" pkg=apiserver
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:34Z" level=debug msg="new node has been added: \"/devops-m1\" ()" pkg=controller
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:34Z" level=debug msg="new node has been added: \"/devops-w1\" ()" pkg=controller
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:34Z" level=debug msg="new node has been added: \"/devops-w2\" ()" pkg=controller
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:34Z" level=debug msg="new node has been added: \"/devops-w3\" ()" pkg=controller
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:39Z" level=info msg="\"ADD\" event has been queued" cluster-name=default/acid-postgres-cluster pkg=controller worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:39Z" level=info msg="creation of the cluster started" cluster-name=default/acid-postgres-cluster pkg=controller worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:39Z" level=info msg="endpoint \"default/acid-postgres-cluster\" has been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:39Z" level=info msg="master service \"default/acid-postgres-cluster\" has been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:39Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:39Z" level=info msg="replica service \"default/acid-postgres-cluster-repl\" has been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:39Z" level=debug msg="team API is disabled, returning empty list of members for team \"acid\"" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:39Z" level=info msg="users have been initialized" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:39Z" level=debug msg="created new secret \"default/postgres.acid-postgres-cluster.credentials\", uid: \"ee8b4216-c031-4841-a88b-3c8ba9f40249\"" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:39Z" level=debug msg="created new secret \"default/standby.acid-postgres-cluster.credentials\", uid: \"173ec9a2-5506-4564-8fc6-f11f705ee651\"" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:39Z" level=debug msg="created new secret \"default/postgres2.acid-postgres-cluster.credentials\", uid: \"e1bf6b8e-b793-49c9-99c2-0445a3ea9222\"" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:39Z" level=info msg="secrets have been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:39Z" level=info msg="pod disruption budget \"default/postgres-acid-postgres-cluster-pdb\" has been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:39Z" level=debug msg="Generating Spilo container, environment variables: [{SCOPE acid-postgres-cluster nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-postgres-cluster.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-postgres-cluster.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2
https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{\"bin_dir\":\"/usr/lib/postgresql/12/bin\",\"parameters\":{\"log_statement\":\"all\",\"shared_buffers\":\"4GB\",\"tcp_keepalives_count\":\"5\",\"tcp_keepalives_idle\":\"60\",\"tcp_keepalives_interval\":\"60\"},\"pg_hba\":[\"hostssl all all 0.0.0.0/0 md5\",\"host    all all 0.0.0.0/0 md5\"]},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"},\"data-checksums\",{\"encoding\":\"UTF8\"},{\"locale\":\"en_US.UTF-8\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{\"ttl\":30,\"loop_wait\":10,\"retry_timeout\":10,\"maximum_lag_on_failover\":33554432,\"postgresql\":{\"parameters\":{\"max_connections\":\"300\"}}}}} nil} {DCS_ENABLE_KUBERNETES_API true nil}]" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:39Z" level=debug msg="created new statefulset \"default/acid-postgres-cluster\", uid: \"d92af24d-f6d5-4afa-b700-79768b87f963\"" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:39Z" level=info msg="statefulset \"default/acid-postgres-cluster\" has been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:26:39Z" level=info msg="waiting for the cluster being ready" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:15Z" level=debug msg="Waiting for 3 pods to become ready" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="pods are ready" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="Create roles" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=debug msg="closing database connection" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="users have been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=debug msg="closing database connection" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="databases have been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="found pod disruption budget: \"default/postgres-acid-postgres-cluster-pdb\" (uid: \"f5a75c49-15d9-45ee-9474-30eae6eb0705\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="found statefulset: \"default/acid-postgres-cluster\" (uid: \"d92af24d-f6d5-4afa-b700-79768b87f963\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="found secret: \"default/postgres.acid-postgres-cluster.credentials\" (uid: \"ee8b4216-c031-4841-a88b-3c8ba9f40249\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="found secret: \"default/standby.acid-postgres-cluster.credentials\" (uid: \"173ec9a2-5506-4564-8fc6-f11f705ee651\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="found secret: \"default/postgres2.acid-postgres-cluster.credentials\" (uid: \"e1bf6b8e-b793-49c9-99c2-0445a3ea9222\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="found master endpoint: \"default/acid-postgres-cluster\" (uid: \"de77444d-04aa-41c3-8c8b-759a7b58d1ba\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="found replica service: \"default/acid-postgres-cluster-repl\" (uid: \"099dda1c-5224-4a38-ad00-3ffe091e261d\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="found master service: \"default/acid-postgres-cluster\" (uid: \"cbcbeb2a-b8e8-42f2-8e90-5e1c56ecdf99\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="found pod: \"default/acid-postgres-cluster-0\" (uid: \"d8129412-21f9-4e3b-b98a-48ad6a9cef00\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="found pod: \"default/acid-postgres-cluster-1\" (uid: \"c38e6de7-2fa8-4e5a-861f-55ded01a7e52\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="found pod: \"default/acid-postgres-cluster-2\" (uid: \"cc145527-cbb7-45f4-a415-063dabf52a16\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="found PVC: \"default/pgdata-acid-postgres-cluster-0\" (uid: \"a74056e1-67a1-4ab3-a811-cb4190c1d58b\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="found PVC: \"default/pgdata-acid-postgres-cluster-1\" (uid: \"f43aa9b8-d4a2-4c5c-93c3-fce1856f0ab5\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="found PVC: \"default/pgdata-acid-postgres-cluster-2\" (uid: \"9541dc4a-6b56-4e7f-8b07-df5e2039b691\")" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="Installing lookup function" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="Install pooler lookup function into postgres" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=info msg="pooler lookup function installed into postgres" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator time="2020-04-07T06:27:57Z" level=debug msg="closing database connection" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-p69mk postgres-operator panic: runtime error: invalid memory address or nil pointer dereference
postgres-operator-7857d8956b-p69mk postgres-operator [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x16a995e]
postgres-operator-7857d8956b-p69mk postgres-operator
postgres-operator-7857d8956b-p69mk postgres-operator goroutine 62 [running]:
postgres-operator-7857d8956b-p69mk postgres-operator github.com/zalando/postgres-operator/pkg/cluster.(*Cluster).installLookupFunction(0xc0000d6c00, 0xc000533330, 0x9, 0xc000533340, 0x9, 0x0, 0x0)
postgres-operator-7857d8956b-p69mk postgres-operator    /workspace/pkg/cluster/database.go:362 +0x9fe
postgres-operator-7857d8956b-p69mk postgres-operator github.com/zalando/postgres-operator/pkg/cluster.(*Cluster).createConnectionPooler(0xc0000d6c00, 0xc00111f220, 0x0, 0x28, 0x0)
postgres-operator-7857d8956b-p69mk postgres-operator    /workspace/pkg/cluster/resources.go:118 +0xd2
postgres-operator-7857d8956b-p69mk postgres-operator github.com/zalando/postgres-operator/pkg/cluster.(*Cluster).Create(0xc0000d6c00, 0x0, 0x0)
postgres-operator-7857d8956b-p69mk postgres-operator    /workspace/pkg/cluster/cluster.go:351 +0xcdc
postgres-operator-7857d8956b-p69mk postgres-operator github.com/zalando/postgres-operator/pkg/controller.(*Controller).processEvent(0xc0003c4000, 0xbf9b2527c09e1bf0, 0x203516170, 0x2c95280, 0xc000ad7f50, 0x24, 0x1b5bc81, 0x3, 0x0, 0xc000c7e800, ...)
postgres-operator-7857d8956b-p69mk postgres-operator    /workspace/pkg/controller/postgresql.go:236 +0x72b
postgres-operator-7857d8956b-p69mk postgres-operator github.com/zalando/postgres-operator/pkg/controller.(*Controller).processClusterEventsQueue(0xc0003c4000, 0x0, 0xc0003286c0, 0xc00038e010)
postgres-operator-7857d8956b-p69mk postgres-operator    /workspace/pkg/controller/postgresql.go:338 +0x260
postgres-operator-7857d8956b-p69mk postgres-operator created by github.com/zalando/postgres-operator/pkg/controller.(*Controller).Run
postgres-operator-7857d8956b-p69mk postgres-operator    /workspace/pkg/controller/controller.go:359 +0x8a

configmap values:
  connection_pooler_schema: "postgres"
  connection_pooler_user: "postgres"

postgres-operator log:
+ postgres-operator-7857d8956b-pz74g › postgres-operator
postgres-operator-7857d8956b-pz74g postgres-operator 2020/04/07 06:45:40 Fully qualified configmap name: default/postgres-operator
postgres-operator-7857d8956b-pz74g postgres-operator 2020/04/07 06:45:40 Spilo operator v1.4.0-22-g64389b8-dirty
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:40Z" level=info msg="Parse role bindings" pkg=controller
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:40Z" level=info msg="successfully parsed" pkg=controller
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:40Z" level=info msg="Listening to all namespaces" pkg=controller
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:40Z" level=info msg="customResourceDefinition \"postgresqls.acid.zalan.do\" has been registered" pkg=controller
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:44Z" level=info msg="config: {\n\t\"ReadyWaitInterval\": 3000000000,\n\t\"ReadyWaitTimeout\": 30000000000,\n\t\"ResyncPeriod\": 1800000000000,\n\t\"RepairPeriod\": 300000000000,\n\t\"EnableCRDValidation\": true,\n\t\"ResourceCheckInterval\": 3000000000,\n\t\"ResourceCheckTimeout\": 600000000000,\n\t\"PodLabelWaitTimeout\": 600000000000,\n\t\"PodDeletionWaitTimeout\": 600000000000,\n\t\"SpiloFSGroup\": null,\n\t\"PodPriorityClassName\": \"\",\n\t\"ClusterDomain\": \"cluster.local\",\n\t\"SpiloPrivileged\": false,\n\t\"ClusterLabels\": {\n\t\t\"application\": \"spilo\"\n\t},\n\t\"InheritedLabels\": null,\n\t\"ClusterNameLabel\": \"cluster-name\",\n\t\"PodRoleLabel\": \"spilo-role\",\n\t\"PodToleration\": null,\n\t\"DefaultCPURequest\": \"100m\",\n\t\"DefaultMemoryRequest\": \"100Mi\",\n\t\"DefaultCPULimit\": \"1\",\n\t\"DefaultMemoryLimit\": \"500Mi\",\n\t\"MinCPULimit\": \"250m\",\n\t\"MinMemoryLimit\": \"250Mi\",\n\t\"PodEnvironmentConfigMap\": \"/\",\n\t\"NodeReadinessLabel\": null,\n\t\"MaxInstances\": -1,\n\t\"MinInstances\": -1,\n\t\"ShmVolume\": true,\n\t\"SecretNameTemplate\": \"{username}.{cluster}.credentials\",\n\t\"PamRoleName\": \"zalandos\",\n\t\"PamConfiguration\": \"https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees\",\n\t\"TeamsAPIUrl\": \"https://teams.example.com/api/\",\n\t\"OAuthTokenSecretName\": \"default/postgresql-operator\",\n\t\"InfrastructureRolesSecretName\": \"/\",\n\t\"SuperUsername\": \"postgres\",\n\t\"ReplicationUsername\": \"standby\",\n\t\"ScalyrAPIKey\": \"\",\n\t\"ScalyrImage\": \"\",\n\t\"ScalyrServerURL\": \"https://upload.eu.scalyr.com\",\n\t\"ScalyrCPURequest\": \"100m\",\n\t\"ScalyrMemoryRequest\": \"50Mi\",\n\t\"ScalyrCPULimit\": \"1\",\n\t\"ScalyrMemoryLimit\": \"500Mi\",\n\t\"LogicalBackupSchedule\": \"30 00 * * *\",\n\t\"LogicalBackupDockerImage\": \"registry.opensource.zalan.do/acid/logical-backup\",\n\t\"LogicalBackupS3Bucket\": \"\",\n\t\"LogicalBackupS3Region\": \"\",\n\t\"LogicalBackupS3Endpoint\": \"\",\n\t\"LogicalBackupS3AccessKeyID\": \"\",\n\t\"LogicalBackupS3SecretAccessKey\": \"\",\n\t\"LogicalBackupS3SSE\": \"AES256\",\n\t\"NumberOfInstances\": 2,\n\t\"Schema\": \"postgres\",\n\t\"User\": \"postgres\",\n\t\"Image\": \"registry.opensource.zalan.do/acid/pgbouncer:master-6\",\n\t\"Mode\": \"transaction\",\n\t\"MaxDBConnections\": 100,\n\t\"ConnectionPoolerDefaultCPURequest\": \"500m\",\n\t\"ConnectionPoolerDefaultMemoryRequest\": \"100Mi\",\n\t\"ConnectionPoolerDefaultCPULimit\": \"1\",\n\t\"ConnectionPoolerDefaultMemoryLimit\": \"100Mi\",\n\t\"WatchedNamespace\": \"\",\n\t\"KubernetesUseConfigMaps\": false,\n\t\"EtcdHost\": \"\",\n\t\"DockerImage\": \"registry.opensource.zalan.do/acid/spilo-12:1.6-p2\",\n\t\"Sidecars\": null,\n\t\"PodServiceAccountName\": \"postgres-pod\",\n\t\"PodServiceAccountDefinition\": \"\\n\\t\\t{\\n\\t\\t\\t\\\"apiVersion\\\": \\\"v1\\\",\\n\\t\\t\\t\\\"kind\\\": \\\"ServiceAccount\\\",\\n\\t\\t\\t\\\"metadata\\\": {\\n\\t\\t\\t\\t\\\"name\\\": \\\"postgres-pod\\\"\\n\\t\\t\\t}\\n\\t\\t}\",\n\t\"PodServiceAccountRoleBindingDefinition\": \"\\n\\t\\t{\\n\\t\\t\\t\\\"apiVersion\\\": \\\"rbac.authorization.k8s.io/v1\\\",\\n\\t\\t\\t\\\"kind\\\": \\\"RoleBinding\\\",\\n\\t\\t\\t\\\"metadata\\\": {\\n\\t\\t\\t\\t   \\\"name\\\": \\\"postgres-pod\\\"\\n\\t\\t\\t},\\n\\t\\t\\t\\\"roleRef\\\": {\\n\\t\\t\\t\\t\\\"apiGroup\\\": \\\"rbac.authorization.k8s.io\\\",\\n\\t\\t\\t\\t\\\"kind\\\": \\\"ClusterRole\\\",\\n\\t\\t\\t\\t\\\"name\\\": \\\"postgres-pod\\\"\\n\\t\\t\\t},\\n\\t\\t\\t\\\"subjects\\\": [\\n\\t\\t\\t\\t{\\n\\t\\t\\t\\t\\t\\\"kind\\\": \\\"ServiceAccount\\\",\\n\\t\\t\\t\\t\\t\\\"name\\\": \\\"postgres-pod\\\"\\n\\t\\t\\t\\t}\\n\\t\\t\\t]\\n\\t\\t}\",\n\t\"MasterPodMoveTimeout\": 1200000000000,\n\t\"DbHostedZone\": \"db.example.com\",\n\t\"AWSRegion\": \"eu-central-1\",\n\t\"WALES3Bucket\": \"\",\n\t\"LogS3Bucket\": \"\",\n\t\"KubeIAMRole\": \"\",\n\t\"AdditionalSecretMount\": \"\",\n\t\"AdditionalSecretMountPath\": \"/meta/credentials\",\n\t\"DebugLogging\": true,\n\t\"EnableDBAccess\": true,\n\t\"EnableTeamsAPI\": false,\n\t\"EnableTeamSuperuser\": false,\n\t\"TeamAdminRole\": \"admin\",\n\t\"EnableAdminRoleForUsers\": true,\n\t\"EnableMasterLoadBalancer\": false,\n\t\"EnableReplicaLoadBalancer\": false,\n\t\"CustomServiceAnnotations\": null,\n\t\"CustomPodAnnotations\": null,\n\t\"EnablePodAntiAffinity\": false,\n\t\"PodAntiAffinityTopologyKey\": \"kubernetes.io/hostname\",\n\t\"EnableLoadBalancer\": null,\n\t\"MasterDNSNameFormat\": \"{cluster}.{team}.{hostedzone}\",\n\t\"ReplicaDNSNameFormat\": \"{cluster}-repl.{team}.{hostedzone}\",\n\t\"PDBNameFormat\": \"postgres-{cluster}-pdb\",\n\t\"EnablePodDisruptionBudget\": true,\n\t\"EnableInitContainers\": true,\n\t\"EnableSidecars\": true,\n\t\"Workers\": 4,\n\t\"APIPort\": 8080,\n\t\"RingLogLines\": 100,\n\t\"ClusterHistoryEntries\": 1000,\n\t\"TeamAPIRoleConfiguration\": {\n\t\t\"log_statement\": \"all\"\n\t},\n\t\"PodTerminateGracePeriod\": 300000000000,\n\t\"PodManagementPolicy\": \"ordered_ready\",\n\t\"ProtectedRoles\": [\n\t\t\"admin\"\n\t],\n\t\"PostgresSuperuserTeams\": null,\n\t\"SetMemoryRequestToLimit\": false\n}" pkg=controller
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:44Z" level=debug msg="acquiring initial list of clusters" pkg=controller
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:44Z" level=info msg="no clusters running" pkg=controller
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:44Z" level=info msg="started working in background" pkg=controller
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:44Z" level=info msg="listening on :8080" pkg=apiserver
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:44Z" level=debug msg="new node has been added: \"/devops-w2\" ()" pkg=controller
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:44Z" level=debug msg="new node has been added: \"/devops-w3\" ()" pkg=controller
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:44Z" level=debug msg="new node has been added: \"/devops-m1\" ()" pkg=controller
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:44Z" level=debug msg="new node has been added: \"/devops-w1\" ()" pkg=controller
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:49Z" level=info msg="\"ADD\" event has been queued" cluster-name=default/acid-postgres-cluster pkg=controller worker=0
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:49Z" level=info msg="creation of the cluster started" cluster-name=default/acid-postgres-cluster pkg=controller worker=0
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:49Z" level=info msg="endpoint \"default/acid-postgres-cluster\" has been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:49Z" level=info msg="master service \"default/acid-postgres-cluster\" has been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:49Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:49Z" level=info msg="replica service \"default/acid-postgres-cluster-repl\" has been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:49Z" level=debug msg="team API is disabled, returning empty list of members for team \"acid\"" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:49Z" level=info msg="users have been initialized" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:49Z" level=debug msg="created new secret \"default/standby.acid-postgres-cluster.credentials\", uid: \"83e3c4bc-77e6-4d58-adf3-90d3a70438de\"" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:49Z" level=debug msg="created new secret \"default/postgres.acid-postgres-cluster.credentials\", uid: \"b3d7bc6c-f4cb-4e3e-8a5a-08db987a4b75\"" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:49Z" level=info msg="secrets have been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:49Z" level=info msg="pod disruption budget \"default/postgres-acid-postgres-cluster-pdb\" has been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:49Z" level=debug msg="Generating Spilo container, environment variables: [{SCOPE acid-postgres-cluster nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-postgres-cluster.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-postgres-cluster.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2
https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{\"bin_dir\":\"/usr/lib/postgresql/12/bin\",\"parameters\":{\"log_statement\":\"all\",\"shared_buffers\":\"4GB\",\"tcp_keepalives_count\":\"5\",\"tcp_keepalives_idle\":\"60\",\"tcp_keepalives_interval\":\"60\"},\"pg_hba\":[\"hostssl all all 0.0.0.0/0 md5\",\"host    all all 0.0.0.0/0 md5\"]},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"},\"data-checksums\",{\"encoding\":\"UTF8\"},{\"locale\":\"en_US.UTF-8\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{\"ttl\":30,\"loop_wait\":10,\"retry_timeout\":10,\"maximum_lag_on_failover\":33554432,\"postgresql\":{\"parameters\":{\"max_connections\":\"300\"}}}}} nil} {DCS_ENABLE_KUBERNETES_API true nil}]" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:49Z" level=debug msg="created new statefulset \"default/acid-postgres-cluster\", uid: \"6276301c-07a6-4026-8487-1a5c9ecd0908\"" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:49Z" level=info msg="statefulset \"default/acid-postgres-cluster\" has been successfully created" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:45:49Z" level=info msg="waiting for the cluster being ready" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:46:46Z" level=debug msg="Waiting for 3 pods to become ready" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:47:25Z" level=info msg="pods are ready" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:47:25Z" level=info msg="Create roles" cluster-name=default/acid-postgres-cluster pkg=cluster worker=0
postgres-operator-7857d8956b-pz74g postgres-operator time="2020-04-07T06:47:25Z" level=error msg="could not create cluster: could not create users: could not init db connection: could not init db connection: pq: password authentication failed for user \"postgres\"" cluster-name=default/acid-postgres-cluster pkg=controller worker=0

FxKu · 2020-04-07T07:23:01Z

@haroonb how does your pg manifest currently look like? Did you reflect the renaming to pooler and use numberOfInstances and not with underscores?

Regarding the first setup, it is expected that such a secret is generated, because a new user postgres2 is created, too. For the latter experiment, I wonder if one can use system users. I'll check that.

haroonb · 2020-04-07T07:48:56Z

You can use system users with pgbouncer. I used postgres to login with another version of pgbouncer already, so I would assume it works.

These are the .yamls I used in the order configmap.yaml operator-service-account-rbac.yaml postgres-operator.yaml api-service.yaml complete-postgres-manifest.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-operator
data:
  # additional_secret_mount: "some-secret-name"
  # additional_secret_mount_path: "/some/dir"
  api_port: "8080"
  aws_region: eu-central-1
  cluster_domain: cluster.local
  cluster_history_entries: "1000"
  cluster_labels: application:spilo
  cluster_name_label: cluster-name
  connection_pooler_default_cpu_limit: "1"
  connection_pooler_default_cpu_request: "500m"
  connection_pooler_default_memory_limit: 100Mi
  connection_pooler_default_memory_request: 100Mi
  connection_pooler_image: "registry.opensource.zalan.do/acid/pgbouncer:master-6"
  connection_pooler_max_db_connections: "100"
  connection_pooler_mode: "transaction"
  connection_pooler_number_of_instances: "2"
  connection_pooler_schema: "postgres"
  connection_pooler_user: "postgres"
  # custom_service_annotations: "keyx:valuez,keya:valuea"
  # custom_pod_annotations: "keya:valuea,keyb:valueb"
  db_hosted_zone: db.example.com
  debug_logging: "true"
  # default_cpu_limit: "1"
  # default_cpu_request: 100m
  # default_memory_limit: 500Mi
  # default_memory_request: 100Mi
  docker_image: registry.opensource.zalan.do/acid/spilo-12:1.6-p2
  # enable_admin_role_for_users: "true"
  # enable_crd_validation: "true"
  # enable_database_access: "true"
  # enable_init_containers: "true"
  enable_master_load_balancer: "false"
  # enable_pod_antiaffinity: "false"
  # enable_pod_disruption_budget: "true"
  enable_replica_load_balancer: "false"
  # enable_shm_volume: "true"
  # enable_sidecars: "true"
  # enable_team_superuser: "false"
  enable_teams_api: "false"
  # etcd_host: ""
  # kubernetes_use_configmaps: "false"
  # infrastructure_roles_secret_name: postgresql-infrastructure-roles
  # inherited_labels: application,environment
  # kube_iam_role: ""
  # log_s3_bucket: ""
  # logical_backup_docker_image: "registry.opensource.zalan.do/acid/logical-backup"
  # logical_backup_s3_access_key_id: ""
  # logical_backup_s3_bucket: "my-bucket-url"
  # logical_backup_s3_region: ""
  # logical_backup_s3_endpoint: ""
  # logical_backup_s3_secret_access_key: ""
  # logical_backup_s3_sse: "AES256"
  # logical_backup_schedule: "30 00 * * *"
  master_dns_name_format: "{cluster}.{team}.{hostedzone}"
  # master_pod_move_timeout: 10m
  # max_instances: "-1"
  # min_instances: "-1"
  # min_cpu_limit: 250m
  # min_memory_limit: 250Mi
  # node_readiness_label: ""
  # oauth_token_secret_name: postgresql-operator
  # pam_configuration: |
  #  https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees
  # pam_role_name: zalandos
  pdb_name_format: "postgres-{cluster}-pdb"
  # pod_antiaffinity_topology_key: "kubernetes.io/hostname"
  pod_deletion_wait_timeout: 10m
  # pod_environment_configmap: "default/my-custom-config"
  pod_label_wait_timeout: 10m
  pod_management_policy: "ordered_ready"
  pod_role_label: spilo-role
  # pod_service_account_definition: ""
  pod_service_account_name: "postgres-pod"
  # pod_service_account_role_binding_definition: ""
  pod_terminate_grace_period: 5m
  # postgres_superuser_teams: "postgres_superusers"
  # protected_role_names: "admin"
  ready_wait_interval: 3s
  ready_wait_timeout: 30s
  repair_period: 5m
  replica_dns_name_format: "{cluster}-repl.{team}.{hostedzone}"
  replication_username: standby
  resource_check_interval: 3s
  resource_check_timeout: 10m
  resync_period: 30m
  ring_log_lines: "100"
  secret_name_template: "{username}.{cluster}.credentials"
  # sidecar_docker_images: ""
  # set_memory_request_to_limit: "false"
  spilo_privileged: "false"
  super_username: postgres
  # team_admin_role: "admin"
  # team_api_role_configuration: "log_statement:all"
  # teams_api_url: http://fake-teams-api.default.svc.cluster.local
  # toleration: ""
  # wal_s3_bucket: ""
  watched_namespace: "*"  # listen to all namespaces
  workers: "4"
apiVersion: v1
kind: ServiceAccount
metadata:
  name: postgres-operator
  namespace: default

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: postgres-operator
rules:
# all verbs allowed for custom operator resources
- apiGroups:
  - acid.zalan.do
  resources:
  - postgresqls
  - postgresqls/status
  - operatorconfigurations
  verbs:
  - create
  - delete
  - deletecollection
  - get
  - list
  - patch
  - update
  - watch
# to create or get/update CRDs when starting up
- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - create
  - get
  - patch
  - update
# to read configuration from ConfigMaps
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
# to manage endpoints which are also used by Patroni
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - create
  - delete
  - deletecollection
  - get
  - list
  - patch
  - update
  - watch
# to CRUD secrets for database access
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - create
  - delete
  - get
  - update
# to check nodes for node readiness label
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
  - list
  - watch
# to read or delete existing PVCs. Creation via StatefulSet
- apiGroups:
  - ""
  resources:
  - persistentvolumeclaims
  verbs:
  - delete
  - get
  - list
 # to read existing PVs. Creation should be done via dynamic provisioning
- apiGroups:
  - ""
  resources:
  - persistentvolumes
  verbs:
  - get
  - list
  - update  # only for resizing AWS volumes
# to watch Spilo pods and do rolling updates. Creation via StatefulSet
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - delete
  - get
  - list
  - patch
  - update
  - watch
# to resize the filesystem in Spilo pods when increasing volume size
- apiGroups:
  - ""
  resources:
  - pods/exec
  verbs:
  - create
# to CRUD services to point to Postgres cluster instances
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - create
  - delete
  - get
  - patch
  - update
# to CRUD the StatefulSet which controls the Postgres cluster instances
- apiGroups:
  - apps
  resources:
  - statefulsets
  - deployments
  verbs:
  - create
  - delete
  - get
  - list
  - patch
# to CRUD cron jobs for logical backups
- apiGroups:
  - batch
  resources:
  - cronjobs
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
# to get namespaces operator resources can run in
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
# to define PDBs. Update happens via delete/create
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - create
  - delete
  - get
# to create ServiceAccounts in each namespace the operator watches
- apiGroups:
  - ""
  resources:
  - serviceaccounts
  verbs:
  - get
  - create
# to create role bindings to the postgres-pod service account
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - rolebindings
  verbs:
  - get
  - create
# to grant privilege to run privileged pods
- apiGroups:
  - extensions
  resources:
  - podsecuritypolicies
  resourceNames:
  - privileged
  verbs:
  - use

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: postgres-operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: postgres-operator
subjects:
- kind: ServiceAccount
  name: postgres-operator
  namespace: default

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: postgres-pod
rules:
# Patroni needs to watch and manage endpoints
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - create
  - delete
  - deletecollection
  - get
  - list
  - patch
  - update
  - watch
# Patroni needs to watch pods
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - patch
  - update
  - watch
# to let Patroni create a headless service
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - create
# to run privileged pods
- apiGroups:
  - extensions
  resources:
  - podsecuritypolicies
  resourceNames:
  - privileged
  verbs:
  - use
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: postgres-operator
  template:
    metadata:
      labels:
        name: postgres-operator
    spec:
      serviceAccountName: postgres-operator
      containers:
      - name: postgres-operator
        image: registry.opensource.zalan.do/acid/postgres-operator:v1.4.0-22-g64389b8-dirty
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            cpu: 100m
            memory: 250Mi
          limits:
            cpu: 500m
            memory: 500Mi
        securityContext:
          runAsUser: 1000
          runAsNonRoot: true
          readOnlyRootFilesystem: true
        env:
        # provided additional ENV vars can overwrite individual config map entries
        - name: CONFIG_MAP_NAME
          value: "postgres-operator"
        # In order to use the CRD OperatorConfiguration instead, uncomment these lines and comment out the two lines above
        # - name: POSTGRES_OPERATOR_CONFIGURATION_OBJECT
        #  value: postgresql-operator-default-configuration
        # Define an ID to isolate controllers from each other
        # - name: CONTROLLER_ID
        #   value: "second-operator"
apiVersion: v1
kind: Service
metadata:
  name: postgres-operator
spec:
  type: ClusterIP
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    name: postgres-operator
apiVersion: "acid.zalan.do/v1"
kind: postgresql
metadata:
  name: acid-postgres-cluster
#  labels:
#    environment: demo
#  annotations:
#    "acid.zalan.do/controller": "second-operator"
spec:
  dockerImage: registry.opensource.zalan.do/acid/spilo-12:1.6-p2
  teamId: "acid"
  volume:
    size: 10Gi
    storageClass: rook-ceph-block
  numberOfInstances: 3
  enableMasterLoadBalancer: false
  enableReplicaLoadBalancer: false
  enableConnectionPooler: true  
  allowedSourceRanges:  # load balancers' source ranges for both master and replica services
  - 127.0.0.1/32
  postgresql:
    version: "12"
    parameters: # Expert section
      tcp_keepalives_count: "5"
      tcp_keepalives_interval: "60"
      tcp_keepalives_idle: "60"
      max_connections: "300"
      shared_buffers: "4GB"
      log_statement: "all"

  enableShmVolume: true
#  spiloFSGroup: 103
#  podAnnotations:
#    annotation.key: value
#  serviceAnnotations:
#    annotation.key: value
#  podPriorityClassName: "spilo-pod-priority"
#  tolerations:
#  - key: postgres
#    operator: Exists
#    effect: NoSchedule
  resources:
    requests:
      cpu: 10m
      memory: 100Mi
    limits:
      cpu: 5000m
      memory: 5000Mi
  patroni:
    initdb:
      encoding: "UTF8"
      locale: "en_US.UTF-8"
      data-checksums: "true"
    pg_hba:
    - hostssl all all 0.0.0.0/0 md5
    - host    all all 0.0.0.0/0 md5
#    slots:
#      permanent_physical_1:
#        type: physical
#      permanent_logical_1:
#        type: logical
#        database: foo
#        plugin: pgoutput
    ttl: 30
    loop_wait: &loop_wait 10
    retry_timeout: 10
    maximum_lag_on_failover: 33554432

# restore a Postgres DB with point-in-time-recovery
# with a non-empty timestamp, clone from an S3 bucket using the latest backup before the timestamp
# with an empty/absent timestamp, clone from an existing alive cluster using pg_basebackup
#  clone:
#    uid: "efd12e58-5786-11e8-b5a7-06148230260c"
#    cluster: "acid-batman"
#    timestamp: "2017-12-19T12:40:33+01:00"  # timezone required (offset relative to UTC, see RFC 3339 section 5.6)
#    s3_wal_path: "s3://custom/path/to/bucket"

# run periodic backups with k8s cron jobs
#  enableLogicalBackup: true
#  logicalBackupSchedule: "30 00 * * *"

#  maintenanceWindows:
#  - 01:00-06:00  #UTC
#  - Sat:00:00-04:00


  initContainers:
  - name: date
    image: busybox
    command: [ "/bin/date" ]
#  sidecars:
#    - name: "telegraf-sidecar"
#      image: "telegraf:latest"
#      resources:
#        limits:
#          cpu: 500m
#          memory: 500Mi
#        requests:
#          cpu: 100m
#          memory: 100Mi
#      env:
#        - name: "USEFUL_VAR"
#          value: "perhaps-true"

# Custom TLS certificate. Disabled unless tls.secretName has a value.
  tls:
    secretName: ""  # should correspond to a Kubernetes Secret resource to load
    certificateFile: "tls.crt"
    privateKeyFile: "tls.key"
    caFile: ""  # optionally configure Postgres with a CA certificate
# When TLS is enabled, also set spiloFSGroup parameter above to the relevant value.
# if unknown, set it to 103 which is the usual value in the default spilo images.

FxKu · 2020-04-07T12:01:29Z

I could reproduce the error when using postgres user. This is expected to not work from the operator side and should be forbidden. It's also not a good practice to use the postgres superuser as connection pooler user.

PR #906 should also cover the nil pointer you've faced.

haroonb · 2020-04-07T12:31:03Z

@FxKu Does that mean, that this won't be resolved or is working as intended? In regards to best practice, I see the point and understand it, however the application accessing the database/pooler can't distinguish between the two and currently only works with the postgres user.

FxKu · 2020-04-07T14:07:41Z

@haroonb the PR will make the operator more strict and give you a reasonable error when system users are picked for the pooler. So yes, it is intended that this doesn't work. Nevertheless, you can connect via the pooler pod/service using the postgres role, even though a different pooler user is configured.

FxKu · 2020-04-09T11:35:03Z

The new image: registry.opensource.zalan.do/acid/postgres-operator:v1.4.0-25-ga1f2bd0-dirty won't allow system users to be used for the connection pool. And should also not see this nil pointer anymore @haroonb

FxKu mentioned this issue Apr 3, 2020

update image and docs for connection pooler #898

Merged

FxKu closed this as completed in #898 Apr 3, 2020

FxKu reopened this Apr 6, 2020

FxKu changed the title ~~pgbouncer/pooler pods are not created~~ using postgres user for pgbouncer/pooler Apr 6, 2020

FxKu added the bug label Apr 7, 2020

haroonb closed this as completed Apr 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

using postgres user for pgbouncer/pooler #892

using postgres user for pgbouncer/pooler #892

haroonb commented Mar 30, 2020

FxKu commented Mar 30, 2020 •

edited

FxKu commented Apr 1, 2020

haroonb commented Apr 1, 2020 •

edited

haroonb commented Apr 3, 2020

FxKu commented Apr 3, 2020 •

edited

FxKu commented Apr 3, 2020 •

edited

vitobotta commented Apr 4, 2020

haroonb commented Apr 6, 2020

FxKu commented Apr 6, 2020 •

edited

haroonb commented Apr 7, 2020

FxKu commented Apr 7, 2020

haroonb commented Apr 7, 2020 •

edited

FxKu commented Apr 7, 2020

haroonb commented Apr 7, 2020

FxKu commented Apr 7, 2020

FxKu commented Apr 9, 2020

using postgres user for pgbouncer/pooler #892

using postgres user for pgbouncer/pooler #892

Comments

haroonb commented Mar 30, 2020

FxKu commented Mar 30, 2020 • edited

FxKu commented Apr 1, 2020

haroonb commented Apr 1, 2020 • edited

haroonb commented Apr 3, 2020

FxKu commented Apr 3, 2020 • edited

FxKu commented Apr 3, 2020 • edited

vitobotta commented Apr 4, 2020

haroonb commented Apr 6, 2020

FxKu commented Apr 6, 2020 • edited

haroonb commented Apr 7, 2020

FxKu commented Apr 7, 2020

haroonb commented Apr 7, 2020 • edited

FxKu commented Apr 7, 2020

haroonb commented Apr 7, 2020

FxKu commented Apr 7, 2020

FxKu commented Apr 9, 2020

FxKu commented Mar 30, 2020 •

edited

haroonb commented Apr 1, 2020 •

edited

FxKu commented Apr 3, 2020 •

edited

FxKu commented Apr 3, 2020 •

edited

FxKu commented Apr 6, 2020 •

edited

haroonb commented Apr 7, 2020 •

edited