Cannot create resource "secrets"

[captainjapeng]

Describe the bug
postgres-operator service account is unable to create secrets
Running pgo create cluster -n pgouser1 postgres after following the installation guide (kubectl apply -f operator.yml)

To Reproduce
Steps to reproduce the behavior:

1. Follow the Install the PostgreSQL Operator guide
2. Run pgo version -n pgouser1 (displays v4.3.0)
3. Run pgo create cluster -n pgouser1 postgres
4. See error

Expected behavior
Cluster should be created

Screenshots
2.
4.

Please tell us about your environment:

• Operating System: Debian do-kube-1.16.6-do.2
• Where is this running ( Local, Cloud Provider): Digital Ocean
• Storage being used (NFS, Hostpath, Gluster, etc): Digital Ocean Block Storage
• Container Image Tag: centos7-12.2-4.3.0
• PostgreSQL Version: N/A
• Platform (Docker, Kubernetes, OpenShift): Kubernetes
• Platform Version: 1.16.6

Additional context
Add any other context about the problem here.
Only changes made to the operator is change admin password and add/change the ff:

                  - name: BACKREST_STORAGE
                    value: "digitalocean"
                  - name: BACKUP_STORAGE
                    value: "digitalocean"
                  - name: PRIMARY_STORAGE
                    value: "digitalocean"
                  - name: REPLICA_STORAGE
                    value: "digitalocean"

                  - name: STORAGE5_NAME
                    value: "digitalocean"
                  - name: STORAGE5_ACCESS_MODE
                    value: "ReadWriteOnce"
                  - name: STORAGE5_SIZE
                    value: "1Gi"
                  - name: STORAGE5_TYPE
                    value: "dynamic"
                  - name: STORAGE5_CLASS
                    value: "do-block-storage"

[jkatz]

kubectl get ns pgouser1

Does this exist?

There was an issue with the deployment file I had noticed earlier today which has since been corrected, i.e. the NAMESPACE block now reads:

- name: NAMESPACE
value: "pgo"

https://github.com/CrunchyData/postgres-operator/blob/master/installers/kubectl/postgres-operator.yml#L82-L83

Try editing that and redeploying.

Sorry about that!

[captainjapeng]

Yes it does exist, but age is 12h ago so that may be from the time I've tested with 4.2.2 but I had already uninstall it from the clustur using the ansible method

Regarding the changes with namespace, I've managed to install 4.3.0 with the updated operator.yml as you can see in the screenshot

[jkatz]

Did you fully uninstall it with the deployment container before reinstalling? Mentinoed here in the docs.

I'm not sure what happened in your previous installation, but there may be a lingering ServiceAccount that does not have the appropriate permissions. Also note that presently the installer itself is not idempotent (uninstallation certainly is): if there is a lingering ClusterRole / ClusterRoleBinding it needs to be removed o/w the installation fails. We're working on a fix for that in the next release.

[captainjapeng]

And also if I change the operator.yml's DEPLOY_ACTION to update and do kubectl apply -f operator.yml it shows this error

> kubectl apply -f operator.yml 
serviceaccount/pgo-deployer-sa unchanged
clusterrolebinding.rbac.authorization.k8s.io/pgo-deployer-crb unchanged
The Job "pgo-deploy" is invalid:
    spec.template:
        Invalid value:
            core.PodTemplateSpec {
                ObjectMeta:v1.ObjectMeta {
                    Name:"pgo-deploy",
                    GenerateName:"",
                    Namespace:"",
                    SelfLink:"",
                    UID:"",
                    ResourceVersion:"",
                    Generation:0,
                    CreationTimestamp:v1.Time { Time:time.Time { wall:0x0, ext:0, loc:(*time.Location)(nil) } },
                    DeletionTimestamp:(*v1.Time)(nil),
                    DeletionGracePeriodSeconds:(*int64)(nil),
                    Labels:map[string]string { "controller-uid":"d54495d8-f025-4754-bff1-00b1f9546ebc", "job-name":"pgo-deploy" },
                    Annotations:map[string]string(nil),
                    OwnerReferences:[]v1.OwnerReference(nil),
                    Finalizers:[]string(nil),
                    ClusterName:"",
                    ManagedFields:[]v1.ManagedFieldsEntry(nil)
                },
                Spec:core.PodSpec {
                    Volumes:[]core.Volume(nil),
                    InitContainers:[]core.Container(nil),
                    Containers:[]core.Container {
                        core.Container {
                            Name:"pgo-deploy",
                            Image:"registry.developers.crunchydata.com/crunchydata/pgo-deployer:centos7-4.3.0",
                            Command:[]string { "/pgo-deploy.sh" },
                            Args:[]string(nil),
                            WorkingDir:"",
                            Ports:[]core.ContainerPort(nil),
                            EnvFrom:[]core.EnvFromSource(nil),
                            Env:[]core.EnvVar {
                                core.EnvVar { Name:"ANSIBLE_CONFIG", Value:"/ansible/ansible.cfg", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"ARCHIVE_MODE", Value:"true", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"ARCHIVE_TIMEOUT", Value:"60", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"BACKREST", Value:"true", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"BADGER", Value:"false", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"CRUNCHY_DEBUG", Value:"false", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"CREATE_RBAC", Value:"true", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"CCP_IMAGE_PREFIX", Value:"registry.developers.crunchydata.com/crunchydata", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"CCP_IMAGE_TAG", Value:"centos7-12.2-4.3.0", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"DB_PASSWORD_LENGTH", Value:"32", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"DB_PORT", Value:"5432", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"DB_REPLICAS", Value:"0", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"DB_USER", Value:"testuser", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"DEFAULT_INSTANCE_MEMORY", Value:"128Mi", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"DEFAULT_PGBACKREST_MEMORY", Value:"", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"DEFAULT_PGBOUNCER_MEMORY", Value:"", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"DEPLOY_ACTION", Value:"update", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"DISABLE_AUTO_FAILOVER", Value:"false", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"EXPORTERPORT", Value:"9187", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"HOME", Value:"/tmp", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"METRICS", Value:"false", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"NAMESPACE", Value:"pgo", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"NAMESPACE_MODE", Value:"dynamic", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"PGBADGERPORT", Value:"10000", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"PGO_ADMIN_PASSWORD", Value:"a3uVDhNFDmSYCj9XsWGGM7gFdcVCqG5v", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"PGO_ADMIN_PERMS", Value:"*", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"PGO_ADMIN_ROLE_NAME", Value:"pgoadmin", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"PGO_ADMIN_USERNAME", Value:"admin", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"PGO_CLIENT_VERSION", Value:"v4.3.0", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"PGO_IMAGE_PREFIX", Value:"registry.developers.crunchydata.com/crunchydata", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"PGO_IMAGE_TAG", Value:"centos7-4.3.0", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"PGO_INSTALLATION_NAME", Value:"devtest", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"PGO_OPERATOR_NAMESPACE", Value:"pgo", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"SCHEDULER_TIMEOUT", Value:"3600", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"BACKREST_STORAGE", Value:"digitalocean", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"BACKUP_STORAGE", Value:"digitalocean", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"PRIMARY_STORAGE", Value:"digitalocean", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"REPLICA_STORAGE", Value:"digitalocean", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"WAL_STORAGE", Value:"", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE1_NAME", Value:"hostpathstorage", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE1_ACCESS_MODE", Value:"ReadWriteMany", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE1_SIZE", Value:"1G", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE1_TYPE", Value:"create", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE2_NAME", Value:"replicastorage", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE2_ACCESS_MODE", Value:"ReadWriteMany", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE2_SIZE", Value:"700M", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE2_TYPE", Value:"create", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE3_NAME", Value:"nfsstorage", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE3_ACCESS_MODE", Value:"ReadWriteMany", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE3_SIZE", Value:"1G", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE3_TYPE", Value:"create", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE3_SUPPLEMENTAL_GROUPS", Value:"65534", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE4_NAME", Value:"nfsstoragered", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE4_ACCESS_MODE", Value:"ReadWriteMany", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE4_MATCH_LABEL", Value:"crunchyzone=red", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE4_TYPE", Value:"create", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE4_SUPPLEMENTAL_GROUPS", Value:"65534", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE5_NAME", Value:"digitalocean", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE5_ACCESS_MODE", Value:"ReadWriteOnce", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE5_SIZE", Value:"1Gi", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE5_TYPE", Value:"dynamic", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE5_CLASS", Value:"do-block-storage", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE6_NAME", Value:"primarysite", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE6_ACCESS_MODE", Value:"ReadWriteOnce", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE6_SIZE", Value:"4G", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE6_TYPE", Value:"dynamic", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE6_CLASS", Value:"primarysite", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE7_NAME", Value:"alternatesite", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE7_ACCESS_MODE", Value:"ReadWriteOnce", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE7_SIZE", Value:"4G", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE7_TYPE", Value:"dynamic", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE7_CLASS", Value:"alternatesite", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE8_NAME", Value:"gce", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE8_ACCESS", Value:"ReadWriteOnce", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE8_SIZE", Value:"300M", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE8_TYPE", Value:"dynamic", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE8_CLASS", Value:"standard", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE9_NAME", Value:"rook", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE9_ACCESS_MODE", Value:"ReadWriteOnce", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE9_SIZE", Value:"1Gi", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE9_TYPE", Value:"dynamic", ValueFrom:(*core.EnvVarSource)(nil) },
                                core.EnvVar { Name:"STORAGE9_CLASS", Value:"rook-ceph-block", ValueFrom:(*core.EnvVarSource)(nil)}
                            },
                            Resources:core.ResourceRequirements {
                                Limits:core.ResourceList(nil),
                                Requests:core.ResourceList(nil)
                            },
                            VolumeMounts:[]core.VolumeMount(nil),
                            VolumeDevices:[]core.VolumeDevice(nil),
                            LivenessProbe:(*core.Probe)(nil),
                            ReadinessProbe:(*core.Probe)(nil),
                            StartupProbe:(*core.Probe)(nil),
                            Lifecycle:(*core.Lifecycle)(nil),
                            TerminationMessagePath:"/dev/termination-log",
                            TerminationMessagePolicy:"File",
                            ImagePullPolicy:"IfNotPresent",
                            SecurityContext:(*core.SecurityContext)(nil),
                            Stdin:false,
                            StdinOnce:false,
                            TTY:false
                        }
                    },
                    EphemeralContainers:[]core.EphemeralContainer(nil),
                    RestartPolicy:"Never",
                    TerminationGracePeriodSeconds:(*int64)(0xc01ccccdd8),
                    ActiveDeadlineSeconds:(*int64)(nil),
                    DNSPolicy:"ClusterFirst",
                    NodeSelector:map[string]string(nil),
                    ServiceAccountName:"pgo-deployer-sa",
                    AutomountServiceAccountToken:(*bool)(nil),
                    NodeName:"",
                    SecurityContext:(*core.PodSecurityContext)(0xc00aac4930),
                    ImagePullSecrets:[]core.LocalObjectReference(nil),
                    Hostname:"",
                    Subdomain:"",
                    Affinity:(*core.Affinity)(nil),
                    SchedulerName:"default-scheduler",
                    Tolerations:[]core.Toleration(nil),
                    HostAliases:[]core.HostAlias(nil),
                    PriorityClassName:"",
                    Priority:(*int32)(nil),
                    PreemptionPolicy:(*core.PreemptionPolicy)(nil),
                    DNSConfig:(*core.PodDNSConfig)(nil),
                    ReadinessGates:[]core.PodReadinessGate(nil),
                    RuntimeClassName:(*string)(nil),
                    Overhead:core.ResourceList(nil),
                    EnableServiceLinks:(*bool)(nil),
                    TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)
                }
            }: field is immutable

[jkatz]

Please ensure you've cleaned up the old job before running the new one, e.g.

kubectl delete -f postgres-operator.yml
kubectl apply -f postgres-operator.yml

[captainjapeng]

Alright, I have cleanup the cluster
Steps taken:

1. Run a new job with uninstall mode
2. Run kubectl delete namespace pgo
3. Check the ff:
   • rolebinding - Clear
   • serviceaccount - pgouser1 and pgouser2 are still there (which are 12 hrs ago so it definitely
     came from 4.2.2)
   • clusterrole - Clear
   • clusterrolebinding - Clear

Thank you for your help :)

[captainjapeng]

I had cleaned the cluster multiple times and still getting this Cannot create resource "secrets" error.
If I edit the clusterrole directly I managed to make the operator execute, but seems stuck on created pgtasks

[captainjapeng]

After provisioning a new cluster I found out that the secrets error is the case, and then I have tried using pgo namespace for creating new a cluster and it worked.

This documentation is outdate by v4.3.0

Just gonna leave it here for those who are new to postgres-operator.
pgo test dev -n pgo