master service gets no selectors on OpenShift

[dejwsz]

Service for masters gets no selectors at all - they must be added manually. Without this cluster is not able to sync properly. Tested with operator 1.3.1 and spilo image: registry.opensource.zalan.do/acid/spilo-cdp-12:1.6-p16.
I needed to patch my "test-minimal-cluster" service dedicated to master PODs adding such

selectors:
  selector:
    application: spilo
    cluster-name: test-minimal-cluster
    spilo-role: master

Replica service has selectors in place so it is fine. This is probably a bug?

[CyberDem0n]

If Patroni is configured to use endpoints for leader race it will put the IP of the master pod into the subsets.
This is our recommended way to run the whole stack on K8s.
OpenShift is a different story. It doesn't allow putting IPs from the pods subnet into subsets, so label selector is required. But we never announced the official support of OpenShift anyway.

[dejwsz]

I can see this: https://github.com/zalando/patroni/tree/master/kubernetes/openshift-example

[dejwsz]

OK I found it. I updated the config map pointed by "pod_environment_configmap" like this:
DEBUG: "true"
KUBERNETES_USE_CONFIGMAPS: "true"
PATRONI_KUBERNETES_ROLE_LABEL: spilo-role
and now master service gets its selectors properly. Of course, I have also set the value of "pod_role_label" to "spilo-role".

[dejwsz]

To be sure it really worked I did it again after cleaning up everything carefully. I can confirm master service gets now proper selectors.

[dejwsz]

This is also interesting regarding setting PATRONI_KUBERNETES_POD_IP, seems possible in openshift:
https://github.com/zalando/patroni/blob/master/kubernetes/openshift-example/templates/template_patroni_persistent.yaml

[CyberDem0n]

Effectively it does nothing, because PATRONI_KUBERNETES_USE_ENDPOINTS isn't set.

[dejwsz]

Yeah, I can see now as I could not repeat it today again, somehow I got these selectors yesterday.

[ReSearchITEng]

I confirm that this issue is there on OpenShift 4.2 as well (even with KUBERNETES_USE_CONFIGMAPS: "true" + PATRONI_KUBERNETES_ROLE_LABEL: spilo-role (2nd one seems to be the default anyway);
In operator configMap I see pod_role_label: spilo-role as well. So all seems good from this config p.o.v.
I confirm that once sector is added to the master service, cluster works fine.

BTW, it's only the master service what is missing the selector. The replica service (-repl) does have the selector populated properly (of course, matching its role: replica) - which makes me think that it's not an issue due to permissions to update a service or something (if this is how it works)

I have noticed in the rbac.yaml, ClusterRole postgres-pod, for service it has only "create" permission for services, with a note: for headless service. Why does it mention "headless" here?

There are no errors in the Operator logs, not clear what is the issue... @dejwsz: did you find any possible root cause?

I guess it's somewhere a bug: Operator believes it's using endpoints, therefore not updating the service from headless to normal one with selectors.

postgres-operator/pkg/cluster/k8sres.go

Line 1339 in 07c5da3

if role == Replica {

:
I think this condition should be extended to include also the master service, when we are using configMaps.

Patroni has this parameter: KUBERNETES_USE_CONFIGMAPS, which we put in another configMap and ask operator to pass it to patroni via pod_environment_configmap.

It look to me Operator does not a parameter to know if we want configMaps or Endpoints, right?
IMO we need an Operator parameter (e.g. "master_selection_using" with possible values: "endpoints", "configmaps" ..)
And when master_selection_using=="configmaps", do:

1. In L1339 add 'or master_selection_using=="configmaps" ' ( so it will add selector also for master, not only for replica)
2. (ideally) inject in the pod env variable: KUBERNETES_USE_CONFIGMAPS="true"
   (alternatively, create a configMap with and populate the pod_environment_configmap.

PS: (for working in non-privileged mode, and rootless, tests were done with PR 414 which is basically this image: researchiteng/spilo3:pgroot

[skalolazka]

Hello everyone! Experiencing this on Kubernetes 1.16.7. Haven't been able to fix it in any way at all, even manually, since even after I edit the service the operator somehow says it's unavailable (when it actually is available, also restarted the operator and tried more things out) (edited this for more info). This seriously hinders our work.

[ReSearchITEng]

Hi @skalolazka - the above PR#887 should solve this. Hopefully would be approved by the maintainers.

[ReSearchITEng]

For OpenShift, you need:
Operator Image should be at least: registry.opensource.zalan.do/acid/postgres-operator:v1.4.0-21-g1249626-dirty
Operator should be configured with these values:
kubernetes_use_configmaps: "true"
docker_image: registry.opensource.zalan.do/acid/spilo-cdp-12:1.6-p114 #or newer

To get the latest versions of the images of both operator and spilo, do:
https://registry.opensource.zalan.do/v2/acid/postgres-operator/tags/list
https://registry.opensource.zalan.do/v2/acid/spilo-cdp-12/tags/list