In Openshift, endpoints is forbidden

[Samusername]

Hi!

I tested with older versions of postgres-operator and spilo,
and also with latest published versions:
https://github.com/zalando/postgres-operator/releases/tag/v1.7.1
and --> registry.opensource.zalan.do/acid/spilo-14:2.1-p3

Following kinds of errors are shown in logs of acid-upgrade-test-0 pod:

patroni.dcs.kubernetes.K8sClient.rest.ApiException: (403)
Reason: Forbidden

endpoints "acid-upgrade-test" is forbidden: endpoint address ... is not allowed",
"reason":"Forbidden","details":{"name":\"acid-upgrade-test\","kind":"endpoints"},"code":403}
...
2021-11-23 14:40:39,183 ERROR: failed to update leader lock
2021-11-23 14:40:39,184 INFO: not promoting because failed to update leader lock in DCS

I have seen e.g. following discussion chain:
#985
I tried to compare and add any permissions which were mentioned there.
Same error happened still.

What kinds of permissions should be given? In ClusterRole?

Btw., is config map needed to be enabled separately nowadays in these installations to Openshift? I did not set it anywhere.

Files:

I used following minimal manifest yaml:

Click to expand!

minimal manifest yaml

apiVersion: "acid.zalan.do/v1"
kind: postgresql
metadata:
name: acid-upgrade-test
namespace: default
spec:
teamId: "acid"
volume:
  size: 1Gi
numberOfInstances: 2
users:
  zalando:  # database owner
  - superuser
  - createdb
  foo_user: []  # role for application foo
databases:
  foo: zalando  # dbname: owner
preparedDatabases:
  bar: {}
postgresql:
  version: "13"

I will need to make one scratch installation test again from a "clean table".

[CyberDem0n]

For openshift you have to use configmaps instead of endpoints.

[FxKu]

@Samusername have a look at the RBAC template of the helm chart to understand how the cluster role should look like

[Samusername]

@Samusername have a look at the RBAC template of the helm chart to understand how the cluster role should look like

Yes, I have used the same ClusterRole definitions (and the whole helm chart) from there, in the mentioned latest test.

(I did not bind it, if it does not get bound automatically in the installation of the helm chart.)

[Samusername]

I am trying to avoid "ConfigMap configuration":
including kubernetes_use_configmaps: true and configTarget: "ConfigMap".
( Which is becoming deprecated. )
When I refer to "ConfigMap configuration", I refer to those.

So, following was kept in comments, in values.yaml:

# kubernetes_use_configmaps: ...
Default is false, I think. So, default was in use.

With configTarget: "OperatorConfigurationCRD",
Following kind of request is problematic in Openshift, when DB cluster starts up:

PATCH   https://<x.y.z.w:port>/api/v1/namespaces/default/endpoints/acid-upgrade-test
add /subnets addresses: ip: "x1.y1.z1.v1"
--> endpoints "acid-upgrade-test" is forbidden: endpoint address ... is not allowed

We may manage to get enough permissions, in a certain namespace, to get such executed in Openshift.
In such case:

Are there some other reasons (problems) why "ConfigMap configuration" should be used instead of the "endpoints" mechanism (CRD)?

Or are we "free" to go with the "endpoints" mechanism ("OperatorConfigurationCRD" configuration), in Openshift, then otherwise?

[FxKu]

It was fixed now with #1760 and #1825 and will be included in the next release this week.

[davtex]

As this is this the first result for searching "endpoint address is not allowed", I will add one other possible solution that may help people dealing with same issue on other operators such as k8ssandra-operator:

You need to modify Role/Clusterrole and use resource "endpoints/restricted" instead of "endpoints", like this:

- apiGroups:
  - ""
  resources:
  - endpoints
  - endpoints/restricted
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

This quirk is only documented in legacy Openshift 3.X documentation here:
https://docs.openshift.com/container-platform/3.3/architecture/core_concepts/pods_and_services.html#endpoints

Verified to work on Openshift 4.11