Breaking Change with Ingress Controller 1.3.0

[rpf3]

Name and Version

bitnami/nginx-ingress-controller 9.2.20

What steps will reproduce the bug?

1. Install the nginx ingress controller chart version 9.2.20 with the default values

Are you using any custom parameters or values?

N/A

What is the expected behavior?

The controller pod should log that the leader election occurred without issue.

What do you see instead?

The leader election is failing due to a change in the upstream app version 1.3.0

leaderelection.go:330] error retrieving resource lock nginx/ingress-controller-leader: leases.coordination.k8s.io "ingress-controller-leader" is forbidden: User "system:serviceaccount:nginx:nginx-nginx-ingress-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "nginx"

Additional information

https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.3.0

[aeimer]

I see similar errors and get a 503 on some ingress routes.

E0721 16:21:33.992844 1 leaderelection.go:330] error retrieving resource lock ingress-dotdev/ingress-controller-leader: leases.coordination.k8s.io "ingress-controller-leader" is forbidden: User "system:serviceaccount:ingress-dotdev:nginx-ingress-ctrl-dotdev" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ingress-dotdev"

Seems to be an DaemonSet issue, as it only pops up on DS deploys.

Edit: Actually I'm wrong. The ingress seems to work as DP and DS, but the leader election still is not working.

[aeimer]

So I tried to add it manually but there seem to be no impact, even after a restart.

https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/templates/clusterrole.yaml#L32

Added to nginx-ingress-ctrl-main:

- apiGroups:
    - coordination.k8s.io
  resources:
    - leases
  verbs:
    - list
    - watch

Edit: Okay seems to be a bit more effort than that => kubernetes/ingress-nginx@cf4dca8

[HerHde]

Problem persists in chart version 9.2.21

[Pentusha]

@aeimer
You are missing few required verbs. At least this code solved my issue:

  - verbs:
      - create
      - update
      - get
      - list
      - watch
    apiGroups:
      - coordination.k8s.io
    resources:
      - leases

Restart is not required.

[EnriqueHormilla]

Rollback to previous version :(

[rpf3]

fixed by #11454

[MaxRink]

@rpf3 unfortunately only partially
https://github.com/bitnami/charts/pull/11454/files#diff-2682c142866340497199d5c59c8ae42e3073cb0a07d433190b143b4624bda95dR100
hardcodes the name, which actually is configurable :S

[lknite]

Also seeing this in the latest chart 9.2.21:

E0901 15:54:34.005066       1 leaderelection.go:330] error retrieving resource lock nginx-ingress/ingress-controller-leader: leases.coordination.k8s.io "ingress-controller-leader" is forbidden: User "system:serviceaccount:nginx-ingress:nginx-ingress-nginx-ingress-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "nginx-ingress"

[aeimer]

@lknite 9.2.21 is not the latest chart, isnt it?

@rpf3 Are you aware of the comment from @MaxRink?

[lknite]

oops, thank you, 9.3.4 is the latest

[nelsou]

Had the following error

k8s.io/client-go@v0.26.4/tools/cache/reflector.go:169: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:default:nginx-ingress" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
k8s.io/client-go@v0.26.4/tools/cache/reflector.go:169: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:default:nginx-ingress" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
error retrieving resource lock default/ingress-controller-leader: leases.coordination.k8s.io "ingress-controller-leader" is forbidden: User "system:serviceaccount:default:nginx-ingress" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "default"

Had to add the following in ClusterRole > nginx and everything worked.

  - verbs:
      - list
      - watch
    apiGroups:
      - coordination.k8s.io
    resources:
      - leases
  - verbs:
      - list
      - watch
    apiGroups:
      - discovery.k8s.io
    resources:
      - endpointslices