Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploying via Helm is unsuccessful #79

Open
paul-snively opened this issue Oct 9, 2022 · 6 comments
Open

Deploying via Helm is unsuccessful #79

paul-snively opened this issue Oct 9, 2022 · 6 comments

Comments

@paul-snively
Copy link

Setup:

Minikube 1.27.0
Kubernetes 1.23.
Strimzi 0.13.1 installed via OperatorHub

KafkaCluster "kafka" created in namespace "moonraker"
registry-schemas KafkaTopic in namespace "moonraker" ready
confluent-schema-registry KafkaUser in namespace "moonraker" ready

Install operator via:

helm install -n operators schema-registry lsstsqre/strimzi-registry-operator --set clusterName="kafka",clusterNamespace="moonraker"

Logs from the operator pod:

[2022-10-09 14:49:36,310] kubernetes.client.re [DEBUG   ] response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"strimzischemaregistries.roundtable.lsst.codes is forbidden: User \"system:serviceaccount:operators:strimzi-registry-operator\" cannot list resource \"strimzischemaregistries\" in API group \"roundtable.lsst.codes\" in the namespace \"moonraker\"","reason":"Forbidden","details":{"group":"roundtable.lsst.codes","kind":"strimzischemaregistries"},"code":403}

[2022-10-09 14:49:36,313] kopf._core.reactor.r [DEBUG   ] Starting Kopf 1.35.6.
[2022-10-09 14:49:36,313] kopf._core.engines.a [INFO    ] Initial authentication has been initiated.
[2022-10-09 14:49:36,313] kopf.activities.auth [DEBUG   ] Activity 'login_via_client' is invoked.
[2022-10-09 14:49:36,314] kopf.activities.auth [DEBUG   ] Client is configured in cluster with service account.
[2022-10-09 14:49:36,315] kopf.activities.auth [INFO    ] Activity 'login_via_client' succeeded.
[2022-10-09 14:49:36,315] kopf._core.engines.a [INFO    ] Initial authentication has finished.
[2022-10-09 14:49:36,330] kopf._cogs.clients.w [DEBUG   ] Starting the watch-stream for customresourcedefinitions.v1.apiextensions.k8s.io cluster-wide.
[2022-10-09 14:49:36,331] kopf._cogs.clients.w [DEBUG   ] Stopping the watch-stream for customresourcedefinitions.v1.apiextensions.k8s.io cluster-wide.
[2022-10-09 14:49:36,332] kopf._core.reactor.o [WARNING ] Not enough permissions to list namespaces. Falling back to a list of namespaces which are assumed to exist: {'moonraker'}
[2022-10-09 14:49:36,332] kopf._cogs.clients.w [DEBUG   ] Starting the watch-stream for namespaces.v1 cluster-wide.
[2022-10-09 14:49:36,332] kopf._cogs.clients.w [DEBUG   ] Starting the watch-stream for strimzischemaregistries.v1beta1.roundtable.lsst.codes in 'moonraker'.
[2022-10-09 14:49:36,333] kopf._cogs.clients.w [DEBUG   ] Starting the watch-stream for secrets.v1 in 'moonraker'.
[2022-10-09 14:49:36,333] kopf._cogs.clients.w [DEBUG   ] Stopping the watch-stream for namespaces.v1 cluster-wide.
[2022-10-09 14:49:36,334] kopf._cogs.clients.w [DEBUG   ] Stopping the watch-stream for strimzischemaregistries.v1beta1.roundtable.lsst.codes in 'moonraker'.
[2022-10-09 14:49:36,334] kopf._cogs.clients.w [DEBUG   ] Stopping the watch-stream for secrets.v1 in 'moonraker'.
[2022-10-09 14:49:36,334] kopf._core.reactor.o [WARNING ] Not enough permissions to watch for resources: changes (creation/deletion/updates) will not be noticed; the resources are only refreshed on operator restarts.
[2022-10-09 14:49:36,335] kopf._core.reactor.o [WARNING ] Not enough permissions to watch for namespaces: changes (deletion/creation) will not be noticed; the namespaces are only refreshed on operator restarts.
[2022-10-09 14:49:36,335] kopf._core.reactor.o [ERROR   ] Watcher for strimzischemaregistries.v1beta1.roundtable.lsst.codes@moonraker has failed: ('strimzischemaregistries.roundtable.lsst.codes is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "strimzischemaregistries" in API group "roundtable.lsst.codes" in the namespace "moonraker"', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'strimzischemaregistries.roundtable.lsst.codes is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "strimzischemaregistries" in API group "roundtable.lsst.codes" in the namespace "moonraker"', 'reason': 'Forbidden', 'details': {'group': 'roundtable.lsst.codes', 'kind': 'strimzischemaregistries'}, 'code': 403})
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 148, in check_response
    response.raise_for_status()
  File "/opt/venv/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://10.96.0.1:443/apis/roundtable.lsst.codes/v1beta1/namespaces/moonraker/strimzischemaregistries')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 108, in guard
    await coro
  File "/opt/venv/lib/python3.10/site-packages/kopf/_core/reactor/queueing.py", line 175, in watcher
    async for raw_event in stream:
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/watching.py", line 82, in infinite_watch
    async for raw_event in stream:
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/watching.py", line 159, in continuous_watch
    objs, resource_version = await fetching.list_objs(
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/fetching.py", line 28, in list_objs
    rsp = await api.get(
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 111, in get
    response = await request(
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/auth.py", line 45, in wrapper
    return await fn(*args, **kwargs, context=context)
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 85, in request
    await errors.check_response(response)  # but do not parse it!
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 150, in check_response
    raise cls(payload, status=response.status) from e
kopf._cogs.clients.errors.APIForbiddenError: ('strimzischemaregistries.roundtable.lsst.codes is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "strimzischemaregistries" in API group "roundtable.lsst.codes" in the namespace "moonraker"', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'strimzischemaregistries.roundtable.lsst.codes is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "strimzischemaregistries" in API group "roundtable.lsst.codes" in the namespace "moonraker"', 'reason': 'Forbidden', 'details': {'group': 'roundtable.lsst.codes', 'kind': 'strimzischemaregistries'}, 'code': 403})
[2022-10-09 14:49:36,336] kopf._core.reactor.o [ERROR   ] Watcher for secrets.v1@moonraker has failed: ('secrets is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "secrets" in API group "" in the namespace "moonraker"', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'secrets is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "secrets" in API group "" in the namespace "moonraker"', 'reason': 'Forbidden', 'details': {'kind': 'secrets'}, 'code': 403})
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 148, in check_response
    response.raise_for_status()
  File "/opt/venv/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://10.96.0.1:443/api/v1/namespaces/moonraker/secrets')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 108, in guard
    await coro
  File "/opt/venv/lib/python3.10/site-packages/kopf/_core/reactor/queueing.py", line 175, in watcher
    async for raw_event in stream:
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/watching.py", line 82, in infinite_watch
    async for raw_event in stream:
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/watching.py", line 159, in continuous_watch
    objs, resource_version = await fetching.list_objs(
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/fetching.py", line 28, in list_objs
    rsp = await api.get(
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 111, in get
    response = await request(
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/auth.py", line 45, in wrapper
    return await fn(*args, **kwargs, context=context)
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 85, in request
    await errors.check_response(response)  # but do not parse it!
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 150, in check_response
    raise cls(payload, status=response.status) from e
kopf._cogs.clients.errors.APIForbiddenError: ('secrets is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "secrets" in API group "" in the namespace "moonraker"', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'secrets is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "secrets" in API group "" in the namespace "moonraker"', 'reason': 'Forbidden', 'details': {'kind': 'secrets'}, 'code': 403})

It seems the created ServiceAccount doesn't have the permissions it needs to do what it needs to do.

@harksodje
Copy link

@paul-snively, I am currently experiencing the same issue.

@harksodje
Copy link

harksodje commented Oct 25, 2022

@paul-snively, I was able to resolve this issue. The clusterrolebinding of the schema-registry is referencing wrong namespace.
This is default:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    meta.helm.sh/release-name: kafka
    meta.helm.sh/release-namespace: kafka
  creationTimestamp: "2022-10-25T14:49:15Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: strimzi-registry-operator
  resourceVersion: "12193"
  uid: 60fa9e7a-d0bb-4811-8a94-e65a31378456
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: strimzi-registry-operator
subjects:
- kind: ServiceAccount
  name: strimzi-registry-operator
  namespace: strimzi-registry-operator

I just changed the and the pod start working
Also, you have to specify while installing the helm chart

@karanalang
Copy link

faced similar issue, resolved by creating the ServiceAccount in namespace - 'strimzi-registry-operator'

@husker-du
Copy link

husker-du commented Dec 26, 2022

The property operatorNamespace of the strimzi-registry-operator sets the namespace where the operator is going to be installed, by default this value is set to the namespace strimzi-registry-operator. Therefore, the helm release has to be deployed in this namespace by setting the option -n strimzi-registry-operator. If this namespace does not exists, it can be created by the helm command using the option --create-namespace.

In summary, the following command deploys the strimzi-registry-operator in the strimzi-registry-operator namespace for a strimzi Kafka cluster named kafka-cluster deployed in the strimzi namespace:

$ helm upgrade --install --create-namespace strimzi-registry-operator lsstsqre/strimzi-registry-operator --set clusterNamespace=strimzi,clusterName=kafka-cluster,operatorNamespace=strimzi-registry-operator -n strimzi-registry-operator

The property operatorNamespace=strimzi-registry-operator can be omitted because this is the default value.

@karanalang
Copy link

karanalang commented Dec 30, 2022 via email

@strowi
Copy link

strowi commented May 23, 2023

Hi,

ran into a similar issue where the ServiceAccount seems to be missing permissions.
With version 2.1.0 i had to patch the ClusterRole to get rid of the following 2:

kopf._cogs.clients.w [DEBUG   ] Stopping the watch-stream for customresourcedefinitions.v1.apiextensions.k8s.io cluster-wide
kopf._cogs.clients.w [DEBUG   ] Stopping the watch-stream for namespaces.v1 cluster-wide.

I just added them to an appropriate rule instead of creating an extra one:

rules:
- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - list
  - get
  **- watch**
- apiGroups:
  - ""
  resources:
  - secrets
  - configmaps
  - services
  **- namespaces**
  verbs:
  - get
  - list
  - watch
  - patch
  - create

Maybe this can help someone else stumbling over this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants