User cannot create resource in API group

[orpiske]

When running an integration on our OpenShift 4.2.9 I got an error stating that the user cannot create resource "servicemonitors" in the the API group "monitoring.coreos.com".

The message is:

{"level":"error","ts":1582651063.927428,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"integration-controller","request":"camel-k-event-streaming-dev/open-aq-consumer","error":"error executing post actions: error during replace resource: could not create or replace resource open-aq-consumer: servicemonitors.monitoring.coreos.com is forbidden: User \"system:serviceaccount:camel-k-event-streaming-dev:camel-k-operator\" cannot create resource \"servicemonitors\" in API group \"monitoring.coreos.com\" in the namespace \"camel-k-event-streaming-dev\"","errorVerbose":"servicemonitors.monitoring.coreos.com is forbidden: User \"system:serviceaccount:camel-k-event-streaming-dev:camel-k-operator\" cannot create resource \"servicemonitors\" in API group \"monitoring.coreos.com\" in the namespace \"camel-k-event-streaming-dev\"\ncould not create or replace resource open-aq-consumer

The full message is available here.

Despite the message, the integration eventually runs after a long time stuck in the Deployment part.

I have tried working around this issue by increasing the permissions for the operator user with:

oc policy add-role-to-user edit system:serviceaccount:camel-k-event-streaming-dev:camel-k-operator

However that did not help either (and, in fact, made it worse) because the integration now gets completely stuck and the operator seems to enter a loop with the error below:

E0226 09:43:17.083486 1 reflector.go:123] k8s.io/client-go@v12.0.0+incompatible/tools/cache/reflector.go:96: Failed to list *v1.ServiceMonitor: servicemonitors.monitoring.coreos.com is forbidden: User "system:serviceaccount:camel-k-event-streaming-dev:camel-k-operator" cannot list resource "servicemonitors" in API group "monitoring.coreos.com" in the namespace "camel-k-event-streaming-dev": RBAC: clusterrole.rbac.authorization.k8s.io "list" not found

The output of my oc get integrationplatform -o yaml is available here.

[heiko-braun]

Is it similar to https://github.com/syndesisio/syndesis/pull/7976/files ?

Syndesis ran into this issue after an update to the operator-sdk

[astefanutti]

I suspect this is caused by the prometheus trait. Having a quick look at the Camel K operator roles, it seems the permissions required for the trait to create the Prometheus resources are missing.

@orpiske could you precise the command you use to run the integration?

[orpiske]

@astefanutti I have this whenever I try to run an integration with --trait prometheus.enabled=true ... any integration.

I first noticed this on a demo I am working on. In that case, the full CLI was something like:

kamel run OpenAQConsumer.java --trait prometheus.enabled=true --property kafka.bootstrap.address=my-kafka-host:9094 --dev

[astefanutti]

@orpiske thanks. that confirms it's caused by the prometheus trait.

As a work-around, you can amend the camel-k-operator role, e.g., kubectl edit role camel-k-operator, and add the missing permissions.

[orpiske]

@orpiske thanks. that confirms it's caused by the prometheus trait.

As a work-around, you can amend the camel-k-operator role, e.g., kubectl edit role camel-k-operator, and add the missing permissions.

Awesome. I will give it a try, thanks!

[lburgazzoli]

@astefanutti @orpiske is this still an issue ?

[astefanutti]

I think it's been fixed with #1453.

[orpiske]

@lburgazzoli not anymore. I remember applying the work-around that @astefanutti pointed and that did the trick.