operator: option to limit CRs the operator should handle

[lburgazzoli]

We should be able to restrict what CRs a camel-k operator should handle, as example, we can add a label like camel.apache.org/operator.id to each CR:

apiVersion: camel.apache.org/foo
kind: bar
metadata:
  labels: 
    camel.apache.org/operator.id: "operator-1"

The operator.id should also be used to acquire the operator lock.

Note:

• We'd need to amend the kamel commands to eventually take into account such label
• We'd need to introduce a new command to move CR between operators

[astefanutti]

Ideally, these labels should be taken into account by the controller-runtime informer that watches for Integration resources, to filter sever-side, so that the corresponding operator does not filter client-side and cache the out-of-scope Integrations, so that the solution scales nicely.

[astefanutti]

Ideally, these labels should be taken into account by the controller-runtime informer that watches for Integration resources, to filter sever-side, so that the corresponding operator does not filter client-side and cache the out-of-scope Integrations, so that the solution scales nicely.

It seems like server-side filtering is not yet possible with controller-runtime: kubernetes-sigs/controller-runtime#244.

We can go for a client-side predicate in the meantime.

[lburgazzoli]

One of the use case where is feature can be useful is to implement "rolling upgrade" of the camel-k operator where you have multiple running version of the camel-k operator and you want to move integrations from one operator to the another by steps.

The simplest approach is to change the label i.e. from camel.apache.org/operator.id: "operator-1" to camel.apache.org/operator.id: "operator-2" however this may not be enough as "operator-1" may be performing some work and changing the label may leave the integration in a inconsistent state so we may need to introduce a camel-k level finalizer concept to let the operator currently owning a resource to gracefully release a resource and moving it to the target operator.

@astefanutti @nicolaferraro what do you think ?

[astefanutti]

Right. As we are introducing a mechanism where multiple operators can concurrently manage the same resources, a locking mechanism is necessary. More generally to the rolling upgrade use-case, that mechanism would make sure that only a single operator manages the same resource at a single point in time. That would prevent issues where the label selectors for multiple operators intersect.

Now on how to implement that locking mechanism, I think we could reuse the approach used for locking by multiple replicas of the same operator, that is with a Lease resource, but owned by the Integration. The Lease resource seems the preferred mechanism (by client-go and controller-runtime), and handle use cases such as renewing and deadlining nicely. It would also be an extension of the locking implementation already in place for managing HA of the operator replicas.

[heiko-braun]

Is the camel.apache.org/operator.id used to describe an instance of an operator?

[heiko-braun]

+1 on the lease resource for managing locks.

[lburgazzoli]

@astefanutti we should also take into account the IntegrationPlatform CR i.e. if multiple operators are installed in the same namespace, it should be possible to have multiple instances of the IntegrationPlatform labeled according so only the relevant operator would use it (note: we may think about naming the maven settings config map after the selector label) .

[astefanutti]

@astefanutti we should also take into account the IntegrationPlatform CR i.e. if multiple operators are installed in the same namespace, it should be possible to have multiple instances of the IntegrationPlatform labeled according so only the relevant operator would use it (note: we may think about naming the maven settings config map after the selector label) .

Right, scoping IntegrationPlatform should be supported in addition to scoping Integration. actually, I think scoping must be implemented for any kind of resources being watched / managed by the operator, like IntegrationKit, Build, ... Even for environments that do not leverage the builder, I think it makes sense to implement a generic scoping mechanism.

[astefanutti]

Selectors support has been added to k8s informers into the latest version of controller-runtime: kubernetes-sigs/controller-runtime#1435.