[Bug]: Connect's dynamic log level changes are only applied to one worker node

[fvaleri]

Bug Description

If we deploy the following 3-nodes Connect cluster:

$ kubectl get po
NAME                                         READY   STATUS    RESTARTS   AGE
my-cluster-entity-operator-dbfcd857c-kfz92   3/3     Running   0          25m
my-cluster-kafka-0                           1/1     Running   0          26m
my-cluster-zookeeper-0                       1/1     Running   0          27m
my-connect-cluster-connect-0                 1/1     Running   0          25m
my-connect-cluster-connect-1                 1/1     Running   0          25m
my-connect-cluster-connect-2                 1/1     Running   0          25m
strimzi-cluster-operator-5dff6ccf95-7tp6p    1/1     Running   0          29m

The Cluster Operator also creates a Service that loops through all matching Connect's REST endpoints (one for each worker node).

$ kubectl get svc my-connect-cluster-connect-api -o yaml | yq '.spec'
clusterIP: 10.97.195.99
clusterIPs:
  - 10.97.195.99
internalTrafficPolicy: Cluster
ipFamilies:
  - IPv4
ipFamilyPolicy: SingleStack
ports:
  - name: rest-api
    port: 8083
    protocol: TCP
    targetPort: 8083
selector:
  strimzi.io/cluster: my-connect-cluster
  strimzi.io/kind: KafkaConnect
  strimzi.io/name: my-connect-cluster-connect
sessionAffinity: None
type: ClusterIP

The issue is that dynamic logging level changes are only applied to one worker node, depending on the Service status. Setting root level to OFF in the above example, is only applied to my-connect-cluster-connect-0 worker node.

$ curl -s -X PUT -H "Content-Type:application/json" "$BASE_URL/admin/loggers/root" -d '{"level": "OFF"}'
....

$ kubectl logs my-connect-cluster-connect-0 | tail -n1
2023-09-04 14:07:54,689 INFO 10.244.0.1 - - [04/Sep/2023:14:07:54 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.25" 2 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp338765435-42]
$ kubectl logs my-connect-cluster-connect-1 | tail -n1
2023-09-04 14:27:04,695 INFO 10.244.0.1 - - [04/Sep/2023:14:27:04 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.25" 4 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp874981105-74]
$ kubectl logs my-connect-cluster-connect-2 | tail -n1
2023-09-04 14:27:04,697 INFO 10.244.0.1 - - [04/Sep/2023:14:27:04 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.25" 3 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp1143968486-39]

KIP-976 has been recently proposed to allow cluster-wide log level changes with just one request. This is something we could potentially use to easily fix this issue.

Steps to reproduce

No response

Expected behavior

No response

Strimzi version

main

Kubernetes version

1.25

Installation method

No response

Infrastructure

No response

Configuration files and logs

No response

Additional context

No response

[scholzj]

Triaged on 7.9.2023: Fixing this now would be quite a lot of effort (being able to address each pod - especially with StableConnectIdentities disabled). Given nobody else reported this, we should wait for the KIP-976 to be implemented and then use the new flag to update all log levels with one call.

Blocked until the KIP is implemented and released in Kafka.