CruiseControl CPU Utilization Computation

[roland-haeusler]

Describe the bug
When trying to do a rebalance with Cruise Control, we always get an OptimizationFailureException saying the CpuCapacityGoal cannot be satisfied and we should add more brokers. The Utilization Computation seems off. The CruiseControl REST API Call to /kafkacruisecontrol/load shows that it assumes our brokers have 1 core, which may be the cause of the malcomputation.

In CruiseControl you can set a num.cores for broker capacity (https://github.com/linkedin/cruise-control/blob/migrate_to_kafka_2_4/config/capacityCores.json), but in Strimzi, this is not possible (https://strimzi.io/docs/operators/latest/using.html#type-CruiseControlSpec-reference)

To Reproduce
Steps to reproduce the behavior:

1. Add cruiseControl to Kafka CR cruiseControl: {}
2. Create KafkaRebalance with spec: {}
3. Wait to see Status:
   Error for request: cluster-main-cruise-control.kafka-devl.svc:9090/kafkacruisecontrol/rebalance?json=true&dryrun=true&verbose=true&skip_hard_goal_check=false. Server returned: Error processing POST request '/rebalance' due to: 'com.linkedin.kafka.cruisecontrol.exception.OptimizationFailureException: [CpuCapacityGoal] Insufficient capacity for cpu (Utilization 858.79, Allowed Capacity 600.00, Threshold: 1.00). Add at least 3 brokers with the same cpu capacity (100.00) as broker-0. Add at least 3 brokers with the same cpu capacity (100.00) as broker-0.'.

Expected behavior
Expect to see a "ProposalReady" Status in the KafkaRebalance CR, something like:
Status: Conditions: Last Transition Time: 2020-05-19T13:50:12.533Z Status: ProposalReady Type: State Observed Generation: 1 Optimization Result: Data To Move MB: 0 Excluded Brokers For Leadership: Excluded Brokers For Replica Move: Excluded Topics: Intra Broker Data To Move MB: 0 Monitored Partitions Percentage: 100 Num Intra Broker Replica Movements: 0 Num Leader Movements: 0 Num Replica Movements: 26 On Demand Balancedness Score After: 81.8666802863978 On Demand Balancedness Score Before: 78.01176356230222 Recent Windows: 1 Session Id: 05539377-ca7b-45ef-b359-e13564f1458c

Environment (please complete the following information):

• Strimzi version: AMQ Streams 1.8 (Strimzi 0.24)
• Installation method: YAML Files
• Kubernetes cluster: Openshift 4.8.13 (Kubernetes v1.21.1
• Infrastructure: Openshift on AWS EC2

YAML files and logs

Kafka CR (attached in the zip file)

KafkaRebalance CR (attached in the zip file)

REST Call to /kafkacruisecontrol/load
`curl https://cluster-main-cruise-control-kafka-devl.apps.ocp4-prod1.helvetia.io/kafkacruisecontrol/load

                                                      HOST         BROKER         RACK         DISK_CAP(MB)            DISK(MB)/_(%)_            CORE_NUM         CPU(%)          NW_IN_CAP(KB/s)       LEADER_NW_IN(KB/s)     FOLLOWER_NW_IN(KB/s)         NW_OUT_CAP(KB/s)        NW_OUT(KB/s)       PNW_OUT(KB/s)    LEADERS/REPLICAS

cluster-main-kafka-0.cluster-main-kafka-brokers.kafka-devl.svc, 0,eu-central-1c, 512000.000, 22101.055/04.32, 1, 113.978, 10000.000, 2.142, 0.518, 10000.000, 5.632, 7.537, 725/1809
cluster-main-kafka-1.cluster-main-kafka-brokers.kafka-devl.svc, 1,eu-central-1a, 512000.000, 23908.867/04.67, 1, 198.530, 10000.000, 2.703, 8.209, 10000.000, 6.910, 142.473, 747/1999
cluster-main-kafka-2.cluster-main-kafka-brokers.kafka-devl.svc, 2,eu-central-1b, 512000.000, 23526.426/04.60, 1, 72.533, 10000.000, 6.253, 4.443, 10000.000, 13.036, 23.018, 693/2006
cluster-main-kafka-3.cluster-main-kafka-brokers.kafka-devl.svc, 3,eu-central-1b, 512000.000, 17625.527/03.44, 1, 35.741, 10000.000, 6.081, 1.061, 10000.000, 123.725, 245.161, 736/1879
cluster-main-kafka-4.cluster-main-kafka-brokers.kafka-devl.svc, 4,eu-central-1c, 512000.000, 17014.631/03.32, 1, 272.399, 10000.000, 0.509, 9.539, 10000.000, 238.128, 371.253, 643/1797
cluster-main-kafka-5.cluster-main-kafka-brokers.kafka-devl.svc, 5,eu-central-1a, 512000.000, 17739.572/03.46, 1, 189.939, 10000.000, 0.825, 0.846, 10000.000, 2.193, 5.035, 683/1734`

[roland-haeusler]

(Updated: Sry uploaded empty files the first time...)
kafkarebalance.zip

[fvaleri]

Hi @roland-haeusler, configuring CPU capacity as number of cores is optional and useful when you have heterogeneous CPUs, which is not the case with Strimzi. What you see from kafkacruisecontrol/load is the default value, which is not used:

https://github.com/linkedin/cruise-control/blob/migrate_to_kafka_2_4/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/config/BrokerCapacityInfo.java#L26

We set CPU capacity as a percentage of each CPU/core (100% by default), which I think should be multiplied by the number of actual Kafka CPUs in order to get the overall capacity (CC assumes same cores on each broker).

https://github.com/strimzi/strimzi-kafka-operator/blob/main/cluster-operator/src/main/java/io/strimzi/operator/cluster/model/cruisecontrol/Capacity.java#L21

The CPU utilization computation is correct, as we are setting CC in "Kubernetes mode", which is needed when running in a container environment (see linkedin/cruise-control#1277).

https://github.com/strimzi/strimzi-kafka-operator/blob/main/cluster-operator/src/main/java/io/strimzi/operator/cluster/model/KafkaBrokerConfigurationBuilder.java#L97

Insufficient capacity for cpu (Utilization 858.79, Allowed Capacity 600.00, Threshold: 1.00). Add at least 3 brokers with the same cpu capacity (100.00) as broker-0. Add at least 3 brokers with the same cpu capacity (100.00) as broker-0.

It looks like you already raised spec.cruiseControl.config."cpu.capacity.threshold" to 100% (default is 70%), but this doesn't help, as well as setting Kafka resources with requests == limits.

---

The problem seems to be in how Strimzi/CC computes the allowed CPU capacity, which assumes that only 1 CPU/core is available (no matter how many Kafka CPU/cores you configure).

I was able to reproduce the issue using a cluster of 3 brokers with 2 CPUs each and 90% threshold (overall: 600 capacity, 540 allowed capacity) and sending a good amount of records in order to trigger CPU utilization. For debugging purpose, I'm also setting broker.metrics.window.ms: 300000, which always give me fresh metrics (default is 1 hour).

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  annotations:
    strimzi.io/rebalance: refresh
  creationTimestamp: "2021-11-28T11:48:55Z"
  generation: 1
  labels:
    strimzi.io/cluster: my-cluster
  name: my-rebalance
  namespace: test
  resourceVersion: "757844"
  uid: 1cb6fa11-7bdb-4016-8645-18fd7870a0cd
spec: {}
status:
  conditions:
  - lastTransitionTime: "2021-11-28T13:31:32.751239Z"
    message: 'Error for request: my-cluster-cruise-control.streams.svc:9090/kafkacruisecontrol/rebalance?json=true&dryrun=true&verbose=true&skip_hard_goal_check=false.
      Server returned: Error processing POST request ''/rebalance'' due to: ''com.linkedin.kafka.cruisecontrol.exception.OptimizationFailureException:
      [CpuCapacityGoal] Insufficient capacity for cpu (Utilization 392.71, Allowed
      Capacity 270.00, Threshold: 0.90). Add at least 2 brokers with the same cpu
      capacity (100.00) as broker-0. Add at least 2 brokers with the same cpu capacity
      (100.00) as broker-0.''.'
    reason: CruiseControlRestException
    status: "True"
    type: NotReady
  observedGeneration: 1

As you can see, the allowed capacity is 270, which is computed as:

    1      (spec.kafka.resources.requests.cpu) 
  * 3      (spec.kafka.replicas) 
  * 100    (spec.cruiseControl.brokerCapacity.cpuUtilization) 
  * 0.9    (spec.cruiseControl.config."cpu.capacity.threshold")

But I have 2 CPUs per-broker and allowed capacity should be 540, enough to satisfy the utilization (392.71).

@tomncooper @kyguy WDYT?

[fvaleri]

The CC code that seems to fail is the following:

https://github.com/linkedin/cruise-control/blob/4e5927b48bf2581ab76acbbecbf42b355b871b65/cruise-control-metrics-reporter/src/main/java/com/linkedin/kafka/cruisecontrol/metricsreporter/metric/ContainerMetricUtils.java#L98

Where we compute the CPU limit as:

double cpuLimit = cpuQuota / getCpuPeriod();

According to the container configuration, this should give 2 in my case (quota/period):

$ kubectl exec my-cluster-kafka-0 -c kafka -- cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
200000
$ kubectl exec my-cluster-kafka-0 -c kafka -- cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
100000

[kyguy]

Hi @roland-haeusler, Strimzi assumes broker pod resources to be homogeneous. Therefore, it is recommended to always set spec.kafka.resources.limits.cpu in the Strimzi Kafka resource so Kubernetes will ensure broker pods are limited to the same CPU capacity. This is especially important when using nodes with heterogeneous resources, e.g. CPU cores.

Thanks for the detailed investigative debugging @fvaleri! Is the spec.kafka.resources.limits.cpu config set to 2 in the Kafka resource? If so, could you post the Kafka resource?

In hindsight, this information should be better documented in the Cruise Control section to avoid confusion, maybe even having spec.kafka.resources.limits.cpu explicitly set in the Cruise Control example yaml by default. I'll make sure this gets done!

[fvaleri]

Thanks for the initial feedback @kyguy.

it is recommended to always set spec.kafka.resources.limits.cpu

Exactly, I would also add that for any stateful application like Kafka, it is always good to have requests==limits to avoid the risk of pods being killed when the node is under pressure and needs some resources back (BestEffort vs Guaranteed QoS).

Is the spec.kafka.resources.limits.cpu config set to 2 in the Kafka resource? If so, could you post the Kafka resource?

Yes. Attached you can find my test Kafka CR.

my-cluster.zip

[fvaleri]

Another example of the same issue. Here it happens on 1 out of 3 brokers (limit.cpu=2, threshold=0.9). As you can see, only 1 of the 2 available CPUs is used to compute capacity limit:

Error for request: my-cluster-cruise-control.test.svc:9090/kafkacruisecontrol/rebalance?json=true&dryrun=true&verbose=true&skip_hard_goal_check=false.
      Server returned: Error processing POST request ''/rebalance'' due to: ''com.linkedin.kafka.cruisecontrol.exception.OptimizationFailureException:
      [CpuCapacityGoal] Utilization (166.85) of host my-cluster-kafka-2.my-cluster-kafka-brokers.test.svc
      violated capacity limit (90.00) for resource cpu. Add at least 1 broker for
      cpu. Add at least 1 broker for cpu.''.

[fvaleri]

@kyguy I think the bug is here:

https://github.com/linkedin/cruise-control/blob/migrate_to_kafka_2_4/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/monitor/sampling/CruiseControlMetricsProcessor.java#L88

The updateCachedNumCoresByBroker always returns default numCpuCores (1) when NOT using num.cores configuration. This behavior doesn't make any sense, but it matches with my tests.

[kyguy]

You are right @fvaleri, those numbers don't add up. I haven't discovered anything definite yet but I wanted to share what I have been thinking about so far.

The updateCachedNumCoresByBroker always returns default numCpuCores (1) when NOT using num.cores configuration. This behavior doesn't make any sense, but it matches with my tests.

This happens because Strimzi generates a CPU percentage config [1] instead of a CPU core config [2] Cruise Control is dependent on Strimzi to provide the number of CPU cores and when Strimzi doesn't, the numCpuCores defaults to 1.

We could attempt to fix this by adding some logic to Strimzi to generate a CPU core config here [3] when the spec.kafka.resources.limits.cpu [3] however we would need a way to convert k8s millicpu values to Cruise Control Integer CPU core values. [2] (e.g. How would we map a 500m value to Cruise Control CPU core value in config? Should it it round to 1? Or should we patch Cruise Control to accept Doubles?) One thing I am wondering is if the Cruise Control capacity value for a broker should ever even be over 100 in the first place? It may be best to continue to treat the Cruise Control capacity as a single virtual CPU comprising of a CPU quota of one or more logical processors.

However, we have one other problem that I am still trying to make sense of and that is:

• How and where the broker CPU utilization value becomes greater than 100??

The CPU utilization metric should initially be between [0.0,1.0] when it is collected here [4] and then from [0,100] when it is multiplied by UNIT_INTERVAL_TO_PERCENTAGE here [5]. but I still haven't found where it could cross 100, and from what I understand, it shouldn't be doing this! This utilization value is what breaks the current computation IMO. There must be something going wrong here [4] but I haven't teased out the issue yet. Maybe nproc is reporting unexpected values in the container? There a couple of things I still need to experiment with

Anyways, I'll continue looking into this tomorrow. If you think or find anything else in the meantime, let me know!

[1] https://github.com/linkedin/cruise-control/blob/4e5927b48bf2581ab76acbbecbf42b355b871b65/config/capacity.json#L7
[2] https://github.com/linkedin/cruise-control/blob/4e5927b48bf2581ab76acbbecbf42b355b871b65/config/capacityCores.json#L7
[3] https://github.com/strimzi/strimzi-kafka-operator/blob/main/cluster-operator/src/main/java/io/strimzi/operator/cluster/model/cruisecontrol/Capacity.java#L34
[4] https://github.com/linkedin/cruise-control/blob/4e5927b48bf2581ab76acbbecbf42b355b871b65/cruise-control-metrics-reporter/src/main/java/com/linkedin/kafka/cruisecontrol/metricsreporter/metric/ContainerMetricUtils.java#L90-L108
[5] https://github.com/linkedin/cruise-control/blob/2.5.73/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/servlet/response/stats/BasicStats.java#L75

[fvaleri]

Thanks @kyguy, I'll continue my analysis as soon as I have some time.

In the meantime, as a workaround, we can exclude CPU goals from the preset hard/default goals like this:

  cruiseControl:
    config:
      hard.goals: >
        com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.MinTopicLeadersPerBrokerGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal
      default.goals: >
        com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.MinTopicLeadersPerBrokerGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.PotentialNwOutGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundUsageDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundUsageDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.TopicReplicaDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderReplicaDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal

This works in my test env with all brokers at around 150% CPU utilization.

[kyguy]

However, we have one other problem that I am still trying to make sense of and that is:

How and where the broker CPU utilization value becomes greater than 100??

I believe I have found the bug here. [1] The value generated here is supposed to be in the interval between [0.0,1.0], however, it is possible for this value to exceed 1.0. Looking at the CPU util formula:

 cpuUtl = (jvmMeasuredCpuUtil * logicalProcessorsOfNode) / (cpuQuota / cpuPeriod)

Plugging in the following numbers from the tests run above:

jvmMeasuredCpuUtil: 1.0 (We will assume this max value for ease of demonstration)
logicalProcessorsOfNode: 4
cpuQuota: 200000
cpuPeriod: 100000

We can see how this formula can produce values that violate the [0.0, 1.0] interval:

cpuUtl = (1.0 * 4) / (200000 / 100000)
cpuUtil = 2.0

The cpuUtil value of 2.0 is outside the expected [0.0, 1.0] interval and the root cause of the incorrect calculations!

Thanks @fvaleri and @roland-haeusler for finding this and helping debug/investigate! We will put in a fix for this for Cruise Control as soon as possible and make sure Strimzi gets updated Cruise Control build. Until then we can stick with the workaround provided by @fvaleri!

[1] https://github.com/linkedin/cruise-control/blob/4e5927b48bf2581ab76acbbecbf42b355b871b65/cruise-control-metrics-reporter/src/main/java/com/linkedin/kafka/cruisecontrol/metricsreporter/metric/ContainerMetricUtils.java#L90-L108

[kyguy]

On second thought, the formula should still hold and provide correct values between the interval of [0.0, 1.0] so long as the behavior of jvmMeasuredCpuUtil is still providing values with respect to the number of logical CPUs available to the node instead of the container.

In the example above, jvmMeasuredCpuUtil shouldn't be able to provide a value greater than 0.5 when the cpuQuota = 200000 and cpuPeriod = 100000 since the JVM would be restricted to use at most 2 CPUs worth of cycles; the max usage of 2 CPUs out of 4 available CPUs is 0.5 utilization overall . However, jvmMeasuredCpuUtil could exceed 0.5 if this method had been patched to be calculated against the logical CPUs available to the container it was run inside of. It's possible this method was patched this way as part of the work done here [1].

That would mean that an upgrade in OpenJDK could have caused the underlying issue. In any case, I am following up the issue with upstream Cruise Control project here [2]

[1] https://bugs.openjdk.java.net/browse/JDK-8226575
[2] linkedin/cruise-control#1747

[fvaleri]

@kyguy thanks for your additional insights.

One fact is that the original use case and all my tests show that the reported available CPU capacity is wrong.

Original use case (brokers=6, requests.cpu=limit.cpu=3, cpu.capacity.threshold=1.0).
Allowed capacity should be 6 * 3 * 100 * 1 = 1500.

Insufficient capacity for cpu (Utilization 858.79, Allowed Capacity 600.00, Threshold: 1.00).

My tests (brokers=3, requests.cpu=limit.cpu=2, cpu.capacity.threshold=0.9).
Allowed capacity should be 3 * 2 * 100 * 0.9 = 540.

# test 1
Insufficient capacity for cpu (Utilization 392.71, Allowed Capacity 270.00, Threshold: 0.90).

# test 2
Utilization (166.85) of host my-cluster-kafka-2.my-cluster-kafka-brokers.test.svc 
violated capacity limit (90.00) for resource cpu.

It is like we are always considering 1 CPU per broker, no matter what you set in Kafka CR (requests.cpu=limit.cpu).

#5951 (comment)

If that is confirmed, a not so easy way to fix this would be to use the ncores capacity configuration type, reconciling that value with what we have in Kafka CR configuration in case of changes.

[kyguy]

There are two issues going on here:

(1) Strimzi Cruise Control always displays 1 CPU per broker

This is strictly a UI issue (it does not cause the rebalance errors) but still needs to be fixed. I can confirm that Strimzi Cruise Control is configured to only ever display 1 CPU core per broker. This single "virtual" CPU core has the cycles of 0 or more CPU cores (however many cores are configured in spec.kafka.resources.limits.cpu). But it has no effect on the rebalance behavior since the ratio between the utilization and capacity remains the same whether we specify the cores explicitly or not. But I do agree that the display is unintuitve, incorrect, and should be fixed!

When we configure the numCore correctly, this error:

Insufficient capacity for cpu (Utilization 858.79, Allowed Capacity 600.00, Threshold: 1.00).

will change to this error:

Insufficient capacity for cpu (Utilization 1717.58, Allowed Capacity 1200.00, Threshold: 1.00).

This is because the utilization and the capacity values are both multiplied by the numCores. This leads us to the second issue:

(2) The CPU utilization value is incorrect

This is causing the rebalance errors. This was introduced with the openjdk 11.0.13 2021-10-19 LTS package where the behavior of getProcessCpuLoad()[2], the method Cruise Control relies on to measure broker CPU utilization, has changed [1]. This change causes Cruise Control to incorrectly calculate CPU utilization values of a broker that are greater than capacity values of that broker! [3] This is incorrect behavior!

The breaking changes of getProcessCpuLoad() were released in openjdk 11.0.13 2021-10-19 LTS. This package was shipped with AMQ Streams 1.8 but not in any released Strimzi versions <= 0.26. So this CPU utilization problem is isolated to AMQ Streams 1.8 for now. It does not affect any released versions of Strimzi <= 0.26, but we need a patch to avoid this issue on Strimzi 0.27. Luckily, the the fix for this issue simply involves setting METRICS_REPORTER_KUBERNETES_MODE to false here [4]

Anyways, @fvaleri I can confirm both issues for AMQ Streams 1.8 but Strimzi should be free of issue (2) and rebalance without error! Regardless, we will get these issues patched for Strimzi!

[1] https://bugs.openjdk.java.net/browse/JDK-8269851
[2] https://github.com/linkedin/cruise-control/blob/4e5927b48bf2581ab76acbbecbf42b355b871b65/cruise-control-metrics-reporter/src/main/java/com/linkedin/kafka/cruisecontrol/metricsreporter/metric/MetricsUtils.java#L406
[3] https://github.com/linkedin/cruise-control/blob/4e5927b48bf2581ab76acbbecbf42b355b871b65/cruise-control-metrics-reporter/src/main/java/com/linkedin/kafka/cruisecontrol/metricsreporter/metric/ContainerMetricUtils.java#L90-L108
[4]

strimzi-kafka-operator/cluster-operator/src/main/java/io/strimzi/operator/cluster/model/KafkaBrokerConfigurationBuilder.java

Line 96 in d440baa

writer.println(CruiseControlConfigurationParameters.METRICS_REPORTER_KUBERNETES_MODE + "=true");

[fvaleri]

Hi @kyguy , thanks for the fix and explanation. Great work!

[kyguy]

Just for transparency, the PR linked above solves the problem (2) listed in the comment above [1] but not problem (1). We will leave this issue open to until problem (1) is addressed!

[1] #5951 (comment)

[fvaleri]

Sure, I guess we can either fix it or simply document it as you have clearly explained.

[kyguy]

Closing issue as this has been resolved by #6892