Soft goals are blocking partition rebalance

[kyguy]

Problem

It appears possible that partition rebalances can still be blocked by other goals that are not listed in the hard.goal list, aka soft.goals. Is this intended behavior? If not, I am happy to contribute a fix for the issue!

Reproducer

(1) Deploy a Cruise Control monitored Kafka cluster with the following Cruise Control configurations:

cruisecontrol.properties

hard.goals: com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal
...

capacity.json

{
  "brokerCapacities":[
    {
      "brokerId": "-1",
      "capacity": {
        "DISK": "100000",
        "CPU": "100",
        "NW_IN": "9E-4",
        "NW_OUT": ""9.E-4""
      },
      "doc": "This is the default capacity. Capacity unit used for disk is in MB, cpu is in percentage, network throughput is in KB."
    }

(2) Attempt a partition rebalance

curl -vv -X POST "http://127.0.0.1:9090/kafkacruisecontrol/rebalance"

Gives the following response:

Error processing POST request '/rebalance' due to: 
'com.linkedin.kafka.cruisecontrol.exception.OptimizationFailureException: 
[NetworkInboundCapacityGoal] Insufficient capacity for networkInbound (Utilization 0.43, Allowed Capacity 0.00, Threshold: 0.80). 
Add at least 553 brokers with the same networkInbound capacity (0.00) as broker-0. 
Add at least 553 brokers with the same networkInbound capacity (0.00) as broker-0.'.

[sreelu27]

Hi @kyguy, Did you have any discussion to see if its an intended behavior? Is there any plan to fix it if its a bug?

[kyguy]

Not yet @sreelu27, latest discussion on the issue will take place here. I'll take a closer look over next couple days!

[sreelu27]

Hi @kyguy , Any updates on this issue?

[kyguy]

Upon further investigation, it appears that the hard.goals config is just list of goals that Cruise Control requires to be specified in default.goals config or an API requests goals parameter config when generating optimization proposals. The hard.goals config is NOT a list of goals "that must be satisfied" but rather list of goals "that must be executed".

Of the supported Cruise Control goals, specified in the goals config, some "must be satisfied", others "may be left unmet" unsatisfied. The former are referred to as "hard goals" and the latter are referred to as "soft goals" in the Cruise Control wiki [1] Whether or not a supported Cruise Control goal is a "hard goal" or "soft goal" is NOT configurable. Cruise Control hard codes whether a goal is a "hard goal" or a "soft goal" in each goal class. For example, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal class is hard coded as a "hard goal", it must be satisfied even if it is not included in the hard.goals list. Whether we add com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal to our hard.goals list does not change whether that goal is a "hard" or a "soft" goal. The only way to avoid the error listed in the original comment above is to exclude com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal from the default.goals and hard.goals lists. This will ensure that this goal is not used at all to generate optimization proposals.

That all being said, the hard.goals configuration is a bit confusing, having it as a configuration option seems to imply that we can configure which goals are "hard" and which goals are "soft". This is not the case and I am not sure if this behavior was intended or an oversight.

In either case, allowing the configuration of hard and soft goals would be a good enhancement for Cruise Control!

[1] https://github.com/linkedin/cruise-control/wiki/Overview#analyzer

[kyguy]

Closing since this issue stems from a misunderstand of the configurations and documentation

In either case, allowing the configuration of hard and soft goals would be a good enhancement for Cruise Control!

If there is interest, I'll open an RFE for this at a later point