DescribeConfigs for brokers not working with multiple brokers

[jlandersen]

describeConfigs does not work when requesting broker configs and the cluster has multiple brokers. The issue is that the implementation sends a describe config request to all brokers and merges the results. As a result requesting configs for broker X will result in errors from all other brokers than X (see [1]).

I see a couple of things:

1. There should not be a need to send describeConfigs to all brokers. Topic requests can be handled by any broker even if that is not hosting any partitions for that topic.

2. describeConfigs for brokers should send only to the requested brokers. I recently added a sendRequestToBroker as part of Add config entries and replica assignment to create topic #1157 which can be used to send a request to a specific broker in a safe way (handling connection setup if not ready etc.). I also started a PR for describeConfigs in Add describe config to kafkaclient #1142 where this logic was implemented - this might be inspiration for a fix.

[1]
https://github.com/apache/kafka/blob/1.0/core/src/main/scala/kafka/server/AdminManager.scala#L321

Environment

• Node version: any
• Kafka-node version: 4.0.0+
• Kafka version: 0.11.0-2.0

For specific cases also provide

• Number of Brokers: > 1
• Number partitions for topic: any

Include Sample Code to reproduce behavior

const kafka = require("kafka-node");

const client = new kafka.KafkaClient({
    autoConnect: true,
    connectRetryOptions: {
        retries: 1,
    },
    connectTimeout: 3000,
    kafkaHost: '' // any host in a multi-broker cluster
});

const adminClient = new kafka.Admin(client);


client.on("ready", () => {
    const resource = {
        resourceType: adminClient.RESOURCE_TYPES.broker,
        resourceName: '1', // any id in the multi-broker cluster
        configNames: []
    };

    const payload = {
        resources: [resource],
        includeSynonyms: false,
    };

    adminClient.describeConfigs(payload,
        (err, res) => {
        // Error: InvalidRequest: Unexpected broker id, expected 2, but received 1
        console.log(err);
    });
});

[jlandersen]

cc @ahmedlafta
do you wanna take a look otherwise I can take a stab at some point :)

[ahmedlafta]

@jlandersen Oh interesting! Thanks for the tag and helpful explanation! I will try and look into it this weekend 😃

[jlandersen]

@ahmedlafta sorry to poke you, but wanted to check if you had any rough estimates on this? I would also be able to take a look if you want :-)

[ahmedlafta]

hey @jlandersen i apologize for the huge delay. I haven't been able to get to this so please feel free to jump on it if you'd like. let me know if you run into any issues!