How about the performance compared with node-rdkafka?

[leolcao]

I am searching some kafka client libraries in node.js, this library seems awesome, and lightweight, and i am only worry about the performance, so does anyone do some benchmark yet?

I am curious about it is not a wrapper from native librdkafka, it just use node.js 'net' and 'tls' to connect kafka cluster.

[Nevon]

We have done benchmarks in isolation just to see what kind of throughput we could achieve, and we are running it in production under heavy load, but we've never done any direct comparisons with other libraries.

So far, it has been fast enough for our needs, so we haven't put any serious effort into optimizations. I would welcome a real comparison between the different libraries to see how we stack up.

[JaapRood]

The library node-rdkafka binds to librdkafka is has got quite a few performance optimisations, mostly in the form of batching messages in both producing and consuming. It's actually done up to such a level that some stuff is pretty hard to work with. Pretty much everything is queued to then be executed in the background. That makes things like producing and committing "synchronously" a lot more difficult, as you need to call other apis and watch for delivery reports, etc. Pretty much all developer experience is sacrificed for message throughput

If necessary, the same strategies could be applied to kafkajs, at which point I reckon you could get pretty far in pure throughput compared to librdkafka. However, given at the level that Node works, I imagine it would never get that far down the performance vs. usability trade-off.

Reality is that you probably won't need it to. The official Java Kafka client doesn't employ strategies up to that extreme point either, and is still used in production everywhere by big tech.

We haven't moved our highest volume topics from node-rdkafka to kafkajs yet, but not for a worry about performance. Unless you already have a big deployment and know exactly what kind of throughput you'll be facing, I wouldn't worry too much about it either.

[leolcao]

In recent days, I tried kafkajs, also node-rdkafka. The scenario what we need is writing a kafka sink connector which can save data from kafka to mongodb, it need detecting the new kafka topics, and call the mongodb native driver to write data. The result is we choose node-rdkafka. It can support to 100 topics in 150 ~ 200ms interval in the same time, in other words, it can consume 500 ~ 600 topics in 1 second. kafkajs seems the much lower performance to subscribe many topics.

[funduck]

Write

I made a comparison
bench_kafka_drivers.txt

Usage difference

node-rdkafka

Config

const rdProducerConf = {
    'metadata.broker.list': url,
    'queue.buffering.max.ms': 10,
    'queue.buffering.max.messages': 100000,
    'queue.buffering.max.kbytes': 1048576
};

And you cant just produce as much as you want, you will catch an error, when internal queue is full - it means you need to poll()

const rdProduceDelay = 100;
let rdProduceAttempt = 0;
const rdProduce = (topic, message) => {
    try {
        producerRD.produce(topic, -1, message, null);
        rdProduceAttempt = 0;
    } catch (e) {
        if (e.message.match(/Queue full/)) {
            rdProduceAttempt++;

            if (rdProduceAttempt == 1) {
                producerRD.poll();
            }

            return when()
            .delay(rdProduceDelay)
            .then(() => {
                return rdProduce(topic, message);
            });
        } else {
            throw e;
        }
    }
};
const produceMessagesRD = (messages) => {
    return when.all(
        messages.map((message) => {
            return rdProduce(TOPIC, message);
        })
    )
};

kafkajs

const produceMessagesJS = (messages) => {
    return producerJS.send({
        topic: TOPIC,
        acks: 0,
        messages: messages.map((message) => {
            return {
                value: message
            };
        })
    })
};

Runs and results

HP laptop core i-5 with Linux mint 18, Node 8.11.1
Started a series of runs with Benchmark.options minSamples 100 and initCount 2

node node/tmp/bench_kafka_drivers.js 1 && node node/tmp/bench_kafka_drivers.js 10 && node node/tmp/bench_kafka_drivers.js 100 && node node/tmp/bench_kafka_drivers.js 1000 && node node/tmp/bench_kafka_drivers.js 10000

Sending single message

N: 1 topic: benchmark
generate 1 buffers x 370,448 ops/sec ±0.60% (190 runs sampled)
generate 1 strings x 517,251 ops/sec ±0.43% (191 runs sampled)
node-rdkafka.produce 1 messages (type Buffer) one by one x 86,086 ops/sec ±16.76% (152 runs sampled)
jafkajs.send 1 messages (type string) at once x 4,924 ops/sec ±2.05% (172 runs sampled)
Fastest is generate 1 strings

Sending 10 at once (in node-rdkafka it is still one by one because there is no bulk method there)

N: 10 topic: benchmark
generate 10 buffers x 40,394 ops/sec ±0.61% (190 runs sampled)
generate 10 strings x 55,626 ops/sec ±0.30% (192 runs sampled)
node-rdkafka.produce 10 messages (type Buffer) one by one x 13,773 ops/sec ±22.75% (156 runs sampled)
jafkajs.send 10 messages (type string) at once x 3,273 ops/sec ±1.49% (175 runs sampled)
Fastest is generate 10 strings

100

N: 100 topic: benchmark
generate 100 buffers x 3,916 ops/sec ±0.68% (188 runs sampled)
generate 100 strings x 5,262 ops/sec ±0.46% (189 runs sampled)
node-rdkafka.produce 100 messages (type Buffer) one by one x 1,480 ops/sec ±22.49% (154 runs sampled)
jafkajs.send 100 messages (type string) at once x 543 ops/sec ±2.74% (171 runs sampled)
Fastest is generate 100 strings

1000

N: 1000 topic: benchmark
generate 1000 buffers x 383 ops/sec ±1.96% (177 runs sampled)
generate 1000 strings x 562 ops/sec ±0.79% (183 runs sampled)
node-rdkafka.produce 1000 messages (type Buffer) one by one x 152 ops/sec ±22.23% (151 runs sampled)
jafkajs.send 1000 messages (type string) at once x 22.72 ops/sec ±1.53% (152 runs sampled)
Fastest is generate 1000 strings

Tried 10000

N: 10000 topic: benchmark
generate 10000 buffers x 40.82 ops/sec ±1.14% (149 runs sampled)
generate 10000 strings x 56.07 ops/sec ±0.84% (154 runs sampled)
node-rdkafka.produce 10000 messages (type Buffer) one by one x 15.35 ops/sec ±18.63% (138 runs sampled)
^C

Run stucked for several minutes, so I removed minSamples 100 and initCount 2

node node/tmp/bench_kafka_drivers.js 10000
N: 10000 topic: benchmark
generate 10000 buffers x 41.56 ops/sec ±1.59% (55 runs sampled)
generate 10000 strings x 56.41 ops/sec ±0.61% (72 runs sampled)
node-rdkafka.produce 10000 messages (type Buffer) one by one x 11.68 ops/sec ±69.16% (30 runs sampled)
jafkajs.send 10000 messages (type string) at once x 0.29 ops/sec ±7.97% (6 runs sampled)
Fastest is generate 10000 strings

In fact, write operation is much faster in node-rdkafka, may because of internal buffer and async delivery, or may be because I found better options for node-kafka producer and did not found any performance improving options (except acks) for kafkajs producer.

Read

I don't have pretty benchmark, so don't want to attach my clumsy code, but the result is that both implementations read messages equally fast (more 100 000 per sec)

Conclusion

I will try to use node-rdkafka for sending messages and kafkajs for receiving because node-rdkafka has issue with consuming (underlying librdkafka lib)

[Nevon]

Interesting results! I'm quite sure that serialization and deserialization is a huge bottleneck for us, that could be improved relatively easily by utilizing buffers in a less wasteful way.

I'm very surprised to see that we are reading messages equally as fast as node-rdkafka. I would suspect that actually it's due to limitations on the broker side, rather than on the consumer side.

[JaapRood]

Having a bit of experience with producing messages in node-rdkafka: the producing API is fully asynchronous and batched, with no promises or callbacks to indicate whether producing was successful. Although I'm sure there's other bottlenecks in KafkaJS, this fire and forget strategy wins you a lot (as described in the reasoning for it it here), but also makes using it way more difficult. The only practical way I've found to get any kind of feedback from the producer is to listen for delivery reports, tagging each message with some unique id and matching that. It might be interesting to try that "synchronous" kind of producing with node-rdkafka and see how that benchmarks.

[funduck]

Well.. producing 10000 messages and then waiting for delivery callbacks for all of them works like this (node-rdkafka producing became 2 times slower)

node node/tmp/bench_kafka_drivers.js 10000
N: 10000 topic: benchmark
node-rdkafka.produce 10000 messages (type Buffer) one by one x 7.34 ops/sec ±33.15% (45 runs sampled)
jafkajs.send 10000 messages (type string) at once x 0.33 ops/sec ±5.98% (6 runs sampled)
Fastest is node-rdkafka.produce 10000 messages (type Buffer) one by one

Making this kind of benchmark is not really correct because poll interval has a huge impact, small batches of messages are delivered almost as fast as big batches, and delivery time almost equals poll interval.
So to receive delivery callbacks without delay I had to reduce poll interval, not sure it is good in production because it decreases profit of batching and internal queue of librdkafka.

[SaiManikantaG]

Having a bit of experience with producing messages in node-rdkafka: the producing API is fully asynchronous and batched, with no promises or callbacks to indicate whether producing was successful. Although I'm sure there's other bottlenecks in KafkaJS, this fire and forget strategy wins you a lot (as described in the reasoning for it it here), but also makes using it way more difficult. The only practical way I've found to get any kind of feedback from the producer is to listen for delivery reports, tagging each message with some unique id and matching that. It might be interesting to try that "synchronous" kind of producing with node-rdkafka and see how that benchmarks.

@JaapRood Lately I some have ended up here looking for alternative to node-rdkafka and also to see other related issues. However just wanted to help out with what I did for making the producer aware of delivery report is as follows:

const generateAcknowledgment = (err, ackMessage) => {
  if (typeof ackMessage.opaque === 'function') {
    ackMessage.opaque.call(null, err, ackMessage);
  }
};

producer.on('delivery-report', generateAcknowledgment);

producer.produce(
        topic,
        partition,
        message,
        key,
        Date.now(),
        (err, delvieryReport) => {
          if (err) {
            return reject(`Error publishing message and error is ${err}`);
          }
          if (delvieryReport) {
            const stringifiedValue: string = delvieryReport.value.toString();
            resp = customResponse(201, {
                partition: delvieryReport.partition,
                offset: delvieryReport.offset,
                key: delvieryReport.key.toString(),
                messageType: 'JSON',
                value: JSON.parse(stringifiedValue),
              });
            producer.removeListener(
              kafkaEvents.PUBLISH_ACKNOWLEDGMENT,
              generateAcknowledgment
            );
            return resolve(resp);
          }
        }
      );

my producer configuration options include:

dr_msg_cb: true, // Enable delivery reports with message payload

[renatocron]

Hello! Sorry for awaking such old thread, but I think it can have a little more hints to what to do

I'm using kafkajs to produce messages with some level of batching (I'm publishing to between 2 and 5 topics at once)
I had set up each topic with 4 partitions and 4 node processes and was getting about 300 messages/s (raw messages, each message can publish N messages, usually is just 1)

I'm using acks=-1 so each message is really persisted before returning

        myLogger.info(`publishing ${msgTotal} msgs to kafka processingTime ${processingTime} encodingTime ${encodingTime} ${JSON.stringify(msgByTopic)}`);
        const beforeSend = Date.now();
        await kafkaProducer.sendBatch({
            topicMessages: topicMessages,
            acks: -1, // all insync replicas
            //compression: CompressionTypes.GZIP
        });
        myLogger.info(`published ${msgTotal} to kafka after ${Date.now() - beforeSend} ms`);

processing/encodingTime usually < 2 ms, so in theory, each worker should process at least 500 messages/s
replicas=3, min.insync.replicas=2, compression.type=gzip (broker only, the compression is disabled on the node as far as I know)

But when I stop the workers, let the queue (rabbitMQ) grow a little, then let de node process start again, some of publish to kafka take 9 seconds:

published 14 to kafka after 1105 ms
published 17 to kafka after 8329 ms
published 14 to kafka after 8441 ms
published 9 to kafka after 9137 ms 
published 60 to kafka after 8348 ms
published 3 to kafka after 9693 ms
published 5 to kafka after 514 ms 
published 3 to kafka after 9539 ms
published 9 to kafka after 9324 ms
published 10 to kafka after 71 ms 

I changed the number of kafka partitions to 40, and set 16 workers and I had no improvement to throughput

I'm running on DigitalOcean k8s, each node has 8 cpu, 16gb ram, I have 3 nodes, hosting 3 kafka brokers and the nodejs processes.
CPU usage during this test reach 20% max, disk 280KB/s max, network 8Mb/s peak

What should I look for? is the latency related just to kafkajs or should I have more kafka-brokers?

Any help is welcome, maybe I should try to batch more messages? or just gave up the sync = -1

--

Edit: with sync=0 and 4 node processes, I can reach 1600 raw messages/s (this load has avg 10 msg published from each original message, so this is 16k/msg/s kafka perspective), so the previous 500 msg/s is actually 5000/s from the kafka perspective