Skip to content

arjun-1/kafkagosaur

Repository files navigation

kafkagosaur

deno module deno doc CI codecov

Kafkagosaur is a Kafka client for Deno built using WebAssembly. The project binds to the kafka-go library.

See this article for an overview of its inner workings.

Supported features

  • Writer
  • Reader
  • SASL
  • TLS
  • Deno streams

Examples

For comprehensive examples on how to use kafkagosaur, head over to the provided examples.

KafkaWriter

To write a message, first create KafkaWriter instance using the createWriter function on the KafkaGoSaur instance:

import KafkaGoSaur from "https://deno.land/x/kafkagosaur/mod.ts";

const kafkaGoSaur = new KafkaGoSaur();
const writer = await kafkaGoSaur.createWriter({
  broker: "localhost:9092",
  topic: "test-0",
});

const enc = new TextEncoder();
const msgs = [{ value: enc.encode("value") }];

await writer.writeMessages(msgs);

KafkaReader

To read a message, first create KafkaReader instance using the createReader function on the KafkaGoSaur instance:

import KafkaGoSaur from "https://deno.land/x/kafkagosaur/mod.ts";

const kafkaGoSaur = new KafkaGoSaur();
const reader = await kafkaGoSaur.createReader({
  brokers: ["localhost:9092"],
  topic: "test-0",
});

const readMsg = await reader.readMessage();

Provided examples

To run the provided examples, ensure you have docker up and running. Then start the kafka broker using

make docker-up

To run the writer example

deno run --allow-read --allow-net --unstable examples/writer.ts

To run the reader example

deno run --allow-read --allow-net --unstable examples/reader.ts

Documentation

The API documentation is hosted here.

Development

To build the WebAssembly module, first run

make build-wasm

To run the tests, ensure first you have docker up and running. Then start the kafka broker using

make docker-up

Then run

make test

Performance benchmarks

The Deno benchmarks are located in bench and can be run via

deno run --allow-read --allow-net --allow-env --unstable bench/reader.ts
deno run --allow-read --allow-net --allow-env --unstable bench/writer.ts

Results

kafkagosaur-perf

kafka-go1 kafkagosaur (DialBackend.Node) kafkagosaur (DialBackend.Deno)
fetchMessage 5236 ± 743 2760 ± 376 2678 ± 361
writeMessages2 5333 ± 148 4455 ± 159 N/A3

Environment

Contributing

Kafkgagosaur is in early stage of development. Nevertheless your contributions are highly valued and welcomed! Feel free to ask for new features, report bugs, or submit your code.

Footnotes

  1. Using a single goroutine.

  2. Batching 10.000 messages.

  3. DialBackend.Deno not yet functional for writing.