Skip to content

Latest commit

 

History

History
201 lines (138 loc) · 4.17 KB

README.md

File metadata and controls

201 lines (138 loc) · 4.17 KB
Raw

Learning Kafka and Connection Kafka To Golang Application

Contents

Installation

To install Kafka package, you need to install Go and set your Go workspace first.

  1. You first need Go installed (version 1.18+ is required), then you can use the below Go command to install Kafka.
$ go get -u github.com/confluentinc/confluent-kafka-go/kafka
  1. Import it in your code:
import "github.com/confluentinc/confluent-kafka-go/kafka"
  1. You first need Makefile
# install windows
C:\> choco install make
# install linux
$ sudo apt-get install make
# install MacOS with Homebrew
$ brew install make

Verify Version Makefile

# verify that Makefile is installed correctly by checking the version
$ make --version

  1. You first need Docker Desktop installed.

  2. You first need Docker Compose.

Windows and macOS

  • Docker Compose is included in Docker Desktop for Windows and macOS.

Linux

  • For Ubuntu and Debian, run:
$ sudo apt-get update
$ sudo apt-get install docker-compose-plugin
  • For RPM-based distros, run:
$ sudo yum update
$ sudo yum install docker-compose-plugin

Verify Version Docker Compose

# verify that Docker Compose is installed correctly by checking the version
$ docker compose version
  1. You first need Kafka CLI or Web UI
# install kafka cli, make sure all container running
$ make compose
# install kafka web ui, make sure all container running, wait 1 minutes, then open to browser http://localhost:9021
$ make compose-ui

Quick Start

  • Producer
# application producer, run:
$ make producer
  • Consumer
# application consumer1, run:
$ make consumer1
# application consumer2, run:
$ make consumer2

Kafka CLI

Note that you need to set the corresponding binding tag on Operating System. For example: winpty, sudo or etc

# entry into docker container kafka, run:
$ make exec

Create Topic

$ kafka-topics --bootstrap-server localhost:9092 --create --topic YOURTOPIC

Create Topic with specific configuration

$ kafka-topics --bootstrap-server localhost:9092 --create --topic YOURTOPIC --replication-factor 1 --partitions 3 --config "cleanup.policy=compact" --config "delete.retention.ms=100"  --config "segment.bytes=204800" --config "min.cleanable.dirty.ratio=0.01"

List Topic

$ kafka-topics --bootstrap-server localhost:9092 --list

Delete Topic

$ kafka-topics --bootstrap-server localhost:9092 --delete --topic YOURTOPIC

Alter Topic

$ kafka-topics --bootstrap-server localhost:9092 --alter --topic YOURTOPIC --partitions 3

List Detail Topic

$ kafka-topics --bootstrap-server localhost:9092 --describe --topic YOURTOPIC

Send Message Producer

$ kafka-console-producer --bootstrap-server localhost:9092 --topic YOURTOPIC

Send Message Producer with Key

# example: >1:test
$ kafka-console-producer --bootstrap-server localhost:9092 --topic YOURTOPIC --property "parse.key=true" --property "key.separator=:"

Received Message Consumer

$ kafka-console-consumer --bootstrap-server localhost:9092 --topic YOURTOPIC --from-beginning

Received Message Consumer Group

$ kafka-console-consumer --bootstrap-server localhost:9092 --topic YOURTOPIC --group YOURGROUP --from-beginning

Received Message Consumer Group with Key

$ kafka-console-consumer --bootstrap-server localhost:9092 --topic YOURTOPIC --group YOURGROUP --from-beginning --property "print.key=true"

List Consumer Group Offset

$ kafka-consumer-groups --bootstrap-server localhost:9092 --all-groups --all-topics --describe

List Consumer Group Offset 1

$ kafka-consumer-groups --bootstrap-server localhost:9092 --group YOURGROUP --describe