This is a simple Go web server that implements a Kafka producer and consumer using the Fiber web framework. The server has two endpoints: one for sending data to the Kafka producer and another for receiving data from the Kafka consumer.
Before running the application, you need to have the following installed:
- Golang (https://go.dev/dl/)
- Kafka (https://kafka.apache.org/downloads)
- Zookeeper (https://zookeeper.apache.org/releases.html)
Clone the repository:
git clone https://github.com/Sahil-4555/go-kafka.git
Install the required dependencies:
go mod tidy
Make sure you have Kafka running on localhost:9092 or update the kafkaBrokers variable in main.go with your Kafka broker addresses.
Apache Kafka relies on ZooKeeper for maintaining metadata. You need to start ZooKeeper before starting Kafka.
Open a terminal in the folder where Kafka is downloaded, and then run ZooKeeper using the following command:
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Open another terminal in the folder where kafka is downloaded, and start the kafka server using the following command:
.\bin\windows\kafka-server-start.bat .\config\server.properties
If You want to consume and print messages from the Kafka topic in real-time, starting from the beginning of the topic, while connecting to the Kafka broker running on localhost:9092. run the following command in another terminal:
.\bin\windows\kafka-console-consumer.bat --topic <topic name> --bootstrap-server localhost:9092 --from-beginning
To run the application, execute the following command:
go run main.go
The server will start listening on http://localhost:3000.
To send data to the Kafka producer, make a GET request to the /producer endpoint with a message as a parameter.
// GET METHOD
http://localhost:3000/producer/:message
To receive data from the Kafka consumer, make a GET request to the /consumer endpoint. The endpoint will respond with the last received message from the Kafka consumer or a default message if no message is available within 4 seconds.
// GET METHOD
http://localhost:3000/consumer
The application uses the Fiber web framework to handle HTTP requests. When a client sends a GET request to the /producer endpoint with a message, the message is sent to the Kafka producer via the producerMessages channel.
The Kafka producer, running in a separate goroutine, reads messages from the producerMessages channel and sends them to the Kafka topic. The producer introduces a random delay between 1 to 3 seconds for message push to simulate real-world scenarios.
The Kafka consumer, also running in a separate goroutine, continuously listens for new messages from the Kafka topic. When a new message is received, it is added to the internal messages slice using a mutex to ensure concurrent-safe access.
The /consumer endpoint, upon receiving a GET request, checks the consumerMessages channel for any available message within 4 seconds. If a message is available, it is sent as a response. Otherwise, a default message is sent indicating that no messages are available at the moment.
The global counter counter is used to keep track of the messages sent to the Kafka producer.