A simple Python demo showcasing the use of Kafka with KRaft mode using Poetry for dependency management. This project contains a producer that sends real-time temperature data and a consumer that listens for messages.
- Real-time data streaming with Kafka
- Command-line interface to specify topic and consumer group
- Dockerized Kafka setup using the latest version (KRaft mode, no ZooKeeper needed)
- Poetry for managing Python dependencies
- Pull the Kafka Docker image:
docker pull apache/kafka:3.9.0
- Run the Kafka container:
docker run -p 9092:9092 apache/kafka:3.9.0
-
Clone the repository:
git clone https://github.com/your-username/Kafka-python-Demo.git cd Kafka-python-Demo -
Install dependencies with Poetry:
poetry install
-
Activate the Poetry shell:
poetry shell
The producer sends real-time temperature data to a Kafka topic.
python producer.py -t <topic-name>🔔 Example:
python producer.py -t temp-topicYou will be prompted to enter the location and temperature repeatedly.
The consumer listens to messages from a Kafka topic.
python consumer.py -t <topic-name> -u <consumer-group>🔔 Example:
python consumer.py -t temp-topic -u temperature-consumer-group💡 Output:
Start consuming on topic: temp-topic, usergroup: temperature-consumer-group
New Message: {'loc': 'New York', 'value': '25'}
Kafka-python-Demo/
├── consumer.py # Kafka consumer script
├── producer.py # Kafka producer script
├── pyproject.toml # Poetry dependency file
├── README.md # Project documentation
└── .gitignore # Git ignored files
Contributions are welcome! 🚀 create your pull request on GitHub
- Fork the repository.
- Create your feature branch:
git checkout -b feature/your-feature. - Commit your changes:
git commit -m 'Add your feature'. - Push to the branch:
git push origin feature/your-feature. - Open a pull request.