kafka

Ramp up Plan

What is Kafka

1. Getting started with Kafka
2. Use docker compose to create Kafka cluster
   1. ./start-kafka
3. Connect to kafka
   1. VS Code (From recommended extensions)
   2. UI for Apache Kafka (Already part of local docker-compose.yaml)
4. Create Topics - learn about different options while creating topics
   1. Use extension
   2. Use UI
   3. Use CLI (Hint: exec into broker container)
5. Test cluster using command line
   1. Produce and consume from CLI
   2. Produce from CLI - see the messages in the UI and the VS Code Extension
6. Create python project
   1. Hint: You need to use 3.9
   2. Install confluent-kafka with pip
7. Create simple python producer
8. Create simple python consumer
9. Add method to send batch of messages in producer
10. Add method to receive batch of messages in consumer
11. Create at-least once consumer
12. Create at most once consumer
13. Create exactly once consumer using transactions
    1. Getting started with transactions
14. Create consumer that handles de-duplication
15. Good to know
    1. Storage
    2. Performance
    3. Testing