Set up Kafka cluster in IBM Cloud using Kubernetes and Docker

[authorjapps]

Objective:

• Setup a data pipeline on IBM Cloud
• Start from single node, then scale to multi node
• Source system - PostGres existing(legacy data)
• Target System - PostGres(or MySql) empty DB
  • Then switch to HBase/Hadoop via another Topic(CDC)
  • Read here CDC to Kafka for Real-Time Data Integration
  • Read here Integrating Databases and Kafka with Change Data Capture
• Use Kafka Connect for the Pipe-Line.

Why IBM Cloud?

• For time being, we can use up some free spaces which IBM Cloud offers.
  Once the setup is stable, we can go for a subscription
  @rpayal to expand this section

Why Kubernetes?

• Helps in orchestrating the containerised deployments
  @rpayal to expand this section