- Terraform
- Ansible
- Apache Maven
- AWS EC2
- Confluent Kafka
- Zookeeper
- Kafka Broker
- Schema Registry
- Control Center
- Kafka Connect Worker
- Docker
- Kafka-Connect-Spooldir Connector
- Creates a VPC
- Creates a Subnet inside that VPC
- Defines a Security Group for later use
- Defines an Internet Gatway and Configures the routing to use the Internet Gateway
- Spins up ec2 instances inside the Subnet created above behind the defined Security Group
- Installs Docker on the above EC2 instances
- Starts the a Zookeeper Container using Confluent's Zookeeper Image
- Starts the a Kafka Container using Confluent's Enterprise Kafka Image
- On one of the EC2 instances it starts a Schema Registry Container using Confluent's Schema Registry Image
- On one of the EC2 instances it starts a Control Center Container using Confluent's Enterprise Control Center Image
- On one of the EC2 instances it starts a Kafka Connect Container using Confluent's Kafka Connect Image and adds Kafka_Connect_Spooldir connector in it's path
- Modiy Kafka to AutoEnable the topic creation
- Make Kafka Connect Topic Visible in Control Center
- [Not a High Priority] Switch to dynamic ec2 ami lookup
- terraform apply -var-file=variables.tfvars && terraform output ec2_ips > output.txt
- ansible-playbook ansible.yml
- It would be very useful to have a central repository of kafka connect connectors that can be accessed programaticaly to download and place the connectors onto connect worker's path
- Current configuration of this project will be using AWS services that are beyond the Free Tier!