Big Data in AWS

This project holds a Demo in AWS using Glue and Redshift and some other things:

What is my plan?

1. Setting up some resources in AWS using Terraform:
2. Have up a service for Publishing and subscribers
   1. Write that data to an Storage
      1. Process that data to have batch analytics
   2. Get this data and make a Near real time analytics dashboard
3. Crete a replication of PostgresSQL Database to Kafka in near real-time using Debezium.

---

Objective architecture

Requirements

• Add .env file with the following values in the root folder:
  • AWS_SECRET_ACCESS_KEY="XXX"
  • AWS_ACCESS_KEY_ID = "YYY"
• Docker
• Minikube
• Task

Demo

• Plan the infrastructure and then apply it

  task infrastructure:plan

  task infrastructure:apply

• Deploy the streaming kafka application following the README.md instructions in this repository or execute the following line with the resume of it.

  task application:generate-app

• Lets take those records to AWS Kinesis...

  task application:kafka-to-kinesis