This project describes the setup and execution of the demo for the DOAG 2022 presentation Kafka as your Data Lake - is it feasible?.
The demos are all based on a containerised platform which can easily be provisioned using Docker Compose.
- Preparation - describes how to setup the platform as well as the necessary artefacts, such as Postgresql Schema and Table, Kafka Topics and a data stream of simulated "live" data
- Demo 1 - Streaming Data Lake - shows how to use ksqlDB to create a "real-time" Delta Lake
- Demo 2 - Batch Processing with Spark - shows how to use Spark to use Batch Processing to access data in one or more Kafka topics
- Demo 3 - Batch Query with Presto - show how to use Presto to use SQL Queries to access one or more Kafka topics