Data-Engineering-Project

This is a project that experiments with an arbitrary API endpoint that coordinates PostgresSQL Kafka, Spark and Airflow to create a streaming data pipeline. This will be the backbone for future GenAI projects.

Name		Name	Last commit message	Last commit date
airflow_resources		airflow_resources
data		data
kafka		kafka
logs		logs
scripts		scripts
spark		spark
src		src
README.md		README.md
docker-compose-airflow.yml		docker-compose-airflow.yml
docker-compose.yml		docker-compose.yml
requirements.txt		requirements.txt

Navigation Menu

Repository files navigation

Data-Engineering-Project

About

Releases

Packages

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Data-Engineering-Project

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages