Skip to content

This project aim to create a scalable data warehouse that uses ELT( Extract, Load, and transform) using DBT and it will host the vehicle trajectory data extracted by analyzing footage taken by swarm drones and static roadside cameras. We will be using Apache Airflow for scheduling and DBT for data transformation.

Notifications You must be signed in to change notification settings

skevin-dev/Data-warehouse-tech-stack

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

24 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Data-warehouse-tech-stack

Table of Contents
  1. About The Project
  2. Getting Started
  3. License
  4. Contact
  5. Acknowledgements

About The Project

ELT A dockerized Extract, Load, Transform (ELT) pipeline with PostgreSQL, Airflow, DBT, and a Redash.

Built With

Tech Stack used in this project

  • Docker
  • Postgres
  • Airflow
  • DBT
  • Redash

Getting Started

Prerequisites

Make sure you have docker installed on local machine.

  • Docker
  • Docker Compose

Installation

  1. Clone the repo

    git clone https://github.com/skevin-dev/Data-warehouse-tech-stack
  2. Navigate to the folder

    cd Data-warehouse-tech-stack
  3. Run

     docker-compose up
  4. Open Airflow web browser

    Navigate to `http://localhost:8089/` on the browser
    activate and trigger data_load
    activate and trigger dbt_airflow
  5. Access redash dashboard

    Navigate to `http://localhost:5000/` on the browser

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Kevin Shyaka - https://www.linkedin.com/in/shyaka-kevin/

Acknowledgments

About

This project aim to create a scalable data warehouse that uses ELT( Extract, Load, and transform) using DBT and it will host the vehicle trajectory data extracted by analyzing footage taken by swarm drones and static roadside cameras. We will be using Apache Airflow for scheduling and DBT for data transformation.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published