Traffic Flow ELT

A data pipeline for Extracting, Loading and Transforming traffic flow data

Data Engineering Traffic Flow Drone Data

A fully dockerized ELT pipeline using PostgreSQL, dbt, Apache Airflow, and Redash.
Explore the docs »

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Contributing
4. License
5. Contact
6. Acknowledgements

About The Project

A completely dockerized ELT pipeline with PostgreSQL for data storage, Airflow for automation and orchestration, DBT for data transformation, and a Redash dashboard connected to the data warehouse.

Built With

Tech Stack used in this project

• PostgreSQL
• Apache Airflow
• dbt
• Redash

Getting Started

Prerequisites

Make sure you have docker installed on local machine.

• Docker
• Docker Compose

Installation

1. Clone the repo

   git clone https://github.com/nahomfix/traffic-flow-ELT.git

2. Navigate to the folder

   cd traffic-flow-ELT

3. Build an airflow image

   docker build . --tag extending_airflow:latest

4. Run the following command once for first time initialization

    docker-compose up airflow-init

5. Run

    docker-compose up

6. Open Airflow web browser

   Navigate to `http://localhost:8000/` on the browser
   activate and trigger load_dag
   activate and trigger transform_dag

7. Access redash dashboard

   Navigate to `http://localhost:5000/` on the browser

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Nahom Bekele - nahom.fix@gmail.com

Acknowledgements

• 10 Academy