ELT pipeline using PostgreSQL, Airflow, DBT, Redash and Superset.
·
Report Bug
·
Request Feature
.
Table of Contents
The objective of this project was to migrate an ELT pipeline developed for the week 11 challenge using(MYSQL, DBT, Apache Airflow, and Redash) to a more scalable and robust ELT pipeline. This was accomplished by changing the two main components, namely the MySQL data warehouse to Postgres and the Redash dashboard to Superset.
Tech Stack used in this project
- Clone the repo
git clone https://github.com/ProgrammingOperative/traffic_data_etl
Adminer (formerly phpMinAdmin) is a full-featured database management tool written in PHP. Used to access MYSQL and Postgres Databases.
- Postgres:
Navigate to `http://localhost:8080/` on the browser use `postgres-dbt` server use `testdb` database use `dbtuser` for username use `pssd` for password
Airflow is used for aurchestration and automation.
Navigate to `http://localhost:8080/` on the browser
use `admin` for username
use `admin` for password
DBT is used for cleaning and transforming the data in the warehouses.
- Airflow is used for automation of running and testing dbt models
open terminal and execute `docker-compose run — rm server create_db`
using adminer create a user and grant read access
Navigate to `http://localhost:5000/` on the browser
- navigate to
localhost:8088
to access Airflow
See the open issues for a list of proposed features (and known issues).
Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature
) - Commit your Changes (
git commit -m 'Add some AmazingFeature'
) - Push to the Branch (
git push origin feature/AmazingFeature
) - Open a Pull Request
Distributed under the MIT License. See LICENSE
for more information.
Titus Wachira - wachura11t@@gmail.com
Project Link: https://github.com/ProgrammingOperative/Migrate_traffic_data