Sensor Data ELT

A fully dockerized ELT pipeline project, using Postgres, MYSQL, dbt, Apache Airflow, Redash and Superset.
Explore the docs »

View Demo · Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Roadmap
5. Contributing
6. License
7. Contact
8. Acknowledgements

About The Project

A fully dockerized using a docker-compose file ELT pipeline using MySQL, PostgreSQL, Airflow, DBT, Redash and Superset. used MySQL and Postgres for data ware house. used DBT for data transforming and airflow for automation and orchestrations. A redash and Superset dasboard is built by connecting it to our data ware .

Built With

Tech Stack used in this project

• Postgres
• MYSQL
• Apache Airflow
• dbt
• Redash
• Superset

Getting Started

Prerequisites

Make sure you have docker installed on local machine.

• Docker
• DockerCompose

Installation

1. Clone the repo

   git clone https://github.com/daniEL2371/sensor-data-ELT.git

2. Run

    docker-compose build
    docker-compose up

3. Open Airflow web browser

   Navigate to `http://localhost:8000/` on the browser
   activate and trigger load_data dag
   activate and trigger migrate_data dag
   activate and trigger dbt_dbt_dag

4. Access your Postgres database using adminar

   Navigate to `http://localhost:8080/` on the browser
   use `postgres` databse
   use `postgres-dbt` server
   use `dbtuser` for username
   use `pssd` for password

5. Access redash dashboard

   open new terminal
   docker-compose run — rm server create_db
   Open pstgres-dbt using adminar
   Create a user for analytics database
   CREATE USER 'redash'@'' IDENTIFIED WITH mysql_native_password BY 'root';
   Grant all privilages for analytics database
   GRANT ALL PRIVILEGES ON analytics.* To 'redash'@'';

   Navigate to `http://localhost:5000/` on the browser
   Login and use your created user on analytics databse to connect to your databse

6. Access Superset dashboard

   Navigate to `http://localhost:8088/` on the browser
   use `root` for username
   use `roor` for password

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Daniel Zelalem - danielzelalemheru@gmail.com

Acknowledgements

• 10 Academy