ETL_OpenWeatherMap_Ukraine

Pet project to demonstrate knowledge and skills in data engineering

Details about this project. Here are steps to get started:

1. Choose an API that provides dynamic data related to Ukraine. You can use Python libraries such as requests to access the API and retrieve data.

2. Set up SQL Server on your local machine as the data staging area. You can use SQL Server Management Studio to create tables and define the schema for your data.

3. Use Python and libraries such as pandas and pyspark to transform and clean the data. You can also use tools such as Apache Spark and Databricks for processing large amounts of data.

4. Use Airflow as your orchestrator to schedule and manage the ETL process. You can define DAGs (Directed Acyclic Graphs) to manage the dependencies between the different tasks.

5. Use Docker to package and deploy your ETL process. You can define a docker-compose file to specify the services and dependencies required for your application.

6. Use Git to version control your code and collaborate with other developers.

7. Set up SQL Server on your local machine as the data warehouse. You can use SQL Server Integration Services (SSIS) to load data from the staging area to the data warehouse.

8. Use Power BI to visualize the data from the data warehouse. You can connect to the data source and create reports and dashboards to analyze the data.