Energy Consumption and Air Quality in Europe - ETL Process Plan

Introduction

Welcome to the "Energy Consumption and Air Quality in Europe" project by Team 16. This study investigates the impact of energy consumption patterns, economic factors, and geographic traits on air quality and public health in European countries.

Team Members

• Hazimé-Zayour Maha - maha.hazime-zayour@stud.hslu.ch
• Batschelet Jimena - jimena.batschelet@stud.hslu.ch
• Devdas Samuel - samuel.devdas@stud.hslu.ch

Date

Thursday, November 9, 2023

Project Overview

This project explores how different types of energy consumption in European countries affect air quality. The goal is to identify patterns and potential causal relationships to inform environmental policy and energy management strategies.

Project Goals

• Provide insights for sustainable energy policy development in Europe.
• Contribute to environmental conservation through analysis of energy usage and its ecological impacts.

Research Questions

• Do air pollution levels in European countries correlate with their energy consumption types?
• How do different energy sources relate to major pollutants in Europe, considering GDP and population density?
• Can a model predict future air pollution based on energy trends, GDP growth, city populations, and industrial activities?

Data Extraction Methodology

Utilizing Python libraries like Requests, Selenium, WebDriver, and BeautifulSoup for data extraction. Data cleaning and organization is done using Pandas, and saved into CSV files for analysis.

Data Sources and Allocation

• European Environment Agency, Eurostat, and other databases for air quality and energy consumption data.
• Team members allocated specific variables and sources.

Risk Analysis and Legal Compliance

• Addressing data discrepancies and website structure changes with cross-verification and monitoring.
• Adhering to GDPR and respecting terms set by data providers.

Expected Data Impurities and Cleansing Strategies

• Strategies include handling name variations, missing or wrong values, outliers, duplicates, formatting issues, and unit conversions.

Project Dependencies

• Data Processing: JSON, Pandas, Numpy, Datetime.
• Web Scraping: MechanicalSoupp, BeautifulSoup4, Selenium.
• Database Interaction: MariaDB.
• Data Visualization: Matplotlib etc.
• Machine Learning (Optional): Scikit-learn.

Project Feasibility

Assessment of data collection, analysis, and model development feasibility.

ETL Process Plan

• Extract: Retrieving data using Python scripts.
• Transform: Cleaning and standardizing data.
• Load: Transferring data to MariaDB.
• Integrate: Combining datasets for analysis.
• Analyze: Statistical analysis for insights.
• Visualize: Data visualization using Python.

Conclusion

Aiming to contribute to climate action through data-driven evidence, this project seeks to inform policy-making for a healthier, more sustainable future.