Weather Forecasting using iTransformer

This project implements a time series forecasting model using the iTransformer architecture to predict weather parameters. The implementation is based on the paper "iTransformer: Inverted Transformers Are Effective for Time Series Forecasting" (https://arxiv.org/abs/2310.06625).

Project Structure

├── data/                           # Data directory
├── itransformer.py                 # Core iTransformer implementation
├── test.py                        # Testing and evaluation script
├── train.py                       # Training script
├── weather_dataloader.py          # Data loading utilities
├── requirements.txt               # Project dependencies
├── best_model_kaggle.pt          # Trained model weights
└── actual_vs_predicted_with_input.png  # Visualization of predictions

Model Architecture

The iTransformer implementation consists of several key components:

1. Temporal Layer Normalization

   • Implements instance normalization specifically for time series data
   • Normalizes across the temporal dimension for each sample and channel

2. Multi-Head Attention

   • Custom implementation of multi-head attention mechanism
   • Handles time series data with channel-wise attention
   • Includes query, key, value projections and output projection

3. Encoder Layer

   • Combines self-attention with feed-forward network
   • Uses temporal layer normalization
   • Implements residual connections

4. Main iTransformer Model

   • Input projection from sequence length to model dimension
   • Stack of encoder layers
   • Output projection for prediction

Features

• Multi-variate time series forecasting
• Support for multiple weather parameters
• Configurable model architecture (number of heads, layers, dimensions)
• Comprehensive evaluation metrics (MSE, RMSE, MAE)
• Visualization tools for comparing predictions

Implementation Details

Model Parameters

• Input sequence length: 96 time steps
• Output sequence length: 96 time steps
• Model dimension: 256
• Number of attention heads: 1
• Number of encoder layers: 5
• Dropout rate: 0.1

Weather Parameters

The model processes 19 weather-related features:

• Pressure (p)
• Temperature (T)
• Potential temperature (Tpot)
• Dew point temperature (Tdew)
• Relative humidity (rh)
• Maximum vapor pressure (VPmax)
• Actual vapor pressure (VPact)
• Vapor pressure deficit (VPdef)
• Specific humidity (sh)
• Water vapor concentration (H2OC)
• Air density (rho)
• Wind velocity (wv)
• Maximum wind velocity (max. wv)
• Wind direction (wd)
• Rainfall (rain)
• Short-wave downward radiation (SWDR)
• Photosynthetically active radiation (PAR)
• Maximum PAR (max. PAR)
• Logarithmic temperature (Tlog)

Usage

1. Install dependencies:

pip install -r requirements.txt

2. Training:

python train.py

3. Testing and Evaluation:

python test.py

Evaluation

The model's performance is evaluated using:

• Mean Squared Error (MSE)
• Root Mean Squared Error (RMSE)
• Mean Absolute Error (MAE)

The test script generates visualizations comparing actual vs. predicted values for temperature (Tlog) over time.

Results

The model's predictions are visualized in actual_vs_predicted_with_input.png, showing:

• Input temperature sequence
• Actual future temperature
• Predicted future temperature

Dependencies

Key dependencies include:

• PyTorch
• NumPy
• Matplotlib
• Pandas

See requirements.txt for the complete list of dependencies.

License

This project is open source and available under the MIT License.