Irradiance forecasting under data transmission constraints using features extracted from a sky-camera. Work completed by Joshua Hammond, Ricardo Lara, Michael Baldea, and Brian Korgel.
A CNN-LSTM model uses tabular features extracted from a sky camera, local meteorological measurements, and the clear sky model to forecast irradiance up to two-hours ahead at 10-minute intervals. Novel contributions include:
- A data-parsimonious approach to irradiance forecasting in contrast to many contemporary models that require high dimensional inputs.
- A noise model inspired by control theory which models the effect of random unmeasured disturbances or dropped features.
- Multiple Irradiance representations. We show strong results when predicting irradiance as the deviation from the persistence prediction at the time of forecast. We believe this creates a normalizing effect allowing the model to learn what inputs are associated with changes from the persistence assumption.
Work on this project is supported by the Industry-Academia partnership "The Center for a Solar-Powered Future by 2050".
If this work helped you, please consider citing the preprint available on ArXiv:
@misc{hammond2024,
title={Short-Term Solar Irradiance Forecasting Under Data Transmission Constraints},
author={Joshua Edward Hammond and Ricardo A. Lara Orozco and Michael Baldea and Brian A. Korgel},
year={2024},
eprint={2403.12873},
archivePrefix={arXiv}
}
}
This repository contains the source code and results for Tabular Solar Forecasting under data transmission constraints. The weights of the final model are saved in the final_model/ directory. Other results data, and the dataset used is available at https://utexas.box.com/v/TabularSolarForecast.
.
├── common_params.py - Stores settings and functions for convenience
├── config.py - stores user-specific API keys
├── sample_config.py - template for config.py to be populated with user-specific values
├── constants.py - Stores commonly used variables for convenience
├── create_env_NoGPU.sh - script to set up environment if no GPU is available
├── create_env.sh - script to set up environemnt if GPU is available
├── create_window_cache.py - Creates cache of timestamps where every parameter has valid data
├── download_data.py - script to download meteorological and sky camera data from the NREL SRRL BMS
├── solarprophet.py - module that contains an object-priented class for pre-processing, training, evaluating, and saving the solar forecasting model.
├── tf_test.py - script to test the environment
├── utils.py - utility functions
├── data/*
│ ├── datetimevalues.h5
│ ├── ideal_series.csv
│ ├── joint_data.csv
│ ├── joint_data.h5
│ ├── literature_results.py
│ ├── neptune_cache.h5
│ └── windows_cache.h5
├── final_model/ - Saved keras weights of final model.
├── notebooks/ - notebooks used during development and figures for reports
└── results/* - Saved model weights and corresponding predictions, true irradiance values, and persistence predictions.
* - available at https://utexas.box.com/v/TabularSolarForecast
Financial support of this work was provided by the Robert A. Welch Foundation (F-1464), the Center for a Solar Powered Future (SPF2050)—an Industry-University Cooperative Research Center (IUCRC) funded by the National Science Foundation (EEC-2052814).
We would like to thank Drs. Herie Soto, Dhruv Aurora, Hadi Jamali-Rad, Pierre Carrette and Ojas Shreikar for their feedback while working on this research.
Data was drawn from the NREL SRRL BMS: Andreas, A.; Stoffel, T.; (1981). NREL Solar Radiation Research Laboratory (SRRL): Baseline Measurement System (BMS); Golden, Colorado (Data); NREL Report No. DA-5500-56488. http://dx.doi.org/10.5439/1052221
Clear sky model predictions were calculated using pvlib
Data is available at https://utexas.box.com/v/TabularSolarForecast. This same data can be generated by running the download_data.py script and other scripts included here.
The included file sample_config.py should be renamed to config.py and should contain values for the neptune project name and api key. This file is included in the .gitignore manifest so that the API key is not committed to a repository by mistake.
Instructions assume a Windows OS with NVIDIA card. Methods should be similar for other operating systems but may require some modifications. If running without a graphics card, the tensorflow installation will be different, and running the model will use the CPU.
Follow the instructions below once the prerequisites are met to create an environment with the required dependencies, or execute the create_env.sh script to automatically do the steps below for an NVIDIA GPU setup or create_env_NoGPU.sh for a CPU installation of TensorFlow.
There may be problems with GPU dependencies on your system. These are the processes I used for my PC and M1 Mac, but your results may vary. For specifics on installation, especially tensorflow, see:
Create conda environment named solarprophet with python 3.8:
conda create -n solarprophet python=3.8
conda activate solarprophetFollow GPU Setup instructions from TensorFlow
conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
pip install --upgrade pip
# Anything above 2.10 is not supported on the GPU on Windows Native
pip install "tensorflow<2.11"
# Check tensorflow connection to GPU
python -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"Install remaining packages
pip install pandas seaborn Pillow scipy dask statsmodels tqdm neptune neptune-tensorflow-keras ipykernel joblib scikit-learn Jinja2
# pytables is a bit finnicky, needs to be installed and upgraded
pip install --user --upgrade tables Create conda environment named solarprophet with python 3.8:
conda create -n solarprophet python=3.8
conda activate solarprophetInstall packages:
python -m pip install tensorflowIf using tensorflow-metal for Apple Silicon or AMD GPUs:
python -m pip install tensorflow-metalInstall remaining packages:
python -m pip install pandas seaborn Pillow scipy dask statsmodels tqdm neptune neptune-tensorflow-keras ipykernel joblib scikit-learn Jinja2
conda install pytablesIf tensorflow installation worked correctly, python tf_test.py should work and start a training bar
SolarProphet is an object-oriented class. When initializing an object, declare any non-default parameters. The parameters accepted are:
# Required Parameters
n_steps_in: int, # number of timesteps used in input
n_steps_out: int, # number of timesteps used in output prediction
selected_features: list = None, # Features to be used as inputs (NOTE: either selected_features or selected_groups should be passed, not both)
selected_groups: list = None, # Groups of features to be used as inputs (NOTE: either selected_features or selected_groups should be passed, not both)
selected_responses: list = [
"GHI"
], # response variable to be predicted (NOTE: format required is a list but currently only one response variable is supported)
scaler_type: str = "minmax", # scaler for raw data {minmax, normalizer, powertransformer, quantiletransformer, robustscaler, standardscaler} (default is minmax)
data_path: str = None, # path to measurement data. Eiter .h5 or .csv format is accepted. See data gathering script for more details on data format (str or path-like)
## Saving and cache parameters (optional)
model_save_path: str = None,
datetimes_cache_path: str = None,
window_cache_path: str = None, # path to saved windows with datetimes, features, responses, etc.
all_past_features: list = constants.PAST_FEATURES,
all_future_features: list = constants.FUTURE_FEATURES,
all_scalar_responses: list = constants.SCALAR_RESPONSES,
all_relative_responses: list = constants.RELATIVE_RESPONSES,
noise_model: int = 0, # noise model to use for unmeasured disturbances (default is 0)
# Model (optional)
model=None, # specified tensorflow model (default will build CNN-LSTM model) (tf model)
# model must have input shape: (batch_size, n_steps_in, n_features)
# model must have output shape: (batch_size, n_steps_out) when multiple regressed features are used: (batch_size, n_steps_out, n_regressed_variables)
model_name: str = None, # name of model if desired. Only letters,numbers, and spaces
# Optimizer and training tuning parameters (optional)
scale_responses: bool = True, # scale responses
epochs: int = 200, # maximum number of epochs to train model (default is 200)
shuffle_training_order: bool = False, # shuffle training order of windows (default is False)
batch_size: int = 1000, # batch size for training (default is 1000 windows)
loss: str = "mae", # loss function for model {mae or mse} (default is mae)
optimizer=None, # custom tf optimizer if desired (default is Adam optimizer)
learning_rate: float = 1e-3, # learning rate may be a static value or a tf.optimizers.schedules object
callbacks=[], # list of tf.keras.callbacks to be used during training (default is empty list)
early_stopping: bool = False, # implement early stopping (default is False)
stopping_patience: int = 50, # (early stopping only) number of epochs to wait without improvement before stopping training (default is 50)
stopping_min_delta: float = 1e-7, # (early stopping only) minimum objective function improvement to reset the patience counter (default is 1e-7)
metrics=None, # list of metrics to be used during training (default is None)
dropout_ratio=0,
fit_verbose: int = 0, # verbosity of model.fit() (default is 0)
# Utility Parameters (optional)
data_cols: list = constants.DATA_COLS, # columns of raw data to be read in (default is constants.CSV_COLS)
feature_groups: dict = constants.FEATURE_GROUPS, # dictionary pairing of feature groups and the features the group contains (default is constants.FEATURE_GROUPS)
scalar_response: list = constants.SCALAR_RESPONSES, #
relative_response: list = constants.RELATIVE_RESPONSES,
seed: int = 42, # random seed for reproducibility used in all rng-based functions (default is 42)
n_job_workers: int = 1, # number of workers to use for parallel processing (default is 1)
# Neptune Parameters (optional)
neptune_log: bool = False, # log results to neptune (default is False)
neptune_run_name: str = None, # name of neptune run (default is None)
tags: list = None, # tags to be added to neptune run (default is None)After initializing an object, SolarProphet has functions to:
- Clean, scale, and format data into input-output pairs as well as re-scale afterward.
- Create or load a keras model with or without the noise model
- Train and save a model
- Evaluate model predictions.
Please see the individual methods within solarprophet.py as well as the examples from model development and refinement in the notebooks/ directory.