Repository navigation
Releases
SOLETE Platform v4.0
SOLETE v4.0
Version 4 merges the dataset cleaning pipeline and the forecasting platform into one repository, and ships the quality-controlled v4 dataset.
Data: https://doi.org/10.11583/DTU.17040767 (unzip into data/, keeping the hdf5/ and parquet/ folders; see data/README.md).
Highlights
- Reproducible v4 dataset release.
dataset/pipeline/build_release.pybuilds the ten HDF5 and Parquet files (1sec, 1min, 5min and 60min, plus an original 1-second file with the nine measured columns), with checksums, manifest, methodology and verification. Builds are resumable and can run in parallel with--jobs N. - One QC vocabulary. Every correction is flagged with a code from
solete/qc_codes.pyrather than applied silently. Measured columns are preserved, and model-derived columns are recomputed per resolution, never averaged up. - Forecasting platform:
- Models and baselines: hybrid wind/solar forecasting and ramp-rate analysis, probabilistic PV and wind forecasting (LightGBM quantile regression with pinball loss, CRPS, coverage and sharpness), and a benchmark leaderboard with a fixed train/val/test split in
benchmarks/. - Model serving:
PrepareMLmodelnow saves the fitted scaler and a metadata manifest.solete.modeling.load_model_bundleandsolete.inference.predict_with_bundlescore new rows without retraining. A worked example isscripts/quickstart/predict_with_saved_model.py. - Configuration:
MLForecasting.pyis configured by JSON file and CLI flags, with no source editing. Running it without arguments reproduces the previous defaults. SOLETEclass:solete.dataset.SOLETEnow supports every resolution and any horizon, and can read eitherv3orv4data.
- Models and baselines: hybrid wind/solar forecasting and ramp-rate analysis, probabilistic PV and wind forecasting (LightGBM quantile regression with pinball loss, CRPS, coverage and sharpness), and a benchmark leaderboard with a fixed train/val/test split in
- Examples and loaders: five notebooks that run on a bundled sample with no download, plus MATLAB and R loaders in
examples/. - Package layout:
pyproject.tomladded, sopip install -e .works.solete.pathsresolves all data paths from any folder, andSOLETE_DATA_DIRoverrides the data location.
Fixes
- Fixed the naive persistence baseline, which was previously taken from
Xscaler-scaled data and inverse-transformed withYscaler. It is now built from the raw series, so earlier persistence numbers are not directly comparable. - Fixed the inverter-capacity clamp, pandas indexing in the thermodynamic model, and scikit-learn metric calls removed in newer versions.
Rincon_Pombo_ThermodynamicModelis about 2.6× faster on the 60-minute data, with bit-for-bit identical output.- Train/val/test slicing now uses half-open intervals, so no rows are silently dropped at sub-hourly resolutions.
- Fixed example notebook paths and Colab download destinations. All five notebooks run without error.
Breaking changes and migration
Functions.pyis removed. Replacefrom Functions import Xwithfrom solete.<module> import X. See "Code layout" in CONTRIBUTING.md.- New v4 file names:
SOLETE_Pombo_<resolution>_v4.<ext>, using60minrather than1h. The original 1-second file isSOLETE_Pombo_1sec_original_v4.h5. - Columns and flags:
- The constant
Azimuth[deg]_qcandElevation[deg]_qccolumns are removed. Code 8 stays reserved. - The 1-second file has 28 columns; coarser files have 36.
- QC codes follow the new vocabulary. Old codes 1 and 3 are now code 11 on v3 files.
- The released files never overwrite the measured
P_Solar[kW]; the cleaned series isP_Solar_clean[kW].
- The constant
generate_persistence()is removed.PreProcessDatasetbuilds the persistence baseline directly.- ANN models: the unused
Maskinglayers were removed fromtrain_LSTMandtrain_CNN_LSTM, andEarlyStoppingis on by default for LSTM, CNN and CNN-LSTM. clean_solete_1sec.pyneeds time-sorted input. The previous whole-file behaviour is available with--in-memory.
Data note
P_Gaia[kW] is zero for 99.56% of the full record, so wind and hybrid forecasting results should be read as methodology rather than evidence about wind behaviour. See KNOWN_ISSUES.md.
Install and test
pip install -r requirements.txt
pip install -e .
pytest
Target interpreter: Python 3.13. Full history is in CHANGELOG.md.
SOLETE Platform v3.0
SOLETE v3.0 — 2023-07-20
⚠️ Corrigendum -- please read before comparing results to earlier versions
Correction from v2.3: versions of the SOLETE Platform up to and including v2.3 contained two bugs:
- In the train / validation / test split of the data, which directly affected accuracy.
- In the postprocessing of results, specifically the RMSE calculation, which affected evaluation quality.
Both are fixed in v3.0. Results computed with v2.3 or earlier are not directly comparable to v3.0+.
Summary
Bug-fix release addressing the two issues above ("the megabug"), plus related cleanup to memory handling and documentation. This was an iterative fix developed over several days (2023-07-15 to 2023-07-20) before being finalized.
Changes
Fixed
- Train/validation/test data splitting bug (see corrigendum above).
- RMSE calculation bug in results postprocessing (see corrigendum above).
- Error computation and postprocessing of results improved as part of the same fix, across all ML models.
Changed
- Data preprocessing updated so the pipeline adapts to different combinations of previous-samples and forecast horizons, for both ensemble and ANN methods.
- Memory handling improved; code simplified.
- Documentation enhanced.
Known issues at this release
None known as of this tag, beyond the corrigendum above (which v3.0 fixes, it does not carry it forward).
SOLETE Platform v2.3
v2.3 changes (2023-07-08)
- Fixed a bug that stopped execution when using the 1-second and 1-minute resolution versions of the dataset.
Known issues at this release (see the v3.0 notes)
- The corrigendum states the train/validation/test splitting and RMSE postprocessing bugs affected "versions up to and including v2.3"
- Treat any v1.0–v2.3 results as not directly comparable to v3.0+ until confirmed otherwise.
SOLETE Platform v2.2
v2.2 changes (2023-06-30)
- Corrected
RunMe_matlab.m: some MATLAB versions reorder the SOLETE file on import; the fix re-sorts by timestamp and also makes the timestamp column easier to convert to MATLAB'sDateTimetype.
Known issues at this release (see the v3.0 notes)
- The corrigendum states the train/validation/test splitting and RMSE postprocessing bugs affected "versions up to and including v2.3"
- Treat any v1.0–v2.3 results as not directly comparable to v3.0+ until confirmed otherwise.
SOLETE Platform v2.1
v2.1 changes (2023-04-27)
- Added
RunMe_matlab.m, a MATLAB script to import the SOLETE dataset directly, for users who don't want to go through Python. - Two
README.mdupdates.
Known issues at this release (see the v3.0 notes)
- The corrigendum states the train/validation/test splitting and RMSE postprocessing bugs affected "versions up to and including v2.3"
- Treat any v1.0–v2.3 results as not directly comparable to v3.0+ until confirmed otherwise.
SOLETE Platform v2.0
Summary
Documentation- and usability-focused release. Expands what the platform lets you do with the dataset and adds basic error handling, without a fundamental rework of the forecasting pipeline.
Changes
- New capability: the dataset can now be further expanded, and the expanded version can be saved and re-imported later.
- Added a function that catches typical/common user errors and reports them with a clearer message, to help new users self-diagnose setup problems.
- Added
CITATION.cffso GitHub surfaces a "Cite this repository" option. - Documentation and comments improved throughout in preparation for this release; release date and TODOs updated.
- Minor
README.mdupdates.
Known issues at this release (see the v3.0 notes)
- The corrigendum states the train/validation/test splitting and RMSE postprocessing bugs affected "versions up to and including v2.3"
- Treat any v1.0–v2.3 results as not directly comparable to v3.0+ until confirmed otherwise.
SOLETE Platform v1.0
Summary
First tagged, citable release of the SOLETE platform: the minimum set of scripts needed to load the SOLETE dataset and reproduce the accompanying "Data in Brief" paper's review materials.
Changes
- Added the initial README describing the SOLETE dataset and repository purpose.
- Added the minimum running scripts needed for paper-review purposes (initial
Functions.py,RunMe.pyequivalents). - Assembled the "Full SOLETE Platform" — the first complete, ready-to-release version of the codebase (merged from a parallel branch).
- Minor README update.
Known issues at this release (see the v3.0 notes)
- The corrigendum states the train/validation/test splitting and RMSE postprocessing bugs affected "versions up to and including v2.3"
- Treat any v1.0–v2.3 results as not directly comparable to v3.0+ until confirmed otherwise.
A Trailer of SOLETE
The SOLETE platform and its twin dataset enable physics informed Solar PV power forecasting. Since the related publications are still under-review and due to the fear of scooping, this release is nothing but a trailer. Users have a tiny piece of the dataset available and a tiny piece of the code. The whole platform will be made public upon publication.
Cheers!
Here you have a copy paste of the ReadME
SOLETE
Author: Daniel Vázquez Pombo - Contact: dvapo@elektro.dtu.dk
LinkedIn: https://www.linkedin.com/in/dvp/
ResearchGate: https://www.researchgate.net/profile/Daniel-Vazquez-Pombo
ORCID: https://orcid.org/0000-0001-5664-9421
This repository is complementary material to a paper covering Solar PV power forecasting [1] and its twin "Data in Brief" article [2]. The objective is to increase the transparency of [1], which is one of the main limitations of Machine Learning in general.
This work was developed under the PhD thesis Energy Management Systems for Isolated Hybrid Power Systems at the Technical University of Denmark (DTU).
Note for reviewers
Whether you are reviewing either one of those papers you should be aware of the following:
- What you can find here is a "trailer" of what will be provided upon acceptance.
- The dataset only contains a minimum number of samples
- There is a file missing called Functions.py which contains the most of the code.
- Files RunMe.py and MlForecasting.py depend on it to be run, but you can see at the moment what they cover.
Dependencies
- Python 3.8.10
- Pandas 1.2.4
- Numpy 1.19.5
- Matplotlib 3.4.2
- Scikit-Learn 0.24.2
- Keras 2.5.0
- TensorFlow 2.5.0
- The SOLETE dataset -> https://figshare.com/s/2c9d8167531e6d611ff3
- Functions.py -> A script that will be added to this git uppon publication of the related papers.
How to use
- Store the SOLETE dataset in the same folder as the scripts from this repository
- Open the RunMe.py file. This allows you to load SOLETE and take a peak at its contents.
- Opend the MLForecasting.py file. This allows you to configure Random Forest (RF), Support Vector Machine (SVM), and three kinds of Artificial Neuronal Networks: Convolutional Neuronal Network (CNN), Long-Short Term Memory (LSTM), and a Hybrid (CNN-LSTM).
- The file itself contains notes explaining how to use it.
- You can basically replicate the study from [1] and build on top.
- Have Fun!
You can of course use your own dataset, you will have to adapt things here and there, but you will be able to reuse most of the code.
How to cite this:
- under construction -
References
[1] D.V. Pombo, H.W. Bindner, S.V. Spataru, P. Sørensen, P. Bacher, Increasing the Accuracy of Hourly Multi-Output Solar Power Forecast with Physics-Informed Machine Learning, Solar Energy. In Press.
[2] D.V. Pombo, O.G. Gehrke, H.W. Bindner, Solete, a 15-month long holistic dataset including: meteorology, co-located wind and solar PV power from Denmark with hourly resolution, Data in Brief. In Press.