Skip to content

Releases

SOLETE Platform v4.0

Choose a tag to compare

@DVPombo DVPombo released this 14 Nov 22:56

SOLETE v4.0

Version 4 merges the dataset cleaning pipeline and the forecasting platform into one repository, and ships the quality-controlled v4 dataset.

Data: https://doi.org/10.11583/DTU.17040767 (unzip into data/, keeping the hdf5/ and parquet/ folders; see data/README.md).

Highlights

  • Reproducible v4 dataset release. dataset/pipeline/build_release.py builds the ten HDF5 and Parquet files (1sec, 1min, 5min and 60min, plus an original 1-second file with the nine measured columns), with checksums, manifest, methodology and verification. Builds are resumable and can run in parallel with --jobs N.
  • One QC vocabulary. Every correction is flagged with a code from solete/qc_codes.py rather than applied silently. Measured columns are preserved, and model-derived columns are recomputed per resolution, never averaged up.
  • Forecasting platform:
    • Models and baselines: hybrid wind/solar forecasting and ramp-rate analysis, probabilistic PV and wind forecasting (LightGBM quantile regression with pinball loss, CRPS, coverage and sharpness), and a benchmark leaderboard with a fixed train/val/test split in benchmarks/.
    • Model serving: PrepareMLmodel now saves the fitted scaler and a metadata manifest. solete.modeling.load_model_bundle and solete.inference.predict_with_bundle score new rows without retraining. A worked example is scripts/quickstart/predict_with_saved_model.py.
    • Configuration: MLForecasting.py is configured by JSON file and CLI flags, with no source editing. Running it without arguments reproduces the previous defaults.
    • SOLETE class: solete.dataset.SOLETE now supports every resolution and any horizon, and can read either v3 or v4 data.
  • Examples and loaders: five notebooks that run on a bundled sample with no download, plus MATLAB and R loaders in examples/.
  • Package layout: pyproject.toml added, so pip install -e . works. solete.paths resolves all data paths from any folder, and SOLETE_DATA_DIR overrides the data location.

Fixes

  • Fixed the naive persistence baseline, which was previously taken from Xscaler-scaled data and inverse-transformed with Yscaler. It is now built from the raw series, so earlier persistence numbers are not directly comparable.
  • Fixed the inverter-capacity clamp, pandas indexing in the thermodynamic model, and scikit-learn metric calls removed in newer versions.
  • Rincon_Pombo_ThermodynamicModel is about 2.6× faster on the 60-minute data, with bit-for-bit identical output.
  • Train/val/test slicing now uses half-open intervals, so no rows are silently dropped at sub-hourly resolutions.
  • Fixed example notebook paths and Colab download destinations. All five notebooks run without error.

Breaking changes and migration

  • Functions.py is removed. Replace from Functions import X with from solete.<module> import X. See "Code layout" in CONTRIBUTING.md.
  • New v4 file names: SOLETE_Pombo_<resolution>_v4.<ext>, using 60min rather than 1h. The original 1-second file is SOLETE_Pombo_1sec_original_v4.h5.
  • Columns and flags:
    • The constant Azimuth[deg]_qc and Elevation[deg]_qc columns are removed. Code 8 stays reserved.
    • The 1-second file has 28 columns; coarser files have 36.
    • QC codes follow the new vocabulary. Old codes 1 and 3 are now code 11 on v3 files.
    • The released files never overwrite the measured P_Solar[kW]; the cleaned series is P_Solar_clean[kW].
  • generate_persistence() is removed. PreProcessDataset builds the persistence baseline directly.
  • ANN models: the unused Masking layers were removed from train_LSTM and train_CNN_LSTM, and EarlyStopping is on by default for LSTM, CNN and CNN-LSTM.
  • clean_solete_1sec.py needs time-sorted input. The previous whole-file behaviour is available with --in-memory.

Data note

P_Gaia[kW] is zero for 99.56% of the full record, so wind and hybrid forecasting results should be read as methodology rather than evidence about wind behaviour. See KNOWN_ISSUES.md.

Install and test

pip install -r requirements.txt
pip install -e .
pytest

Target interpreter: Python 3.13. Full history is in CHANGELOG.md.

SOLETE Platform v3.0

Choose a tag to compare

@DVPombo DVPombo released this 20 Jul 09:59

SOLETE v3.0 — 2023-07-20

⚠️ Corrigendum -- please read before comparing results to earlier versions

Correction from v2.3: versions of the SOLETE Platform up to and including v2.3 contained two bugs:

  1. In the train / validation / test split of the data, which directly affected accuracy.
  2. In the postprocessing of results, specifically the RMSE calculation, which affected evaluation quality.

Both are fixed in v3.0. Results computed with v2.3 or earlier are not directly comparable to v3.0+.

Summary

Bug-fix release addressing the two issues above ("the megabug"), plus related cleanup to memory handling and documentation. This was an iterative fix developed over several days (2023-07-15 to 2023-07-20) before being finalized.

Changes

Fixed

  • Train/validation/test data splitting bug (see corrigendum above).
  • RMSE calculation bug in results postprocessing (see corrigendum above).
  • Error computation and postprocessing of results improved as part of the same fix, across all ML models.

Changed

  • Data preprocessing updated so the pipeline adapts to different combinations of previous-samples and forecast horizons, for both ensemble and ANN methods.
  • Memory handling improved; code simplified.
  • Documentation enhanced.

Known issues at this release

None known as of this tag, beyond the corrigendum above (which v3.0 fixes, it does not carry it forward).

SOLETE Platform v2.3

Choose a tag to compare

@DVPombo DVPombo released this 08 Jul 11:54

v2.3 changes (2023-07-08)

  • Fixed a bug that stopped execution when using the 1-second and 1-minute resolution versions of the dataset.

Known issues at this release (see the v3.0 notes)

  • The corrigendum states the train/validation/test splitting and RMSE postprocessing bugs affected "versions up to and including v2.3"
  • Treat any v1.0–v2.3 results as not directly comparable to v3.0+ until confirmed otherwise.

SOLETE Platform v2.2

Choose a tag to compare

@DVPombo DVPombo released this 30 Jun 21:28

v2.2 changes (2023-06-30)

  • Corrected RunMe_matlab.m: some MATLAB versions reorder the SOLETE file on import; the fix re-sorts by timestamp and also makes the timestamp column easier to convert to MATLAB's DateTime type.

Known issues at this release (see the v3.0 notes)

  • The corrigendum states the train/validation/test splitting and RMSE postprocessing bugs affected "versions up to and including v2.3"
  • Treat any v1.0–v2.3 results as not directly comparable to v3.0+ until confirmed otherwise.

SOLETE Platform v2.1

Choose a tag to compare

@DVPombo DVPombo released this 27 Apr 15:14

v2.1 changes (2023-04-27)

  • Added RunMe_matlab.m, a MATLAB script to import the SOLETE dataset directly, for users who don't want to go through Python.
  • Two README.md updates.

Known issues at this release (see the v3.0 notes)

  • The corrigendum states the train/validation/test splitting and RMSE postprocessing bugs affected "versions up to and including v2.3"
  • Treat any v1.0–v2.3 results as not directly comparable to v3.0+ until confirmed otherwise.

SOLETE Platform v2.0

Choose a tag to compare

@DVPombo DVPombo released this 11 Apr 12:32

Summary

Documentation- and usability-focused release. Expands what the platform lets you do with the dataset and adds basic error handling, without a fundamental rework of the forecasting pipeline.

Changes

  • New capability: the dataset can now be further expanded, and the expanded version can be saved and re-imported later.
  • Added a function that catches typical/common user errors and reports them with a clearer message, to help new users self-diagnose setup problems.
  • Added CITATION.cff so GitHub surfaces a "Cite this repository" option.
  • Documentation and comments improved throughout in preparation for this release; release date and TODOs updated.
  • Minor README.md updates.

Known issues at this release (see the v3.0 notes)

  • The corrigendum states the train/validation/test splitting and RMSE postprocessing bugs affected "versions up to and including v2.3"
  • Treat any v1.0–v2.3 results as not directly comparable to v3.0+ until confirmed otherwise.

SOLETE Platform v1.0

Choose a tag to compare

@DVPombo DVPombo released this 02 Feb 17:27

Summary

First tagged, citable release of the SOLETE platform: the minimum set of scripts needed to load the SOLETE dataset and reproduce the accompanying "Data in Brief" paper's review materials.

Changes

  • Added the initial README describing the SOLETE dataset and repository purpose.
  • Added the minimum running scripts needed for paper-review purposes (initial Functions.py, RunMe.py equivalents).
  • Assembled the "Full SOLETE Platform" — the first complete, ready-to-release version of the codebase (merged from a parallel branch).
  • Minor README update.

Known issues at this release (see the v3.0 notes)

  • The corrigendum states the train/validation/test splitting and RMSE postprocessing bugs affected "versions up to and including v2.3"
  • Treat any v1.0–v2.3 results as not directly comparable to v3.0+ until confirmed otherwise.

A Trailer of SOLETE

Choose a tag to compare

@DVPombo DVPombo released this 18 Nov 11:04

The SOLETE platform and its twin dataset enable physics informed Solar PV power forecasting. Since the related publications are still under-review and due to the fear of scooping, this release is nothing but a trailer. Users have a tiny piece of the dataset available and a tiny piece of the code. The whole platform will be made public upon publication.
Cheers!

Here you have a copy paste of the ReadME

SOLETE

Author: Daniel Vázquez Pombo - Contact: dvapo@elektro.dtu.dk

LinkedIn: https://www.linkedin.com/in/dvp/

ResearchGate: https://www.researchgate.net/profile/Daniel-Vazquez-Pombo
ORCID: https://orcid.org/0000-0001-5664-9421

This repository is complementary material to a paper covering Solar PV power forecasting [1] and its twin "Data in Brief" article [2]. The objective is to increase the transparency of [1], which is one of the main limitations of Machine Learning in general.

This work was developed under the PhD thesis Energy Management Systems for Isolated Hybrid Power Systems at the Technical University of Denmark (DTU).

Note for reviewers

Whether you are reviewing either one of those papers you should be aware of the following:

  1. What you can find here is a "trailer" of what will be provided upon acceptance.
  2. The dataset only contains a minimum number of samples
  3. There is a file missing called Functions.py which contains the most of the code.
  4. Files RunMe.py and MlForecasting.py depend on it to be run, but you can see at the moment what they cover.

Dependencies

  1. Python 3.8.10
  2. Pandas 1.2.4
  3. Numpy 1.19.5
  4. Matplotlib 3.4.2
  5. Scikit-Learn 0.24.2
  6. Keras 2.5.0
  7. TensorFlow 2.5.0
  8. The SOLETE dataset -> https://figshare.com/s/2c9d8167531e6d611ff3
  9. Functions.py -> A script that will be added to this git uppon publication of the related papers.

How to use

  1. Store the SOLETE dataset in the same folder as the scripts from this repository
  2. Open the RunMe.py file. This allows you to load SOLETE and take a peak at its contents.
  3. Opend the MLForecasting.py file. This allows you to configure Random Forest (RF), Support Vector Machine (SVM), and three kinds of Artificial Neuronal Networks: Convolutional Neuronal Network (CNN), Long-Short Term Memory (LSTM), and a Hybrid (CNN-LSTM).
    • The file itself contains notes explaining how to use it.
    • You can basically replicate the study from [1] and build on top.
  4. Have Fun!

You can of course use your own dataset, you will have to adapt things here and there, but you will be able to reuse most of the code.

How to cite this:

  • under construction -

References

[1] D.V. Pombo, H.W. Bindner, S.V. Spataru, P. Sørensen, P. Bacher, Increasing the Accuracy of Hourly Multi-Output Solar Power Forecast with Physics-Informed Machine Learning, Solar Energy. In Press.

[2] D.V. Pombo, O.G. Gehrke, H.W. Bindner, Solete, a 15-month long holistic dataset including: meteorology, co-located wind and solar PV power from Denmark with hourly resolution, Data in Brief. In Press.