Repository navigation
SOLETE v4.0
Version 4 merges the dataset cleaning pipeline and the forecasting platform into one repository, and ships the quality-controlled v4 dataset.
Data: https://doi.org/10.11583/DTU.17040767 (unzip into data/, keeping the hdf5/ and parquet/ folders; see data/README.md).
Highlights
- Reproducible v4 dataset release.
dataset/pipeline/build_release.pybuilds the ten HDF5 and Parquet files (1sec, 1min, 5min and 60min, plus an original 1-second file with the nine measured columns), with checksums, manifest, methodology and verification. Builds are resumable and can run in parallel with--jobs N. - One QC vocabulary. Every correction is flagged with a code from
solete/qc_codes.pyrather than applied silently. Measured columns are preserved, and model-derived columns are recomputed per resolution, never averaged up. - Forecasting platform:
- Models and baselines: hybrid wind/solar forecasting and ramp-rate analysis, probabilistic PV and wind forecasting (LightGBM quantile regression with pinball loss, CRPS, coverage and sharpness), and a benchmark leaderboard with a fixed train/val/test split in
benchmarks/. - Model serving:
PrepareMLmodelnow saves the fitted scaler and a metadata manifest.solete.modeling.load_model_bundleandsolete.inference.predict_with_bundlescore new rows without retraining. A worked example isscripts/quickstart/predict_with_saved_model.py. - Configuration:
MLForecasting.pyis configured by JSON file and CLI flags, with no source editing. Running it without arguments reproduces the previous defaults. SOLETEclass:solete.dataset.SOLETEnow supports every resolution and any horizon, and can read eitherv3orv4data.
- Models and baselines: hybrid wind/solar forecasting and ramp-rate analysis, probabilistic PV and wind forecasting (LightGBM quantile regression with pinball loss, CRPS, coverage and sharpness), and a benchmark leaderboard with a fixed train/val/test split in
- Examples and loaders: five notebooks that run on a bundled sample with no download, plus MATLAB and R loaders in
examples/. - Package layout:
pyproject.tomladded, sopip install -e .works.solete.pathsresolves all data paths from any folder, andSOLETE_DATA_DIRoverrides the data location.
Fixes
- Fixed the naive persistence baseline, which was previously taken from
Xscaler-scaled data and inverse-transformed withYscaler. It is now built from the raw series, so earlier persistence numbers are not directly comparable. - Fixed the inverter-capacity clamp, pandas indexing in the thermodynamic model, and scikit-learn metric calls removed in newer versions.
Rincon_Pombo_ThermodynamicModelis about 2.6× faster on the 60-minute data, with bit-for-bit identical output.- Train/val/test slicing now uses half-open intervals, so no rows are silently dropped at sub-hourly resolutions.
- Fixed example notebook paths and Colab download destinations. All five notebooks run without error.
Breaking changes and migration
Functions.pyis removed. Replacefrom Functions import Xwithfrom solete.<module> import X. See "Code layout" in CONTRIBUTING.md.- New v4 file names:
SOLETE_Pombo_<resolution>_v4.<ext>, using60minrather than1h. The original 1-second file isSOLETE_Pombo_1sec_original_v4.h5. - Columns and flags:
- The constant
Azimuth[deg]_qcandElevation[deg]_qccolumns are removed. Code 8 stays reserved. - The 1-second file has 28 columns; coarser files have 36.
- QC codes follow the new vocabulary. Old codes 1 and 3 are now code 11 on v3 files.
- The released files never overwrite the measured
P_Solar[kW]; the cleaned series isP_Solar_clean[kW].
- The constant
generate_persistence()is removed.PreProcessDatasetbuilds the persistence baseline directly.- ANN models: the unused
Maskinglayers were removed fromtrain_LSTMandtrain_CNN_LSTM, andEarlyStoppingis on by default for LSTM, CNN and CNN-LSTM. clean_solete_1sec.pyneeds time-sorted input. The previous whole-file behaviour is available with--in-memory.
Data note
P_Gaia[kW] is zero for 99.56% of the full record, so wind and hybrid forecasting results should be read as methodology rather than evidence about wind behaviour. See KNOWN_ISSUES.md.
Install and test
pip install -r requirements.txt
pip install -e .
pytest
Target interpreter: Python 3.13. Full history is in CHANGELOG.md.