Skip to content

SOLETE Platform v4.0

Latest

Choose a tag to compare

@DVPombo DVPombo released this 14 Nov 22:56

SOLETE v4.0

Version 4 merges the dataset cleaning pipeline and the forecasting platform into one repository, and ships the quality-controlled v4 dataset.

Data: https://doi.org/10.11583/DTU.17040767 (unzip into data/, keeping the hdf5/ and parquet/ folders; see data/README.md).

Highlights

  • Reproducible v4 dataset release. dataset/pipeline/build_release.py builds the ten HDF5 and Parquet files (1sec, 1min, 5min and 60min, plus an original 1-second file with the nine measured columns), with checksums, manifest, methodology and verification. Builds are resumable and can run in parallel with --jobs N.
  • One QC vocabulary. Every correction is flagged with a code from solete/qc_codes.py rather than applied silently. Measured columns are preserved, and model-derived columns are recomputed per resolution, never averaged up.
  • Forecasting platform:
    • Models and baselines: hybrid wind/solar forecasting and ramp-rate analysis, probabilistic PV and wind forecasting (LightGBM quantile regression with pinball loss, CRPS, coverage and sharpness), and a benchmark leaderboard with a fixed train/val/test split in benchmarks/.
    • Model serving: PrepareMLmodel now saves the fitted scaler and a metadata manifest. solete.modeling.load_model_bundle and solete.inference.predict_with_bundle score new rows without retraining. A worked example is scripts/quickstart/predict_with_saved_model.py.
    • Configuration: MLForecasting.py is configured by JSON file and CLI flags, with no source editing. Running it without arguments reproduces the previous defaults.
    • SOLETE class: solete.dataset.SOLETE now supports every resolution and any horizon, and can read either v3 or v4 data.
  • Examples and loaders: five notebooks that run on a bundled sample with no download, plus MATLAB and R loaders in examples/.
  • Package layout: pyproject.toml added, so pip install -e . works. solete.paths resolves all data paths from any folder, and SOLETE_DATA_DIR overrides the data location.

Fixes

  • Fixed the naive persistence baseline, which was previously taken from Xscaler-scaled data and inverse-transformed with Yscaler. It is now built from the raw series, so earlier persistence numbers are not directly comparable.
  • Fixed the inverter-capacity clamp, pandas indexing in the thermodynamic model, and scikit-learn metric calls removed in newer versions.
  • Rincon_Pombo_ThermodynamicModel is about 2.6× faster on the 60-minute data, with bit-for-bit identical output.
  • Train/val/test slicing now uses half-open intervals, so no rows are silently dropped at sub-hourly resolutions.
  • Fixed example notebook paths and Colab download destinations. All five notebooks run without error.

Breaking changes and migration

  • Functions.py is removed. Replace from Functions import X with from solete.<module> import X. See "Code layout" in CONTRIBUTING.md.
  • New v4 file names: SOLETE_Pombo_<resolution>_v4.<ext>, using 60min rather than 1h. The original 1-second file is SOLETE_Pombo_1sec_original_v4.h5.
  • Columns and flags:
    • The constant Azimuth[deg]_qc and Elevation[deg]_qc columns are removed. Code 8 stays reserved.
    • The 1-second file has 28 columns; coarser files have 36.
    • QC codes follow the new vocabulary. Old codes 1 and 3 are now code 11 on v3 files.
    • The released files never overwrite the measured P_Solar[kW]; the cleaned series is P_Solar_clean[kW].
  • generate_persistence() is removed. PreProcessDataset builds the persistence baseline directly.
  • ANN models: the unused Masking layers were removed from train_LSTM and train_CNN_LSTM, and EarlyStopping is on by default for LSTM, CNN and CNN-LSTM.
  • clean_solete_1sec.py needs time-sorted input. The previous whole-file behaviour is available with --in-memory.

Data note

P_Gaia[kW] is zero for 99.56% of the full record, so wind and hybrid forecasting results should be read as methodology rather than evidence about wind behaviour. See KNOWN_ISSUES.md.

Install and test

pip install -r requirements.txt
pip install -e .
pytest

Target interpreter: Python 3.13. Full history is in CHANGELOG.md.