v4.0.0

🎉 OpenSTEF 4.0 has been released!

We merged OpenSTEF V4 into main with the new documentation, and therefore it is fully ready to use for everyone! 🎉

The previous version (V3) has been moved into the release/v3 branch

This means that OpenSTEF V4 can now be installed using just:

pip install openstef

If you really need to stay using V3 for a while, then you need to pin openstef>=3.0,<4 like Daan mentioned before. If you need to upgrade, please look here for a guide: https://openstef.github.io/openstef/user_guide/getting_started/migration.html

The documentation is now on the main branch link: https://openstef.github.io/openstef/, and the V3 documentation has moved to https://openstef.github.io/openstef/v3

Please take a look at the new documentation, it is now full of User Guides, Examples, and more!

What's Changed

• feature(STEF-2297): Setup OpenSTEF 4.0 branch by @egordm in #638
• feature(STEF-2298): [OpenSTEF 4.0] Setup project dev tools by @egordm in #639
• feature(STEF-2299): [OpenSTEF 4.0] Setup CI/CD by @egordm in #640
• feature(STEF-2300): Added contributing guide and docs by @egordm in #641
• feature(629): Added project skeleton, added default pydantic models, added documentation on how to do logging. by @egordm in #643
• feature(#628): Added openstef core package for datasets and other shared code. by @egordm in #645
• feature(#628): Minor adjustments in reference doc generation. by @egordm in #646
• feature(#628): Addition of STEF BEAM by @egordm in #647
• feature(#626): Refactored VersionedTimeSeriesDataset to consist of versioned parts to avoid O(n^2) problem when doing cross dataset operations. by @egordm in #657
• feature(#626): Add feature engineering by @lschilders in #650
• feature(626): Completeness check by @MvLieshout in #655
• feature(626): Flatliner check transform by @MvLieshout in #651
• feature(#626): Moved transforms to the openstef-models package. by @egordm in #659
• feature(#628): Updated package deps. by @egordm in #660
• feature(#626): Added feature pipeline and horizon splitting by @egordm in #664
• feature(#628): Added lag transform. Fixed bug where timeseries dataset part can have duplicate timestamps in index. by @egordm in #665
• feature(#626): Air-related Features Transform by @MvLieshout in #663
• feature(626): Energy domain features by @MvLieshout in #666
• Fixed #656 by updating style guide typing guidelines. And fixed #661 by fixing github links to docs and disabling source button. by @egordm in #667
• feature(#627): Implemented model interfaces by @egordm in #669
• feature(#633): Added model pipeline, interfaces for postprocessing, storage, callbacks. Refactor by reorganizing transforms. by @egordm in #674
• add rIQD and relative pinball loss metrics by @MentReeze in #675
• Add option to plot timeseries without interpolated lines where gaps in data by @MentReeze in #676
• feature(#633): Refactored transform, predictor and stateful bases. Added split component pipeline. by @egordm in #677
• feature(#687): Implemented xgboost model by @egordm in #694
• feature(#687): Implemented gblinear model by @lschilders in #695
• add quantile calibration boxplot by @MentReeze in #682
• feature(#688): add FlatlinerForecaster by @lschilders in #696
• feat(#679): added dimensionality reduction transform by @fleur-petit in #699
• feature: Added n_estimators to GBLinear to control how many iterations to run by @egordm in #700
• add windowed metrics averaged over locations by @MentReeze in #701
• feature(#689): Add basecase forecaster by @lschilders in #702
• fix: various forecasting fixes by @egordm in #703
• feature: Added license check / summary. by @egordm in #704
• feature(#678): Created a new explicit multihorizon type. Added input data splitting. by @egordm in #706
• feature(#691): State serialization fixes. Added validation to MultiHorizon. Standardized dataset saving. by @egordm in #707
• feature(#691): Implemented train-test splitting in model. MLFlow storage, model freshness check, and compare new model against the old one. by @egordm in #708
• feature: add get window versioned in backtest to allow backtesting with versioned data. by @egordm in #710
• feature(#632): Added quantile sorter, confidence interval applicator, and postprocessing pipeline. by @egordm in #709
• fix: Fixed various warnings during tests. by @egordm in #711
• feature(#681): rename transforms and workflows by @egordm in #712
• feature(#681): Added standardized column selection for transforms. Fixed bug in holiday feature adder. Added a list of added columns. by @egordm in #716
• feature(#697): added benchmark for public liander2024 dataset to openstef-beam by @bartpleiter in #717
• Feature/697 liander2024 benchmark forecaster and example by @bartpleiter in #720
• feature(#681): Added workflow preset with some out of the box configuration for short term energy forecasting by @egordm in #723
• feature(#681): OSTEF3 style lags. by @egordm in #725
• feature(#681): Added sample weighting. Updated forecasting workflow. by @egordm in #726
• feature(#690) added feature importance by @egordm in #727
• feature: improve preset data split, added more validation by @egordm in #730
• feature(#718): Removed lambdas from liander 2024 target provider. #728 Made various fixes to improve gblinear performance. Normalization, loss function, sample weighting. by @egordm in #731
• feature: Add sample weighted rCRPS by @MvLieshout in #734
• feature(#728): state save refactor by @egordm in #733
• fix: Retraining data bug Liander 2024 benchmark by @MvLieshout in #740
• added iterative imputer by @fleur-petit in #737
• feature: forecaster hyperparams and eval metrics by @egordm in #746
• fix(STEF-2475) fix and refactor backtesting iction in context of backtestforecasting config for clarity. Added more colors. Fixed data split function to handle 0.0 splits. by @egordm in #747
• feature: Disabled data cutoff by default to be consistent with openstef 3. And other minor improvements. by @egordm in #748
• fix(STEF-2475): Fixed rolling aggregate adder by adding forward filling and stating support for only one horizon. by @egordm in #750
• fix(#728): Fixed parallelism stability issues, and gblinear feature pipeline. by @egordm in #752
• fix(STEF-2475): Fixed benchmarking issue concering concatting sorted datasets. Fixed sample weight incompatibility with openstef 3. by @egordm in #753
• Feature/685 energy split by @bartpleiter in #754
• change(#635): release changes to docs by @egordm in #757
• chore: replaced alliander emails with lfenergy email. by @bartpleiter in #767
• fix: Improved mlflow to use run names and load proper models for reuse. Fixed time series plotter to use correct sample interval paramter. by @egordm in #768
• fix: fixes in EvaluationPipeline and TimeSeriesPlotter by @lschilders in #769
• feature: added use_median option to flatliner forecaster so it predic… by @bartpleiter in #773
• chore: change radiation unit to Wm-2 by @lschilders in #776
• feature: add standard devation column to ForecastDataset and add it in ConfidenceIntervalApplicator by @lschilders in #778
• feature: nonzero flatliner preset by @lschilders in #777
• feat(mlflow): suppress MLflow emoji URL logs by @egordm in #780
• fix: clip values of wind and solar components to below 0 by @lschilders in #779
• feature: check for model config change and skip model selection by @lschilders in #781
• fix(STEF-2549): Added none check for model end date from mlflow. Added experiment tags. by @egordm in #782
• fix: exclude stdev column from quantile column checking. by @bartpleiter in #783
• feature: added tutorials for basic functionality. Added convenience method for simple openstef baselines. by @egordm in #785
• feature: Selector transform by @lschilders in #786
• feature: add regex pattern matching in FeatureSelection and fix combine bug by @lschilders in #787
• fix: Tutorials readme and templating by @egordm in #790
• fix: handling no valid data in flatliner check by @lschilders in #798
• fix: add backward compatibility regex feature selection by @lschilders in #800
• fix: handle all-nan target during predict by @lschilders in #801
• Add possibility to specify model run for prediction by @ylvab in #797
• isotonic calibration by @fleur-petit in #795
• feature: Added support for evaluation relative to a single date. Useful for production usecases. by @egordm in #806
• fix: restored benchmark results dir and removed openstef 3 comparison. by @bartpleiter in #808
• add inverse frequency sample weighting method by @MentReeze in #807
• chore: add backwards compatibility new SampleWeighter method by @lschilders in #810
• feature(STEF-2717): save last valid rolling aggregate during training by @lschilders in #811
• fix: add getter method for _is_fitted in RollingAggregatesAdder by @lschilders in #812
• Implement-median-model-v4 by @JanMaartenvanDoorn in #793
• fix(STEF-2802): skip RollingAggregatesAdder when no aggregation functions by @egordm in #813
• fix(STEF-2802): anchor mlflow gitignore patterns to repo root by @egordm in #814
• OpenSTEF Meta V0.1 by @Lars800 in #771
• chore: Add publish of openstef-meta by @MvLieshout in #819
• feat(STEF-2702): openstef-meta cleanup & release pipeline fixes by @egordm in #822
• fix(STEF-2702): normalize MLflow tracking URI for Windows compatibility by @egordm in #826
• fix: apply flatliner value to all quantiles by @lschilders in #828
• chore: OpenSTEF-meta Workflow Improvements by @MvLieshout in #829
• fix: Ensemble Forecasting Model postprocessing fit and transform by @MvLieshout in #830
• feature: add apply() and timezone-awareness to AvailableAt by @lschilders in #831
• fix: timezone handling for offset in AvailableAt apply() (matching apply_index()) by @lschilders in #832
• fix: Fix length mismatch bug in combiner fit by @MvLieshout in #834
• feature: add metrics and providers MAE, Completeness and extend rMAE by @lschilders in #833
• chore: GHA workflow security updates by @MvLieshout in #835
• chore: GHA workflow security updates by @MvLieshout in #836
• fix(STEF-2854): handle backtest robustness issues by @egordm in #837
• add shifter transform and tests by @MentReeze in #839
• feat(examples): add custom benchmark examples for BEAM by @egordm in #821
• feat: Expand clipper functionality with replacement by NaN by @majidkhoshrou in #853
• add shifter and feature selection to ensemble (hybrid model) config by @MentReeze in #855
• feature: add helper methods to Quantile and ensure precision safety by @lschilders in #856
• feat(STEF-2869): add nan_features to workflow configs by @MvLieshout in #858
• feat: enable flatliner fallback by @MvLieshout in #859
• fix(STEF-3036): flatliner feature_importances shape mismatch by @MvLieshout in #862
• feat(STEF-3001): added lag load fallback. by @bartpleiter in #860
• fix: handle both 1D and 2D input shapes in XGBoost custom objectives by @Soutehkeshan in #866
• Fix/lags fallback backwards compatibility by @bartpleiter in #863
• test(loss_functions): add XGBoost 3.2 2D input regression tests by @egordm in #867
• feature: Expose max lag days by @MvLieshout in #868
• fix(STEF-3054): preserve preprocessing NaN in restore_target by @MvLieshout in #869
• Keep custom target column when calling forecaster.predict() by @don-alejandrino in #873
• fix(STEF-3054): exclude __-prefixed columns from feature_names by @MvLieshout in #870
• chore: rename ConstantMedianForecaster to ConstantQuantileForecaster by @lschilders in #874
• feat(tuning): Bayesian hyperparameter tuning via Optuna integration by @egordm in #851
• feature: add ModelPerformanceCallback that raises ModelUnderperfomingError by @lschilders in #875
• fix: spelling in ModelUnderperformingError by @lschilders in #876
• Fix standard lag generation when working with multi-horizon data by @don-alejandrino in #878
• feat: integrate ModelPerformanceCallback and ConstantQuantileForecaster into workflows by @lschilders in #879
• docs: integrate myst-nb notebook rendering infrastructure (#884) by @egordm in #885
• feat(models): add ContributionsPlotter for SHAP visualization by @fleur-petit in #877
• docs: fix API reference completeness and navigation (#888) by @egordm in #889
• feature: completeness and input validation checks for ConstantQuantileForecaster by @lschilders in #890
• docs(tutorials): add inline API links and contextual cross-references by @egordm in #891
• docs: navigation structure with embedded tutorial pages by @egordm in #893
• fix(docs): ensure backtesting tutorial downloads dataset before use by @egordm in #895
• fix(docs): add kaleido for notebook plot rendering in CI by @egordm in #896
• docs: update front page with proper links and openstef-meta by @egordm in #897
• docs: add datasets tutorial notebook by @egordm in #899
• docs: auto-update documentation pages for release release/v4.0.0 by @egordm in #902
• chore(release): retarget CI and docs deploy for main by @egordm in #904
• merge: bring main into v4 (drop v3 content, clear PR #637 conflicts) by @egordm in #905
• release(4.0.0): OpenSTEF 4.0.0 tracking branch. by @egordm in #637

New Contributors

• @fleur-petit made their first contribution in #699
• @Soutehkeshan made their first contribution in #866

Full Changelog: v3.4.93...v4.0.0