[ENH] probabilistic forecasting rework part 2 - distribution forecast metrics log-loss, CRPS #4276
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fkiraly
changed the title
7 tasks
fkiraly
added a commit
that referenced
this pull request
Mar 18, 2023
…ability distributions (#4190) This experimental PR introduces backend agnostic probability distributions, based on `BaseObject`, towards #4359 Currently, `BaseForecaster.predict_proba` relies on `tensorflow_probability` as a return type, which is a heavy dependency (>300MB in the python env). This PR, instead, introduces a `BaseObject`-based interface for probability distributions, which is back-end agnostic, but has `tensorflow-probability` as one of the options for a back-end. The conceptual model mixes indexed objects as in `pandas` and the array-based distributions in `tensorflow-probability`, i.e., array distributions which have `pandas`-based `index`, `columns`, and can be sub-set using `loc` and `iloc` indexing, similar to `pd.DataFrame`. Advantages: * row/column subsetting of the return, compatibility with `pandas` indices * decouples the distributions from the `tensorflow` back-end, foundation for distribution metrics/losses * allows to add methods to the distribution outside `tensorflow`, in `sktime`, e.g., heuristical integrated cdf or energy statistics required for CRPS metric (see `energy` method) STEP: sktime/enhancement-proposals#31 Related issue: #1746 Includes a full test suite for the interface, based on the `skbase` design. Roadmap for subsequent PR: * part 2: probabilistic (distributional) metrics, see #4276 * part 3: use in forecasters as return object if `tensorflow-probability` is not present; deprecation for 0.18.0 or 0.19.0 * part 4: use in tuned forecasters * further probability distributions native to `sktime` * interfacing more distributions from `tfp`
fkiraly
added
the
module:probability&simulation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Experimental PR that introduces distribution forecast metrics, based on the back-end agnostic, pandas-like,
skbase-based distribution object interface in #4190.The design allows both approximative calculations and exact calculations, as the responsibility with providing values for certain mathematical expressions rests with the base distribution object.
This is still in a preliminary state, as the base class duplicates probabilistic, non-distributional metrics. Future changes should not change the interface though.