Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ENH] ForecastingBayesianSearchCV, skopt interface #3390

Closed
hliebert opened this issue Sep 6, 2022 · 3 comments · Fixed by #4580
Closed

[ENH] ForecastingBayesianSearchCV, skopt interface #3390

hliebert opened this issue Sep 6, 2022 · 3 comments · Fixed by #4580
Labels
enhancement Adding new functionality feature request New feature or request good first issue Good for newcomers implementing algorithms Implementing algorithms, estimators, objects native to sktime module:forecasting forecasting module: forecasting, incl probabilistic and hierarchical forecasting

Comments

@hliebert
Copy link
Contributor

hliebert commented Sep 6, 2022

Would it be possible to add support for Bayesian search, e.g. make skopt.BayesSearchCV accessible through ForecastingBayesSearchCV? Similar to how ForecastingGridSearchCV provides an interface to GridSearchCV from sklearn.

This would be a great help to tune more complex models with large parameter space.
@hliebert hliebert added the enhancement Adding new functionality label Sep 6, 2022
@fkiraly fkiraly added feature request New feature or request implementing algorithms Implementing algorithms, estimators, objects native to sktime module:forecasting forecasting module: forecasting, incl probabilistic and hierarchical forecasting good first issue Good for newcomers labels Sep 11, 2022
@fkiraly
Copy link
Collaborator

fkiraly commented Sep 11, 2022

Yes, great idea!

Perhaps we could, in a second step, also add an abstraction step to what search algorithm is being used.

@fkiraly fkiraly changed the title [ENH] Add suport for BayesianSearchCV [ENH] ForecastingBayesianSearchCV Sep 11, 2022
@fkiraly fkiraly changed the title [ENH] ForecastingBayesianSearchCV [ENH] ForecastingBayesianSearchCV, skopt interface Sep 11, 2022
@MokshitSurana
Copy link

Hi, I'd like to work on this. Just need a quick tip on where to start

@fkiraly
Copy link
Collaborator

fkiraly commented Nov 19, 2022

thanks, @GigaMoksh!

This would be very similar to ForecastingGridSearchCV, that would be a good place to start.

You may also like the "implementing estimators" guide, which has a link to a tutorial video:
https://www.sktime.org/en/stable/developer_guide/add_estimators.html

@fkiraly fkiraly mentioned this issue Feb 18, 2023
Open
4 tasks
@hazrulakmal hazrulakmal mentioned this issue Feb 20, 2023
Closed
8 tasks
@rohan-gt rohan-gt mentioned this issue Apr 19, 2023
Closed
@hazrulakmal hazrulakmal mentioned this issue May 12, 2023
Merged
9 tasks
fkiraly pushed a commit that referenced this issue Jul 13, 2023
@hazrulakmal
[ENH] SkoptForecastingCV - hyperparameter tuning using `scikit-opti…
5e16db8 
…mize` (#4580)

#### Reference Issues/PRs
Fixes #3390 
See also #4188 & #3359
Continuation from #4251 

#### What does this implement/fix? Explain your changes.

Implementing hyperparameter optimisation algorithm from
`scikit-optimize` into sktime forecastingCV. This is a draft PR so works
are in progress.

#### Does your contribution introduce a new dependency? If yes, which
one?

<!--
If your contribution does add a new hard dependency, we may suggest to
initially develop your contribution in a separate companion package in
https://github.com/sktime/ to keep external dependencies of the core
sktime package to a minimum.
-->
Yes, this PR introduces soft dependency, `scikit-optimize` 

#### Architecture design decision
_Sktime Side_
Decision: subclassing `BaseGridCV` rather than directly from
`_DelegatedForecaster`
- to inherit usable methods like `_get_fitted_params` , `_update` and
all the `__init__` setup
- `_fit` method has to be overwritten because the `BaseGridCV` is
restrive nature, it assumes that the collection of hyperparameter
combinations is defined beforehand. this is not possible for iterative
or sequential algorithm where the the collection of hyperparameter
combinations depends on previous iterations.

other things to consider:
1. ensure that implementation also supports `pipeline` object.

_Skopt Side_
there are two possibilities
1. (high-level abstraction) a high-level interface to various
[pre-configured
optimizers](https://scikit-optimize.github.io/stable/modules/minimize_functions.html).
i.e `gp_minimize` etc
3. (low-level abstaction) a low-level interface that controls
`optimizer` directly. `optimizer` is a black box algorithm that tells
which hyperparameter combo to search for.

Decision: go with the second option so that sktime users can interact
directly with the optimizer and control how it behaves.

#### To-do lists
<!--
Please go through the checklist below. Please feel free to remove points
if they are not applicable.
-->
Some notes for myself. 
- [x] get the optimizer initialisation ready for each dict
- [x] `_evaluate_step` method - for fitting and evaluating a sample of
hyperparameter from `optimizer` at each iteration. keep updating the cv
result for every step.
- [x] `_run_search` method - running sequential hyperparameter
optimisation. at each iteration, `optimizer` will suggest the next-best
hyperparameters to search.
- [x] `_fit` method - reranking and updating the attributes
- [x] handle optimisation direction internally 
- [x] pytest
- [x] docstring writing 
- [x] quality_check on param_distribution
- [x] add an interface for custom iteration for each parameter space
(when users give a list of dicts and want to customise the num of
iterations for each parameter space)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement Adding new functionality feature request New feature or request good first issue Good for newcomers implementing algorithms Implementing algorithms, estimators, objects native to sktime module:forecasting forecasting module: forecasting, incl probabilistic and hierarchical forecasting
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ENH] SkoptForecastingCV - hyperparameter tuning using scikit-optimize hazrulakmal/sktime
3 participants
@fkiraly @hliebert @MokshitSurana