class Tune

[martins0n]

🚀 Feature Request

Implement Tune class for tuning pipelines with Optuna.

Proposal

class Tune(AutoBase):
    pipeline: Pipeline
    target_metric: Metric
    horizon: int
    metric_aggregation: MetricAggregationStatistics = "mean"
    backtest_params: Optional[dict] = None
    experiment_folder: Optional[str] = None
    runner: Optional[AbstractRunner] = None
    storage: Optional[BaseStorage] = None
    metrics: Optional[List[Metric]] = None
    sampler: Optional[BaseSampler] = None
    params_to_tune: Optional[dict] = None  # custom params to tune

    def _init_optuna():
        # init optuna study with TPE sampler by default
        # if sampler is not None, use it
        # if params_to_tune is not None, use it
        # if params_to_tune is None, use Pipeline.params_to_tune
        
    def objective(trial):
        # objective function for optuna with sampling logic
        # return metric value to minimize
        # use pipeline.set_params to set sampled params
        #  params = {
        #     "model.n_iterations": optuna.distributions.CategoricalDistribution((10, 100, 200)),
        #     "transforms.0.mode": optuna.distributions.CategoricalDistribution(("per-segment", "macro")),
        #   }
        # We could sample params via Trial._suggest(params.key, params.value)

Test cases

• add tests cases like for Auto class
• check if it works with other samplers
• check if custom params_to_tune works

Additional context

Is blocked by #1026 #1023 #1025

[GooseIt]

I'd like to take this issue.

[GooseIt]

I have stumbled upon a problem.
In optuna, to optimize a hyperparameter, its range of values must be specified (using .suggest_float(..) and its likes). These ranges are usually set manually and a big challenge in implementing Tune class is to develop a way to propose these ranges automatically.
Is there an inbuilt way to get hyperparameters value range in Etna (maybe there are some descriptions of features are passed which I don't know about)?
If there is not, do you have any algorithm for hyperparameter value range setting on your mind that you think I should implement? I could go browse the web for possible implementation ideas, but I think I should ask for your vision beforehand.

[martins0n]

@GooseIt you're right. we already have a method for getting a range of parameters #1026

So, we by default should just call params_to_tune of Pipeline. And in case we need something to customize we could call Pipeline.set_params additionally

[Mr-Geekman]

@GooseIt, are you doing this task? Do you have any news, problems to discuss?

[GooseIt]

@GooseIt, are you doing this task? Do you have any news, problems to discuss?

Yes, I've done my research, now all I have left is to write the code.
I'm quite busy with my studying assignments, though.
I'll do it by March 28th, 23:59 with my current pace. Is this alright, or do I have to prioritize the task more?

[Mr-Geekman]

It is fine, I just wanted to make sure that task is in progress.

[Mr-Geekman]

May be it wasn't mentioned. Definition of default params_to_tune are currently present in automl-2.0 branch.

[GooseIt]

Is it true that params_to_tune in automl-2.0 currently provide no functional of getting said params for a basic model or a transform beside default {}?
I've seen the Pipeline method params_to_tune, it aggregates results for basic models and transforms well, the question is what these results are.
If what I asked is true, should I implement class Tune with assumption that params_to_tune method for a base model or a transform returns all tunable params, as it is supposed to do?

[GooseIt]

Why aren't changes in automl-2.0 (namely params_to_tune definitions) merged into master?

[Mr-Geekman]

1. Yes, params_to_tune currently are just giving some interface to work with. Probably, some objects will have this empty result {} for a long time and in Tune we should be able to work with it.
2. You can examine tests for params_to_tune in tests/test_pipeline/test_mixins.py. It should give some idea how the result looks like.
3. We thought that this track (automl) should be made in a separate branch until it is complete. We forgot about it during merging your pull requests. Probably, we should continue doing this track in a automl-2.0 branch. You can continue doing this task as you do. We will discuss this details during PR.
4. In Tune we have external params_to_tune parameter. If it is empty, pipeline.params_to_tune() is used. If it is empty too let's check what happens to optimization if we do nothing. After that we will decide are we going to raise exception manually or not.

[GooseIt]

I've encountered the following problem.
params_to_tune returns params_dict such that params_dict[param_name] = param_distr, where param_distr is an optuna.distribution object.
Current optuna API provides the following options to set up optimization process:

1. Pick up sampler from optuna.samplers, then for each parameters call trial.suggest_* specifying params ranges from params_to_tune in objective.
   The downside of this approach is that code will have to use multiple if clauses to set trial params. As example, for a given param_name with param_distribution we will have to go through all the possible distributions of said param:

   if type(param_distribution) == optuna.distributions.UniformDistribution:
       params[param_name] = trial.suggest_uniform(...)
   elif type(param_distribution) == optuna.distributions.CategoricalDistribution:
       params[param_name] = trial.suggest_categorical(...)
   ...
   

2. Create custom optuna sampler that encapsulates neccessary logic of sampling from optuna.distribution in it (etna.auto.optuna.ConfigSampler is an example of encapsulating logic of sampling that consists of picking pipelines from a predetermined pool)
   The downside of this approach is inability to use many samplers from Optuna (possibility of having a hyperparameter responsible for which sampler from Optuna to use is very attractive).

Which of the approaches do you think is the best one? Or is there a better alternative implementation you can propose?

[Mr-Geekman]

Ok, we will investigate this problem and suggest some solution.

[Mr-Geekman]

Hello. I think that first solution is better. Some details for the better implementation:

• We can probably define some dict that specifies all cases for this if/else logic. For example, key is distribution class, value is suggest_xxx function with tuple of parameters to extract from distribution object to pass inside suggest_xxx function.
• It is better to use isinstance instead of type.

[GooseIt]

There is another issue.
As stated in poetry.lock file, an outdated verison of optuna is used:

name = "optuna"
version = "2.10.1"

Version 3.0.0 is out for more than half a year, and it introduced changes that impact optuna.distributions logic.
In version 2.10.1 there are many distribution classes: UniformDistribution, LogUniformDistribution, DiscreteUniformDistribution, IntUniformDistribution, IntLogUniformDistribution, CategoricalDistribution.
Version 3.0.0 implemented unified interface for these classes - new optuna.distributions IntDistribution, FloatDistribution, CategoricalDistribution and made the 2.10.1 classes deprecated due to removal in 6.0.0 (though documentation states removal schedule may change).

The question is:
Should I implement Tune in 2.10.1 code (creating a dict of all deprecated optuna.distributions like LogIntUniformDistribution, etc) or should we prioritize moving Etna to a newer optuna version?

I think sticking with outdated version of optuna will cause numerous problems integrating the new version into Etna later (and we want Etna to support newly-added features of optuna, don't we?)

In my perspective, moving Etna to optuna 3.x.y will make implementation of first solution easier, it can be done in two steps:

• cast a (maybe deprecated) distribution to 3.0.0 distribution class (IntDistribution, FloatDistribution, CategoricalDistribution)
• dict as proposed by you in the previous comment with only 3.0.0 distributions as keys

[Mr-Geekman]

Let's try to update optuna up to version 3.

• Update pyproject.toml: find optuna and set: version="^3.0"
• Update poetry.lock: poetry lock.
• Check that tests tests/test_auto are working.

If there are some difficult to fix problems with this transfer, let's postpone it and don't do this in this task.

[GooseIt]

poetry lock gives the following error message:

Because no versions of pytorch-forecasting match >0.9.0,<0.9.1 || >0.9.1,<0.9.2 || >0.9.2,<0.10.0
 and pytorch-forecasting (0.9.0) depends on optuna (>=2.3.0,<3.0.0), pytorch-forecasting (>=0.9.0,<0.9.1 || >0.9.1,<0.9.2 || >0.9.2,<0.10.0) requires optuna (>=2.3.0,<3.0.0).
And because pytorch-forecasting (0.9.1) depends on optuna (>=2.3.0,<3.0.0)
 and pytorch-forecasting (0.9.2) depends on optuna (>=2.3.0,<3.0.0), pytorch-forecasting (>=0.9.0,<0.10.0) requires optuna (>=2.3.0,<3.0.0).
So, because etna depends on both pytorch-forecasting (^0.9.0) and optuna (^3.0.0), version solving failed.

It seems to me to be caused by fact that pytorch-forecasting library uses optuna 2.x.y (namely, 2.10.1 in its latest version).

Which course of further action do you advise?

[Mr-Geekman]

There isn't easy fix for this. Let's for now stick with optuna 2. We will think what to do with upgrading later.

[GooseIt]

I've encountered a problem with passing params_to_tune into objective function. It cannot be done through the function's arguments, as it contradicts objective definition in AutoAbstract. It also cannot be done via class field, as objective is an staticmethod. It cannot be done via sampler object (as it was done with pool in Auto via ConfigSampler) either, as we've decided to support samplers inbuilt in optuna.

Do you have an idea on how to pass params_to_tune into objective without modifying current inheritance structure? And if not, what should be done about the structure?

I have no idea how to solve the issue, as optimizable pipeline and its params_to_tune are very closely tied to Tune object, while objective is staticmethod and is object-independent.

I'll take a break until early Sunday, so you can take your time to answer.

[Mr-Geekman]

Let's change the signature of def objective:

• Remove it from AbstractAuto
• Add new signature into Tune
  • Add parameter pipeline after parameter ts

I think it should solve our problem.

[Mr-Geekman]

Closed by #1200.