Add tuning history to OptHistory #228

MorrisNein · 2023-10-25T23:29:14Z

Adds an option to save graph to OptHistory after each objective evaluation in the process of tuning (Solves #226).

Allows tuners to work with Individual and Fitness for graphs comparison & history saving.

Changes OptHistory fields:

adds tuning_result
renames final_choices -> evolution_results
renames archive_history -> evolution_best_archive

Allows histories backward compatibility.

Fixes previously unknown bug with multi objective tuning.

aim-pep8-bot · 2023-10-25T23:29:33Z

Hello @MorrisNein! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2023-11-04 12:07:43 UTC

pep8speaks · 2023-10-25T23:29:39Z

Hello @MorrisNein! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file golem/core/optimisers/random/random_search.py:

Line 38:25: E131 continuation line unaligned for hanging indent

Comment last updated at 2023-11-04 12:07:51 UTC

codecov-commenter · 2023-10-26T00:05:48Z

Codecov Report

Merging #228 (02e3ecd) into main (29cd362) will increase coverage by 0.13%.
Report is 3 commits behind head on main.
The diff coverage is 92.66%.

@@            Coverage Diff             @@
##             main     #228      +/-   ##
==========================================
+ Coverage   72.16%   72.29%   +0.13%     
==========================================
  Files         135      136       +1     
  Lines        8061     8184     +123     
==========================================
+ Hits         5817     5917     +100     
- Misses       2244     2267      +23

Files	Coverage Δ
golem/core/optimisers/genetic/gp_optimizer.py	`98.80% <100.00%> (+0.01%)`	⬆️
...core/optimisers/opt_history_objects/opt_history.py	`76.37% <100.00%> (+0.79%)`	⬆️
golem/core/optimisers/populational_optimizer.py	`95.74% <100.00%> (+0.04%)`	⬆️
golem/core/tuning/hyperopt_tuner.py	`97.61% <100.00%> (+0.05%)`	⬆️
golem/core/tuning/iopt_tuner.py	`100.00% <100.00%> (ø)`
golem/core/tuning/optuna_tuner.py	`93.10% <100.00%> (+0.08%)`	⬆️
golem/core/tuning/sequential.py	`95.45% <100.00%> (+0.06%)`	⬆️
golem/core/tuning/simultaneous.py	`90.14% <100.00%> (+2.81%)`	⬆️
...em/serializers/coders/opt_history_serialization.py	`98.76% <100.00%> (+0.03%)`	⬆️
...em/visualisation/opt_history/graphs_interactive.py	`0.00% <ø> (ø)`
... and 6 more

... and 21 files with indirect coverage changes

golem/core/tuning/tuner_interface.py

kasyanovse · 2023-10-26T11:37:37Z

golem/core/tuning/tuner_interface.py

        for tuned_graph in tuned_graphs:
-            obtained_metric = self.get_metric_value(graph=tuned_graph)
+            obtained_metric = self.evaluate_graph(graph=tuned_graph, label='tuning_result')


Немного не догоняю что происходит)) В этом цикле будут создаваться новые поколения в history для каждого посчитанного графа?

В этом цикле будут создаваться новые поколения в history для каждого посчитанного графа?

Да, но так быть не должно. Исправил, разделив оценку графов и сохранение истории.

Я правильно понимаю, что есть тест, в котором проверяется соответствие количества новых поколений количеству шагов тюнера?

test/unit/optimizers/test_composing_history.py

test/unit/tuning/test_tuning.py

MorrisNein · 2023-10-30T00:26:23Z

Добавил в тюнеры работу с классами истории оптимизации, раз уж теперь они могут сохранять в неё результаты.

Сравнение графов между собой, на мой взгляд, стало удобнее, особенно для MultiObjFitness: не нужно перегонять метрики в класс MultiObjFitness и обратно.

golem/core/optimisers/genetic/gp_optimizer.py

test/unit/optimizers/test_composing_history.py

YamLyubov · 2023-11-07T10:14:11Z

golem/core/tuning/iopt_tuner.py


 import numpy as np
 from iOpt.method.listener import ConsoleFullOutputListener
 from iOpt.problem import Problem
 from iOpt.solver import Solver
 from iOpt.solver_parametrs import SolverParameters
-from iOpt.trial import Point, FunctionValue
+from iOpt.trial import FunctionValue, Point


Почему в этом тюнере не добавлена возможность сохранить в историю?

Пока что сохраняются изначальный и финальный графы через initial_check и final_check.

Но тут можно прокинуть в GolemProblem функцию для сохранения промежуточных результатов. Пожалуй, так и сделаю.

YamLyubov · 2023-11-07T10:32:13Z

golem/core/optimisers/opt_history_objects/opt_history.py

@@ -36,8 +43,7 @@ def __init__(self,
                 default_save_dir: Optional[os.PathLike] = None):
        self._objective = objective or ObjectiveInfo()
        self._generations: List[Generation] = []
-        self.archive_history: List[List[Individual]] = []
-        self._tuning_result: Optional[Graph] = None
+        self.evolution_best_archive: List[List[Individual]] = []


Я не совсем понимаю зачем нужно такое переименование?

Я пока не реализовал ведение archive_history для тюнинга, потому что для оптимизации его ведёт фронт Парето в составе оптимизатора. Поэтому название поля явно отражает, что оно относится к истории эволюции, но не тюнинга.

Честно, не знаю, как тут лучше поступить.

Ну точно не нужна отсылка к эволюции, для большей абстрактности.
Мне кажется идея что archive_history это основная история, а для файн-тюнинга - отдельная, будет доступной пользователю.

nicl-nno · 2023-11-22T16:16:26Z

golem/core/optimisers/opt_history_objects/opt_history.py

+class OptHistoryLabels(str, Enum):
+    initial_assumptions = 'initial_assumptions'
+    extended_initial_assumptions = 'extended_initial_assumptions'
+    evolution_results = 'evolution_results'


Аналогично, можно про эволюцию явно не писать.

nicl-nno · 2023-11-22T16:17:01Z

golem/core/optimisers/opt_history_objects/opt_history.py

@@ -61,8 +67,8 @@ def add_to_history(self, individuals: Sequence[Individual], generation_label: Op
        generation = Generation(individuals, self.generations_count, generation_label, generation_metadata)
        self.generations.append(generation)

-    def add_to_archive_history(self, individuals: Sequence[Individual]):
-        self.archive_history.append(list(individuals))
+    def add_to_evolution_best_archive(self, individuals: Sequence[Individual]):


Может сделать единый метод, просто передавать в параметрах нужную метку?

nicl-nno · 2023-11-22T16:17:50Z

golem/core/tuning/tuner_interface.py

-        self.init_metric = None
-        self.obtained_metric = None
+        self.init_individual = None
+        self.obtained_individual = None


Лучше final

nicl-nno · 2023-11-22T16:20:50Z

Хорошо бы в одном из примеров показать как использовать новую фунциональность. У @YamLyubov был пример с раскраской графа, например.

MorrisNein added 3 commits October 26, 2023 02:15

add "history" as a parameter of tuner interface

2656a42

rename fields of history to avoid ambiguity; test history after tuning

d2eb0f2

add test for tuner classes to support tuning history

7068a0d

MorrisNein added 2 commits October 26, 2023 02:31

зуз8

be1457e

fix default fitness value support for multi objective tuning

2b21d96

MorrisNein linked an issue Oct 25, 2023 that may be closed by this pull request

Save tuning history as historical individuals #226

Open

fix test function name

d9d6c32

MorrisNein added a commit to aimclub/FEDOT that referenced this pull request Oct 26, 2023

adapt to aimclub/GOLEM#228

5a8f2ef

MorrisNein mentioned this pull request Oct 26, 2023

Add tuning history to OptHistory aimclub/FEDOT#1188

Draft

kasyanovse self-requested a review October 26, 2023 11:06

kasyanovse requested changes Oct 26, 2023

View reviewed changes

MorrisNein added 7 commits October 30, 2023 01:18

use individuals & fitness in tuner interface

03915f1

fix tests & add test for multi-objective tuning

5beeb22

fix historical individuals parenting

ae5b242

add n_jobs=-1 to test of generated history

e98ec48

add test for multi-objective history, fix tuner interface

c19ebbb

interface fixes

56a51f9

fix consistency of history

1c7ef13

MorrisNein force-pushed the add-tuning-history branch from 0af205f to 37de94d Compare October 30, 2023 00:19

create constants for frequently used labels

1c42b3f

MorrisNein force-pushed the add-tuning-history branch from 37de94d to 1c42b3f Compare October 30, 2023 00:32

kasyanovse approved these changes Oct 30, 2023

View reviewed changes

golem/core/optimisers/genetic/gp_optimizer.py Outdated Show resolved Hide resolved

test/unit/optimizers/test_composing_history.py Show resolved Hide resolved

MorrisNein added 2 commits October 30, 2023 13:07

wrap labels into str enumeration

c286828

fix

ff5b6e6

MorrisNein requested a review from YamLyubov November 4, 2023 11:23

fix import

02e3ecd

YamLyubov requested changes Nov 7, 2023

View reviewed changes

MorrisNein requested a review from nicl-nno November 22, 2023 11:47

nicl-nno reviewed Nov 22, 2023

View reviewed changes

MorrisNein marked this pull request as draft December 14, 2023 18:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tuning history to OptHistory #228

Add tuning history to OptHistory #228

MorrisNein commented Oct 25, 2023 •

edited

aim-pep8-bot commented Oct 25, 2023 •

edited

pep8speaks commented Oct 25, 2023 •

edited

codecov-commenter commented Oct 26, 2023 •

edited

kasyanovse Oct 26, 2023

MorrisNein Oct 29, 2023

kasyanovse Oct 30, 2023

MorrisNein commented Oct 30, 2023 •

edited

YamLyubov Nov 7, 2023

MorrisNein Nov 7, 2023

YamLyubov Nov 7, 2023

MorrisNein Nov 7, 2023

nicl-nno Nov 22, 2023

nicl-nno Nov 22, 2023

nicl-nno Nov 22, 2023

nicl-nno Nov 22, 2023

nicl-nno commented Nov 22, 2023

Add tuning history to OptHistory #228

Are you sure you want to change the base?

Add tuning history to OptHistory #228

Conversation

MorrisNein commented Oct 25, 2023 • edited

aim-pep8-bot commented Oct 25, 2023 • edited

Comment last updated at 2023-11-04 12:07:43 UTC

pep8speaks commented Oct 25, 2023 • edited

Comment last updated at 2023-11-04 12:07:51 UTC

codecov-commenter commented Oct 26, 2023 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MorrisNein commented Oct 30, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicl-nno commented Nov 22, 2023

MorrisNein commented Oct 25, 2023 •

edited

aim-pep8-bot commented Oct 25, 2023 •

edited

pep8speaks commented Oct 25, 2023 •

edited

codecov-commenter commented Oct 26, 2023 •

edited

MorrisNein commented Oct 30, 2023 •

edited