Timeseries surrogate #85

ShikovEgor · 2023-11-28T13:48:07Z

What's new:

Unification of meta-dataset file construction for surrogate model training for table and timeseries data.
Rework of the training code.
Fix quality metrics and data sampling, taking into account the difference in number of pipelines for different datasets.
Added checkpoints and configes for timeseries.

pep8speaks · 2023-11-28T13:48:24Z

Hello @ShikovEgor! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file gamlet/components/datasets_loaders/__init__.py:

Line 1:1: F401 '.datasets_loader.DatasetsLoader' imported but unused
Line 2:1: F401 '.custom_datasets_loader.CustomDatasetsLoader' imported but unused
Line 3:1: F401 '.openml_datasets_loader.OpenMLDatasetsLoader' imported but unused
Line 4:1: F401 '.timeseries_dataset_loader.TimeSeriesDatasetsLoader' imported but unused

In the file gamlet/components/models_loaders/__init__.py:

Line 1:1: F401 '.models_loader.ModelsLoader' imported but unused
Line 2:1: F401 '.fedot_pipelines_loader.FEDOTPipelinesLoader' imported but unused
Line 3:1: F401 '.knowledge_base_models_loader.KBTSModelsLoader' imported but unused
Line 3:1: F401 '.knowledge_base_models_loader.KnowledgeBaseModelsLoader' imported but unused
Line 3:1: F401 '.knowledge_base_models_loader.CompatKBModelsLoader' imported but unused
Line 4:1: F401 '.fedot_history_loader.FedotHistoryLoader' imported but unused

In the file gamlet/components/models_loaders/knowledge_base_models_loader.py:

Line 45:29: W291 trailing whitespace
Line 46:48: W291 trailing whitespace
Line 47:51: W291 trailing whitespace
Line 48:53: W291 trailing whitespace
Line 49:73: W291 trailing whitespace
Line 50:48: W291 trailing whitespace
Line 71:1: E302 expected 2 blank lines, found 1
Line 105:45: W291 trailing whitespace
Line 106:45: W291 trailing whitespace
Line 107:31: E127 continuation line over-indented for visual indent
Line 107:47: W291 trailing whitespace
Line 108:31: E127 continuation line over-indented for visual indent
Line 109:31: E127 continuation line over-indented for visual indent
Line 129:1: E302 expected 2 blank lines, found 1
Line 130:121: E501 line too long (142 > 120 characters)
Line 154:1: W293 blank line contains whitespace
Line 162:45: W291 trailing whitespace
Line 163:31: E127 continuation line over-indented for visual indent
Line 163:47: W291 trailing whitespace
Line 164:31: E127 continuation line over-indented for visual indent
Line 165:31: E127 continuation line over-indented for visual indent
Line 204:1: W293 blank line contains whitespace

In the file gamlet/data_preparation/file_system/__init__.py:

Line 1:1: F401 '.file_system.PathType' imported but unused
Line 1:1: F401 '.file_system.ensure_dir_exists' imported but unused
Line 1:1: F401 '.file_system.get_checkpoints_dir' imported but unused
Line 1:1: F401 '.file_system.get_configs_dir' imported but unused
Line 1:1: F401 '.file_system.get_data_dir' imported but unused
Line 1:1: F401 '.file_system.get_project_root' imported but unused
Line 3:1: F401 '.cache.CacheOperator' imported but unused
Line 3:1: F401 '.cache.get_cache_dir' imported but unused
Line 3:1: F401 '.cache.get_dataset_cache_path' imported but unused
Line 3:1: F401 '.cache.get_dataset_cache_path_by_id' imported but unused
Line 3:1: F401 '.cache.update_openml_cache_dir' imported but unused

In the file gamlet/data_preparation/surrogate_dataset/__init__.py:

Line 1:1: F401 '.data.GraphDataset' imported but unused
Line 1:1: F401 '.data.PairDataset' imported but unused
Line 1:1: F401 '.data.SingleDataset' imported but unused
Line 2:1: F401 '.dataset_generate.KnowledgeBaseToDataset' imported but unused
Line 2:1: F401 '.dataset_generate.dataset_from_id_without_data_loading' imported but unused
Line 2:1: F401 '.dataset_generate.dataset_from_id_with_data_loading' imported but unused

In the file gamlet/data_preparation/surrogate_dataset/data.py:

Line 2:1: F401 'random.choice' imported but unused
Line 231:89: W291 trailing whitespace

In the file gamlet/data_preparation/surrogate_dataset/dataset_generate.py:

Line 15:1: F401 'gamlet.components.models_loaders.KBTSModelsLoader' imported but unused
Line 15:1: F401 'gamlet.components.models_loaders.CompatKBModelsLoader' imported but unused
Line 99:1: W293 blank line contains whitespace

In the file gamlet/surrogate/training/__init__.py:

Line 1:1: F401 '.train_surrogate_model.train_surrogate_model' imported but unused
Line 1:1: F401 '.train_surrogate_model.setup_loaders' imported but unused
Line 1:1: F401 '.train_surrogate_model.do_training' imported but unused
Line 1:59: W291 trailing whitespace
Line 4:36: E124 closing bracket does not match visual indentation
Line 5:1: F401 '.tune_surrogate_model.tune_surrogate_model' imported but unused

In the file gamlet/surrogate/training/train_surrogate_model.py:

Line 210:1: W293 blank line contains whitespace
Line 211:1: W293 blank line contains whitespace
Line 283:62: W291 trailing whitespace

In the file gamlet/surrogate/training/tune_surrogate_model.py:

Line 4:1: F401 'typing.List' imported but unused
Line 8:1: F401 'torch' imported but unused
Line 12:1: F401 'pytorch_lightning.Trainer' imported but unused
Line 13:1: F401 'pytorch_lightning.callbacks.EarlyStopping' imported but unused
Line 13:1: F401 'pytorch_lightning.callbacks.ModelCheckpoint' imported but unused
Line 14:1: F401 'pytorch_lightning.loggers.TensorBoardLogger' imported but unused
Line 17:1: F401 'gamlet.surrogate.surrogate_model' imported but unused
Line 18:52: E231 missing whitespace after ','
Line 20:1: E302 expected 2 blank lines, found 1

In the file scripts/generate_training_dataset.py:

Line 16:1: W293 blank line contains whitespace
Line 23:25: F821 undefined name 'CompatKBModelsLoader'
Line 23:46: F821 undefined name 'knowledge_base_directory'
Line 23:88: W291 trailing whitespace
Line 28:25: F821 undefined name 'KBTSModelsLoader'
Line 28:42: F821 undefined name 'knowledge_base_directory'
Line 30:64: W291 trailing whitespace
Line 31:1: W293 blank line contains whitespace
Line 43:1: E305 expected 2 blank lines after class or function definition, found 1

Comment last updated at 2024-01-15 13:31:38 UTC

MorrisNein · 2023-12-04T17:53:25Z

@ShikovEgor, нужно сделать ребейз на главную ветку и в процессе пофиксить конфликты.

Также много всего по pep8 - рекомендую воспользоваться хоткеями в IDE, чтобы автоматически пофиксить форматирование и убрать неиспользуемые импорты

meta_automl/data_preparation/models_loaders/knowledge_base_models_loader.py

meta_automl/surrogate/training/tune_surrogate_model.py

scripts/generate_training_dataset.py

MorrisNein · 2023-12-18T07:32:05Z

Также обратите внимание на падающий юнит тест. Где-то нужно перепроверить путь к файлу и при необходимости заменить на путь с использованием функций из file_system.py

MorrisNein · 2023-12-18T07:53:02Z

И всё-таки, покройте используемые классы и функции минимальными юнит-тестами на работоспособность на минимальном подмножестве данных, которые можно сохранить в tests/data. Это убережёт ваш код от вымирания с каждым последующим PR.

Я уверен, что авторам кода это проще сделать, чем кому-либо другому

ShikovEgor · 2023-12-18T08:49:47Z

Не могу пока пофиксить:
python scripts/main.py --train --config configs/train_surrogate_TS.yml
Traceback (most recent call last):
File "/data/home/egor/GAMLET/scripts/main.py", line 5, in
from meta_automl.surrogate import training
File "/data/home/egor/GAMLET/meta_automl/surrogate/training/init.py", line 1, in
from .train_surrogate_model import train_surrogate_model, test_ranking
File "/data/home/egor/GAMLET/meta_automl/surrogate/training/train_surrogate_model.py", line 22, in
from meta_automl.data_preparation.surrogate_dataset import GraphDataset, PairDataset, SingleDataset
File "/data/home/egor/GAMLET/meta_automl/data_preparation/surrogate_dataset/init.py", line 2, in
from .dataset_generate import (
File "/data/home/egor/GAMLET/meta_automl/data_preparation/surrogate_dataset/dataset_generate.py", line 16, in
from meta_automl.data_preparation.feature_preprocessors import FeaturesPreprocessor
File "/data/home/egor/GAMLET/meta_automl/data_preparation/feature_preprocessors/init.py", line 1, in
from .feature_preprocessors import FeaturesPreprocessor
File "/data/home/egor/GAMLET/meta_automl/data_preparation/feature_preprocessors/feature_preprocessors.py", line 9, in
from meta_automl.data_preparation.meta_features_extractors.dataset_meta_features import DatasetMetaFeatures
File "/data/home/egor/GAMLET/meta_automl/data_preparation/meta_features_extractors/init.py", line 2, in
from .openml_dataset_meta_features_extractor import OpenMLDatasetMetaFeaturesExtractor
File "/data/home/egor/GAMLET/meta_automl/data_preparation/meta_features_extractors/openml_dataset_meta_features_extractor.py", line 7, in
from meta_automl.data_preparation.feature_preprocessors import FeaturesPreprocessor
ImportError: cannot import name 'FeaturesPreprocessor' from partially initialized module 'meta_automl.data_preparation.feature_preprocessors' (most likely due to a circular import) (/data/home/egor/GAMLET/meta_automl/data_preparation/feature_preprocessors/init.py)
(new) [egor@bb565d71c869 GAMLET]$

MorrisNein · 2023-12-18T11:58:34Z

Не могу пока пофиксить: ...

Готово. Но лучше совсем уберите FeaturesPreprocessor из OpenMLDatasetMetaFeaturesExtractor.

MorrisNein · 2023-12-19T09:01:21Z

Пофиксил конфликты повторно

codecov · 2023-12-19T09:07:13Z

Codecov Report

Attention: 297 lines in your changes are missing coverage. Please review.

Comparison is base (4c4929b) 41.10% compared to head (415365d) 40.37%.

❗ Current head 415365d differs from pull request most recent head 7ff26a0. Consider uploading reports for the commit 7ff26a0 to get more accurate results

Files	Patch %	Lines
gamlet/surrogate/training/train_surrogate_model.py	0.00%	112 Missing ⚠️
...nts/models_loaders/knowledge_base_models_loader.py	19.56%	74 Missing ⚠️
gamlet/data_preparation/surrogate_dataset/data.py	0.00%	32 Missing ⚠️
..._preparation/surrogate_dataset/dataset_generate.py	0.00%	26 Missing ⚠️
gamlet/surrogate/surrogate_model.py	25.71%	26 Missing ⚠️
gamlet/surrogate/training/tune_surrogate_model.py	0.00%	21 Missing ⚠️
...tractors/openml_dataset_meta_features_extractor.py	75.00%	1 Missing ⚠️
...omponents/models_loaders/fedot_pipelines_loader.py	66.66%	1 Missing ⚠️
...let/data_preparation/surrogate_dataset/__init__.py	0.00%	1 Missing ⚠️
gamlet/surrogate/encoders/dataset_encoder.py	0.00%	1 Missing ⚠️
... and 2 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #85      +/-   ##
==========================================
- Coverage   41.10%   40.37%   -0.73%     
==========================================
  Files          60       60              
  Lines        2472     2576     +104     
==========================================
+ Hits         1016     1040      +24     
- Misses       1456     1536      +80

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…ataset

ShikovEgor requested review from nicl-nno, valer1435 and MorrisNein November 28, 2023 13:48

valer1435 approved these changes Nov 29, 2023

View reviewed changes

MorrisNein force-pushed the timeseries_surrogate branch from 7363133 to 2cffe12 Compare December 15, 2023 14:18

MorrisNein requested changes Dec 18, 2023

View reviewed changes

MorrisNein force-pushed the timeseries_surrogate branch from 162092d to 1e8e0cf Compare December 19, 2023 09:01

Egor Shikov added 15 commits December 20, 2023 00:34

cvgenerator fix for fedot 0.7.2

e873c70

timeseries models loader

d2ce442

training pipeline fixes

9062ff8

train and tune code unification

ee9e043

model parameters rework

b1c7b61

small fixes

d16bd88

model checkpoints added

ad805a0

config filenames change

36ba885

small fixes

3fcb3a6

testing pipeline fix

ebc165d

Training dataloader change. Now number of samples is equal for each d…

6278d44

…ataset

NDCG calculation changed

d3d810d

small changes

4db92e3

config fixes

3873c37

Surrogate TS data files added

8bbfea7

Egor Shikov and others added 11 commits December 20, 2023 00:44

models weights update

7e3a736

Fixes

8145205

small changes

9c85a35

small config changes

d95267b

model checkpoints updata

b8240eb

minor indent fix

8b1aca2

little fix with dim

9f9c337

quick fix

c9f3a8b

fix a circular import

45349c6

test fix

22e6a1a

fix rebase

263e5c7

MorrisNein force-pushed the timeseries_surrogate branch from 7f9e9c4 to 263e5c7 Compare December 19, 2023 21:58

MorrisNein and others added 4 commits December 20, 2023 01:11

simplify imports

7dc0d81

fixes

58a531d

fix

fde0e83

kb loaders fix

66b8181

ShikovEgor requested a review from MorrisNein January 9, 2024 10:49

fixes

dac5d5f

MorrisNein approved these changes Jan 11, 2024

View reviewed changes

Egor Shikov and others added 4 commits January 15, 2024 13:21

kb models_loader fix

a296266

small fix 1

415365d

pep8 fixes

69d1b6d

tune surrogate fixes

7ff26a0

ShikovEgor merged commit 2f41916 into main Jan 20, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Timeseries surrogate #85

Timeseries surrogate #85

ShikovEgor commented Nov 28, 2023

pep8speaks commented Nov 28, 2023 •

edited

Loading

MorrisNein commented Dec 4, 2023

MorrisNein commented Dec 18, 2023 •

edited

Loading

MorrisNein commented Dec 18, 2023 •

edited

Loading

ShikovEgor commented Dec 18, 2023

MorrisNein commented Dec 18, 2023

MorrisNein commented Dec 19, 2023

codecov bot commented Dec 19, 2023 •

edited

Loading

Timeseries surrogate #85

Timeseries surrogate #85

Conversation

ShikovEgor commented Nov 28, 2023

pep8speaks commented Nov 28, 2023 • edited Loading

Comment last updated at 2024-01-15 13:31:38 UTC

MorrisNein commented Dec 4, 2023

MorrisNein commented Dec 18, 2023 • edited Loading

MorrisNein commented Dec 18, 2023 • edited Loading

ShikovEgor commented Dec 18, 2023

MorrisNein commented Dec 18, 2023

MorrisNein commented Dec 19, 2023

codecov bot commented Dec 19, 2023 • edited Loading

Codecov Report

pep8speaks commented Nov 28, 2023 •

edited

Loading

MorrisNein commented Dec 18, 2023 •

edited

Loading

MorrisNein commented Dec 18, 2023 •

edited

Loading

codecov bot commented Dec 19, 2023 •

edited

Loading