Sb/time series checker #2362

bes-dev · 2021-04-29T10:37:01Z

Add annotation converter for electricity dataset (data source https://archive.ics.uci.edu/ml/machine-learning-databases/00321/LD2011_2014.txt.zip)
Add Annotation representation for Time Series
Add Adapter for temporal fusion transformer
Add normalized_quantile_loss metric
Add postprocessor for time series electricity

openvino-pushbot · 2021-04-29T10:37:04Z

Can one of the admins verify this patch?

bes-dev · 2021-04-29T10:37:14Z

@eaidova please look at PR

eaidova · 2021-04-29T10:39:30Z

jenkins ok to test

tools/accuracy_checker/accuracy_checker/adapters/time_series.py

...curacy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py

tools/accuracy_checker/accuracy_checker/metrics/time_series.py

vladimir-dudnik · 2021-04-29T13:28:52Z

@bes-dev please fix pre-commit checks failed:

14:07:05  ./tools/accuracy_checker/accuracy_checker/metrics/time_series.py:19:1: F401 '.metric.PerImageEvaluationMetric' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/metrics/time_series.py:20:1: F401 '..config.NumberField' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/metrics/time_series.py:20:1: F401 '..config.StringField' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/metrics/time_series.py:20:1: F401 '..config.BoolField' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/metrics/time_series.py:20:1: F401 '..config.ConfigError' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:17:1: F401 'abc' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:25:1: F401 '..config.BoolField' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:40:5: E741 ambiguous variable name 'l'
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:225:11: E114 indentation is not a multiple of      (comment)
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:226:11: E111 indentation is not a multiple of     
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:348:49: E231 missing whitespace after ':'
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:348:75: E231 missing whitespace after ':'
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:350:55: E231 missing whitespace after ':'
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:352:55: E231 missing whitespace after ':'
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:354:57: E231 missing whitespace after ':'
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/__init__.py:106:1: F401 '.electricity_time_series_forecasting.ElectricityTimeSeriesForecastingConverter' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/__init__.py:107:1: F401 '.kaldi_speech_recognition_pipeline.KaldiSpeechRecognitionDataConverter' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/adapters/time_series.py:17:1: F401 'numpy as np' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/adapters/time_series.py:20:1: F401 '..config.PathField' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/adapters/time_series.py:20:1: F401 '..config.NumberField' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/adapters/time_series.py:20:1: F401 '..config.BoolField' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/representation/time_series_representation.py:17:1: F401 'numpy as np' imported but unused
14:07:05  running documentation checks...
14:07:06  tools/accuracy_checker/requirements.in:42: last line doesn't end with a newline character

bes-dev · 2021-04-29T13:45:39Z

@bes-dev please fix pre-commit checks failed:

14:07:05  ./tools/accuracy_checker/accuracy_checker/metrics/time_series.py:19:1: F401 '.metric.PerImageEvaluationMetric' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/metrics/time_series.py:20:1: F401 '..config.NumberField' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/metrics/time_series.py:20:1: F401 '..config.StringField' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/metrics/time_series.py:20:1: F401 '..config.BoolField' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/metrics/time_series.py:20:1: F401 '..config.ConfigError' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:17:1: F401 'abc' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:25:1: F401 '..config.BoolField' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:40:5: E741 ambiguous variable name 'l'
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:225:11: E114 indentation is not a multiple of      (comment)
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:226:11: E111 indentation is not a multiple of     
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:348:49: E231 missing whitespace after ':'
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:348:75: E231 missing whitespace after ':'
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:350:55: E231 missing whitespace after ':'
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:352:55: E231 missing whitespace after ':'
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:354:57: E231 missing whitespace after ':'
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/__init__.py:106:1: F401 '.electricity_time_series_forecasting.ElectricityTimeSeriesForecastingConverter' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/annotation_converters/__init__.py:107:1: F401 '.kaldi_speech_recognition_pipeline.KaldiSpeechRecognitionDataConverter' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/adapters/time_series.py:17:1: F401 'numpy as np' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/adapters/time_series.py:20:1: F401 '..config.PathField' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/adapters/time_series.py:20:1: F401 '..config.NumberField' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/adapters/time_series.py:20:1: F401 '..config.BoolField' imported but unused
14:07:05  ./tools/accuracy_checker/accuracy_checker/representation/time_series_representation.py:17:1: F401 'numpy as np' imported but unused
14:07:05  running documentation checks...
14:07:06  tools/accuracy_checker/requirements.in:42: last line doesn't end with a newline character

@vladimir-dudnik done

vladimir-dudnik · 2021-04-29T14:12:09Z

Jenkins please retry a build

vladimir-dudnik · 2021-04-29T14:47:32Z

@bes-dev seems not all yet
17:16:38 ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:224:11: E114 indentation is not a multiple of (comment) 17:16:38 ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:225:11: E111 indentation is not a multiple of 17:16:38 ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:226:15: E111 indentation is not a multiple of 17:16:38 ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:227:15: E111 indentation is not a multiple of 17:16:38 ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:229:15: E111 indentation is not a multiple of 17:16:38 running documentation checks... 17:16:39 tools/accuracy_checker/requirements.in:42: last line doesn't end with a newline character

bes-dev · 2021-04-29T15:08:56Z

@bes-dev seems not all yet
17:16:38 ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:224:11: E114 indentation is not a multiple of (comment) 17:16:38 ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:225:11: E111 indentation is not a multiple of 17:16:38 ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:226:15: E111 indentation is not a multiple of 17:16:38 ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:227:15: E111 indentation is not a multiple of 17:16:38 ./tools/accuracy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:229:15: E111 indentation is not a multiple of 17:16:38 running documentation checks... 17:16:39 tools/accuracy_checker/requirements.in:42: last line doesn't end with a newline character

done

vladimir-dudnik · 2021-04-29T17:47:29Z

Jenkins please retry a build

vladimir-dudnik · 2021-05-04T12:31:30Z

15:04:35 tools/accuracy_checker/requirements.in:43: new blank line at EOF

Co-authored-by: Ekaterina Aidova <ekaterina.aidova@intel.com>

bes-dev · 2021-05-11T21:18:17Z

@eaidova done, but I didn't check it with so old versions of pandas :/

eaidova · 2021-05-14T12:50:38Z

@bes-dev looks like pandas is trouble maker for passing precommit checks....
Actually, we do not need this packge for running AC unit-tests. Could you please wrap its import to unsupported package for now?
Example this structure usage can be found here

vladimir-dudnik · 2021-05-18T08:11:55Z

@bes-dev please resolve conflicts

bes-dev · 2021-05-18T11:00:43Z

@eaidova pandas import has been wrapped, is it looks good for you now?)

...curacy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py

…electricity_time_series_forecasting.py Co-authored-by: Ekaterina Aidova <ekaterina.aidova@intel.com>

...curacy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py

…electricity_time_series_forecasting.py Co-authored-by: Ekaterina Aidova <ekaterina.aidova@intel.com>

eaidova · 2021-05-18T14:33:44Z

@bes-dev could you please fix linting inssues:

17:16:14  ************* Module accuracy_checker.annotation_converters.electricity_time_series_forecasting
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:172: [W9901(backslash-line-continuation), ] use of \ for line continuation
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:175: [W9901(backslash-line-continuation), ] use of \ for line continuation
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:347: [W9901(backslash-line-continuation), ] use of \ for line continuation
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:348: [W9901(backslash-line-continuation), ] use of \ for line continuation
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:350: [W9901(backslash-line-continuation), ] use of \ for line continuation
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:352: [W9901(backslash-line-continuation), ] use of \ for line continuation
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:354: [W9901(backslash-line-continuation), ] use of \ for line continuation
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:62: [C0330(bad-continuation), ] Wrong hanging indentation (add 2 spaces).
17:16:14        tup[0]
17:16:14        ^ |
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:63: [C0330(bad-continuation), ] Wrong hanging indentation (add 2 spaces).
17:16:14        for tup in column_definition
17:16:14        ^ |
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:64: [C0330(bad-continuation), ] Wrong hanging indentation (add 2 spaces).
17:16:14        if tup[1] == data_type and tup[2] not in excluded_input_types
17:16:14        ^ |
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:97: [C0330(bad-continuation), ] Wrong hanging indentation (add 2 spaces).
17:16:14        ('id', DataTypes.REAL_VALUED, InputTypes.ID),
17:16:14        ^ |
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:98: [C0330(bad-continuation), ] Wrong hanging indentation (add 2 spaces).
17:16:14        ('hours_from_start', DataTypes.REAL_VALUED, InputTypes.TIME),
17:16:14        ^ |
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:99: [C0330(bad-continuation), ] Wrong hanging indentation (add 2 spaces).
17:16:14        ('power_usage', DataTypes.REAL_VALUED, InputTypes.TARGET),
17:16:14        ^ |
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:100: [C0330(bad-continuation), ] Wrong hanging indentation (add 2 spaces).
17:16:14        ('hour', DataTypes.REAL_VALUED, InputTypes.KNOWN_INPUT),
17:16:14        ^ |
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:101: [C0330(bad-continuation), ] Wrong hanging indentation (add 2 spaces).
17:16:14        ('day_of_week', DataTypes.REAL_VALUED, InputTypes.KNOWN_INPUT),
17:16:14        ^ |
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:102: [C0330(bad-continuation), ] Wrong hanging indentation (add 2 spaces).
17:16:14        ('hours_from_start', DataTypes.REAL_VALUED, InputTypes.KNOWN_INPUT),
17:16:14        ^ |
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:103: [C0330(bad-continuation), ] Wrong hanging indentation (add 2 spaces).
17:16:14        ('categorical_id', DataTypes.CATEGORICAL, InputTypes.STATIC_INPUT),
17:16:14        ^ |
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:154: [C0330(bad-continuation), ] Wrong continued indentation (remove 6 spaces).
17:16:14                                                         column_definitions)
17:16:14                                                   |     ^
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:156: [C0330(bad-continuation), ] Wrong continued indentation (remove 6 spaces).
17:16:14                                                             column_definitions)
17:16:14                                                       |     ^
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:191: [C0330(bad-continuation), ] Wrong hanging indentation (add 2 spaces).
17:16:14                srs.values)
17:16:14                ^ |
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:356: [C0326(bad-whitespace), ] No space allowed around keyword argument assignment
17:16:14                          .rename(columns = {target_col[0]: 'group_count'})
17:16:14                                          ^
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:362: [C0326(bad-whitespace), ] No space allowed around keyword argument assignment
17:16:14                            df_total_count], axis = 1).reset_index(drop = True)
17:16:14                                                  ^
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:362: [C0326(bad-whitespace), ] No space allowed around keyword argument assignment
17:16:14                            df_total_count], axis = 1).reset_index(drop = True)
17:16:14                                                                        ^
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:134: [W9904(bad-function-call), ElectricityFormatter.split_data] using prohibited function
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:150: [W9904(bad-function-call), ElectricityFormatter.set_scalers] using prohibited function
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:242: [R0201(no-self-use), ElectricityFormatter.get_fixed_params] Method could be a function
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:332: [W0212(protected-access), ElectricityTimeSeriesForecastingConverter.build_data_index] Access to a protected member _column_definition of a client class
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:345: [R0201(no-self-use), ElectricityTimeSeriesForecastingConverter.get_index_filtering] Method could be a function
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:381: [W0212(protected-access), ElectricityTimeSeriesForecastingConverter.get_sample] Access to a protected member _target_scaler of a client class
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:396: [W9904(bad-function-call), ElectricityTimeSeriesForecastingConverter.aggregating_to_hourly_data] using prohibited function
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:385: [R0201(no-self-use), ElectricityTimeSeriesForecastingConverter.aggregating_to_hourly_data] Method could be a function
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:24: [C0411(wrong-import-order), ] third party import "import numpy as np" should be placed before "from ..utils import UnsupportedPackage"
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:25: [C0411(wrong-import-order), ] third party import "import sklearn.preprocessing" should be placed before "from ..utils import UnsupportedPackage"
17:16:14  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:31: [C0411(wrong-import-order), ] third party import "from tqdm import tqdm" should be placed before "from ..utils import UnsupportedPackage"
17:16:14  ************* Module accuracy_checker.metrics.time_series
17:16:14  accuracy_checker/metrics/time_series.py:27: [W9901(backslash-line-continuation), ] use of \ for line continuation
17:16:14  ************* Module accuracy_checker.postprocessor.time_series_denormalize
17:16:14  accuracy_checker/postprocessor/time_series_denormalize.py:27: [C0200(consider-using-enumerate), TimeSeriesDenormalize.process_image] Consider using enumerate instead of iterating with range and len

bes-dev · 2021-05-19T08:59:04Z

@eaidova done. please look at PR

eaidova · 2021-05-19T09:31:33Z

...curacy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py

+
+    df_list = []
+    for label in output:
+        print('Processing {}'.format(label))


please use print_info from logging instead

...curacy_checker/accuracy_checker/annotation_converters/electricity_time_series_forecasting.py

eaidova · 2021-05-19T09:33:53Z

@bes-dev linter still is unhappy :(

11:46:32  ************* Module accuracy_checker.annotation_converters.electricity_time_series_forecasting
11:46:32  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:116: [C0301(line-too-long), ] Line too long (148/120)
11:46:32  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:117: [C0301(line-too-long), ] Line too long (133/120)
11:46:32  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:118: [C0301(line-too-long), ] Line too long (140/120)
11:46:32  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:119: [C0301(line-too-long), ] Line too long (128/120)
11:46:32  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:78: [W9904(bad-function-call), aggregating_to_hourly_data] using prohibited function
11:46:32  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:188: [R0201(no-self-use), ElectricityFormatter.get_time_steps] Method could be a function
11:46:32  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:191: [R0201(no-self-use), ElectricityFormatter.get_num_encoder_steps] Method could be a function
11:46:32  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:205: [W9904(bad-function-call), ElectricityFormatter.split_data] using prohibited function
11:46:32  accuracy_checker/annotation_converters/electricity_time_series_forecasting.py:221: [W9904(bad-function-call), ElectricityFormatter.set_scalers] using prohibited function
11:46:32  ************* Module accuracy_checker.metrics.time_series
11:46:32  accuracy_checker/metrics/time_series.py:28: [C0301(line-too-long), ] Line too long (127/120)

eaidova · 2021-05-19T10:22:32Z

@bes-dev the last issue:

13:19:26  accuracy_checker/metrics/time_series.py:27: [W9901(backslash-line-continuation), ] use of \ for line continuation

bes-dev · 2021-05-19T10:32:53Z

@eaidova I think all done now :)

bes-dev added 6 commits April 26, 2021 00:40

add time series accurcy checker

f22ebf1

update time_series metric output format

d29d324

update accuracy_checker for time_series

496ad00

update anontation converter for electricity

3cef72a

add URL for electricity

d6d9be6

remove debug code

30349b7

Merge branch 'develop' into sb/time_series_checker

673b4d9