AutoMLBenchmark TimeSeries Prototype. #6

limpbot · 2022-09-16T15:56:32Z

Done

FIXME: Why does leaderboard claim a different test score than AutoMLBenchmark for RMSE?

-> ensured correct data loading, with timestamp parsing and correct split into train and test dataset

FIXME: Currently ignoring test_path, just using train data for evaluation

-> done

TODO: How to evaluate more complex metrics like MAPE?

-> added metrics MASE, NCRPS, SMAPE, ND, NRMSE, for that created new TimeSeriesResult class with quantiles and y_past_period_error

How to pass timestamp_column?

How to pass id_column?

How to pass prediction_length?

-> added configuration and passed them to the Dataset object

TODO:

doublecheck metrics implementation

cumbersome testing to ensure not to break other things

Innixma

Awesome contribution! Added some comments, once they are addressed I can test it out and see if it works properly, and if so we can merge.

Innixma · 2022-09-20T00:59:07Z

frameworks/AutoGluonTS/exec.py

@@ -61,16 +61,18 @@ def run(dataset, config):
        )

    with Timer() as predict:
-        predictions = predictor.predict(train_data)
+        test_data_past = test_data.copy().slice_by_timestep(slice(None, -prediction_length))


Do this prior to entering with Timer() so it doesn't add to the recorded inference time.

Innixma · 2022-09-20T01:01:44Z

frameworks/AutoGluonTS/exec.py

@@ -88,28 +90,43 @@ def run(dataset, config):
                  target_is_encoded=False,
                  models_count=num_models_trained,
                  training_duration=training.duration,
-                  predict_duration=predict.duration)
+                  predict_duration=predict.duration,
+                  quantiles=predictions.iloc[:, 1:])


can make more explicit by dropping the 'mean' column by name rather than by assumed position. This makes it easier to understand.

Innixma · 2022-09-20T01:02:16Z

frameworks/AutoGluonTS/exec.py

+    timestamp_column = dataset.timestamp_column
+    id_column = dataset.id_column
+    prediction_length = dataset.prediction_length


Can remove the TODOs that are resolved due to this.

Innixma · 2022-09-20T01:03:20Z

frameworks/AutoGluonTS/__init__.py

+        timestamp_column=dataset.timestamp_column if dataset.timestamp_column is not None else None,
+        id_column=dataset.id_column if dataset.id_column is not None else None,
+        prediction_length=dataset.prediction_length if dataset.prediction_length is not None else None


Should we instead check if dataset.timestamp_column exists rather than if it is not None? ditto for the others.

Currently these if/else don't actually do anything

Innixma · 2022-09-20T01:05:42Z

amlb/results.py

@@ -308,6 +313,16 @@ def save_predictions(dataset: Dataset, output_file: str,

        df = df.assign(predictions=preds)
        df = df.assign(truth=truth)
+        if quantiles is not None:
+            quantiles.reset_index(drop=True, inplace=True)


It is bold to do an inplace operation that alters outer context, let's be safe and avoid inplace operations. (I know that it is probably ok here, but trust me when I say that the nastiest bugs are those involving outer context manipulation caused by inplace operations)

Innixma · 2022-09-20T01:08:48Z

amlb/results.py

+            quantiles.reset_index(drop=True, inplace=True)
+            df = pd.concat([df, quantiles], axis=1)
+        if dataset.type == DatasetType.timeseries:
+            period_length = 1 # this period length could be adapted to the Dataset, but then we need to pass this information as well. As of now this should be fine.


If this is a TODO style comment, then mark it as TODO so it isn't forgotten

Innixma · 2022-09-20T01:24:41Z

amlb/datautils.py

@@ -39,11 +39,15 @@ def read_csv(path, nrows=None, header=True, index=False, as_data_frame=True, dty
    :param dtype: data type for columns.
    :return: a DataFrame
    """
+    if dtype is not None and timestamp_column is not None and timestamp_column in dtype:
+            del dtype[timestamp_column]


Avoid outer context manipulation, instead copy dtype to a new object and then delete timestamp_column.

added copy()

Innixma · 2022-09-20T01:25:47Z

amlb/results.py

@@ -255,7 +259,8 @@ def save_predictions(dataset: Dataset, output_file: str,
                         predictions: Union[A, DF, S] = None, truth: Union[A, DF, S] = None,
                         probabilities: Union[A, DF] = None, probabilities_labels: Union[list, A] = None,
                         target_is_encoded: bool = False,
-                         preview: bool = True):
+                         preview: bool = True,
+                         quantiles: Union[A, DF] = None):


Add quantiles to docstring

Innixma · 2022-09-20T01:26:26Z

amlb/results.py

+            item_ids, inverse_item_ids = np.unique(dataset.test.X[dataset.id_column].squeeze().to_numpy(), return_index=False, return_inverse=True)
+            y_past = [dataset.test.y.squeeze().to_numpy()[inverse_item_ids == i][:-dataset.prediction_length] for i in range(len(item_ids))]
+            y_past_period_error = [np.abs(y_past_item[period_length:] - y_past_item[:-period_length]).mean() for y_past_item in y_past]
+            y_past_period_error_rep = np.repeat(y_past_period_error, dataset.prediction_length)


Add inline comments to explain this, it is a lot to take in when unfamiliar.

Innixma · 2022-09-20T01:29:01Z

.gitignore

@@ -16,6 +16,7 @@ venv/
 .idea/
 *.iml
 *.swp
+launch.json


visualcode creates it for debugging

Innixma · 2022-09-21T02:58:20Z

frameworks/AutoGluonTS/__init__.py

+    if hasattr(dataset, 'timestamp_column') is False:
+        dataset.timestamp_column = None


This edits outer context of dataset. No need to do this.

Innixma

Thanks for this update!!

* Add AutoGluon TimeSeries Prototype * AutoMLBenchmark TimeSeries Prototype. (#6) * fixed loading test & train, changed pred.-l. 5->30 * ignore launch.json of vscode * ensuring timestamp parsing * pass config, save pred, add results * remove unused code * add readability, remove slice from timer * ensure autogluonts has required info * add comments for readability * setting defaults for timeseries task * remove outer context manipulation * corrected spelling error for quantiles * adding mape, correct available metrics * beautify config options * fixed config for public access * Update readme * Autogluon timeseries, addressed comments by sebhrusen (#7) * fixed loading test & train, changed pred.-l. 5->30 * ignore launch.json of vscode * ensuring timestamp parsing * pass config, save pred, add results * remove unused code * add readability, remove slice from timer * ensure autogluonts has required info * add comments for readability * setting defaults for timeseries task * remove outer context manipulation * corrected spelling error for quantiles * adding mape, correct available metrics * beautify config options * fixed config for public access * no outer context manipulation, add dataset subdir * add more datasets * include error raising for too large pred. length. * mergin AutoGluonTS framework folder into AutoGluon * renaming ts.yaml to timeseries.yaml, plus ext. * removing presets, correct latest config for AGTS * move dataset timeseries ext to datasets/file.py * dont bypass test mode * move quantiles and y_past_period_error to opt_cols * remove whitespaces * deleting merge artifacts * delete merge artifacts * renaming prediction_length to forecast_range_in_steps * use public dataset, reduced range to maximum * fix format string works * fix key error bug, remove magic time limit * Addressed minor comments, and fixed version call for tabular and timeseries modularities (#8) * fixed loading test & train, changed pred.-l. 5->30 * ignore launch.json of vscode * ensuring timestamp parsing * pass config, save pred, add results * remove unused code * add readability, remove slice from timer * ensure autogluonts has required info * add comments for readability * setting defaults for timeseries task * remove outer context manipulation * corrected spelling error for quantiles * adding mape, correct available metrics * beautify config options * fixed config for public access * no outer context manipulation, add dataset subdir * add more datasets * include error raising for too large pred. length. * mergin AutoGluonTS framework folder into AutoGluon * renaming ts.yaml to timeseries.yaml, plus ext. * removing presets, correct latest config for AGTS * move dataset timeseries ext to datasets/file.py * dont bypass test mode * move quantiles and y_past_period_error to opt_cols * remove whitespaces * deleting merge artifacts * delete merge artifacts * renaming prediction_length to forecast_range_in_steps * use public dataset, reduced range to maximum * fix format string works * fix key error bug, remove magic time limit * swapped timeseries and tabular to set version * make warning message more explicit * remove outer context manipulation * split timeseries / tabular into functions Co-authored-by: Leo <LeonhardSommer96@gmail.com>

limpbot added 5 commits September 14, 2022 13:49

fixed loading test & train, changed pred.-l. 5->30

fdac87d

ignore launch.json of vscode

acae465

ensuring timestamp parsing

b5723cf

pass config, save pred, add results

55c63e9

remove unused code

0f38986

Innixma reviewed Sep 20, 2022

View reviewed changes

limpbot added 6 commits September 20, 2022 14:02

add readability, remove slice from timer

f932669

ensure autogluonts has required info

16a165b

add comments for readability

758b92d

setting defaults for timeseries task

04872e7

remove outer context manipulation

888a1cb

corrected spelling error for quantiles

e15de3e

Innixma reviewed Sep 21, 2022

View reviewed changes

limpbot added 3 commits September 21, 2022 12:34

adding mape, correct available metrics

866492f

beautify config options

9252835

fixed config for public access

18cc6af

Innixma approved these changes Sep 21, 2022

View reviewed changes

Innixma merged commit d4412e3 into Innixma:autogluon_timeseries Sep 21, 2022

Innixma mentioned this pull request Sep 21, 2022

[PoC] AutoGluon TimeSeries Prototype openml/automlbenchmark#494

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AutoMLBenchmark TimeSeries Prototype. #6

AutoMLBenchmark TimeSeries Prototype. #6

limpbot commented Sep 16, 2022

Innixma left a comment

Innixma Sep 20, 2022

limpbot Sep 21, 2022

Innixma Sep 20, 2022

limpbot Sep 21, 2022

Innixma Sep 20, 2022

limpbot Sep 20, 2022

Innixma Sep 20, 2022

Innixma Sep 20, 2022

Innixma Sep 20, 2022

limpbot Sep 21, 2022

Innixma Sep 20, 2022

Innixma Sep 20, 2022

limpbot Sep 20, 2022

Innixma Sep 20, 2022

limpbot Sep 20, 2022

Innixma Sep 20, 2022

Innixma Sep 20, 2022

limpbot Sep 20, 2022

Innixma Sep 21, 2022

Innixma left a comment

		if hasattr(dataset, 'timestamp_column') is False:
		dataset.timestamp_column = None

AutoMLBenchmark TimeSeries Prototype. #6

AutoMLBenchmark TimeSeries Prototype. #6

Conversation

limpbot commented Sep 16, 2022

Done

FIXME: Why does leaderboard claim a different test score than AutoMLBenchmark for RMSE?

FIXME: Currently ignoring test_path, just using train data for evaluation

TODO: How to evaluate more complex metrics like MAPE?

How to pass timestamp_column?

How to pass id_column?

How to pass prediction_length?

TODO:

doublecheck metrics implementation

cumbersome testing to ensure not to break other things

Innixma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Innixma left a comment

Choose a reason for hiding this comment