Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[timeseries] Implement OOF prediction caching #3062

Merged
merged 8 commits into from
Mar 24, 2023

Conversation

shchur
Copy link
Collaborator

@shchur shchur commented Mar 17, 2023

This is part 1 of the improved multi-window backtesting for time series (part 2: #3051).

Description of changes:

  • Cache the predictions on the validation set for all time series models (as a model attribute + saved to disk). This ensures that we only call predict on the val_data once for all models. This saves training time and deals with the fact that predictions are non-deterministic for some models.
  • Ensure that the reported prediction time for the test/val sets corresponds to the actual prediction time (before it included the time spent slicing the data / computing the metric).
  • Simplify scoring logic in AbstractTimeSeriesModel and AbstractTimeSeriesTrainer
  • Fix the bug where AutoGluonTabularModel would create an empty directory at initialization.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@github-actions
Copy link

Job PR-3062-d9c1ac8 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-3062/d9c1ac8/index.html

Copy link
Contributor

@canerturkmen canerturkmen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great! added some comments.

Copy link
Contributor

@canerturkmen canerturkmen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@github-actions
Copy link

Job PR-3062-47b8cfd is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-3062/47b8cfd/index.html

@shchur shchur merged commit 4b479bc into autogluon:master Mar 24, 2023
@shchur shchur deleted the cache-predictions branch March 24, 2023 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants