[ML] Assorted time series modelling stability fixes #1675
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tveasey
added a commit
that referenced
this pull request
Jan 28, 2021
#1675 introduced a subtle bug in the updating of newly initialised seasonal components. It switched to using state to track when it refreshes the seasonal model predictions to reflect new data (to correct behaviour when there are long gaps in the data). However, we test for seasonal components on a window which can sometimes extend beyond the current time (with future values set to "null"). As a result we were sometimes setting the time to refresh into the future and the model wasn't reflecting new data immediately after a seasonal component was added.
tveasey
added a commit
to tveasey/ml-cpp-1
that referenced
this pull request
Feb 5, 2021
This addresses various sources of instability in the time series modelling for anomaly detection:
1. Our update dynamics were stable (just). However, we do other things which aren't considered in this analysis, such
as apply different weights to different updates. I added a small margin.
2. There was an error in the times at which we asked for model predictions when component models were restricted
to time windows, i.e. weekdays and weekends.
3. We could use stale predictions when there were long gaps in the data.
4. Changing the seasonal derivative too fast can create transient instability after data changes.
I've also added some more unit testing. Particular, in the area of when we refresh predictions and explicitly test long
term stability.
tveasey
added a commit
to tveasey/ml-cpp-1
that referenced
this pull request
Feb 5, 2021
elastic#1675 introduced a subtle bug in the updating of newly initialised seasonal components. It switched to using state to track when it refreshes the seasonal model predictions to reflect new data (to correct behaviour when there are long gaps in the data). However, we test for seasonal components on a window which can sometimes extend beyond the current time (with future values set to "null"). As a result we were sometimes setting the time to refresh into the future and the model wasn't reflecting new data immediately after a seasonal component was added.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This addresses various sources of instability in the time series modelling for anomaly detection:
I've also added some more unit testing. Particular, in the area of when we refresh predictions and an explicit testing long term stability.