[ML] Discrepancy in resetting of soft_limit status

[droberts195]

If an anomaly detection job is approaching its memory limit then we start pruning old models more aggressively and the memory status is set to soft_limit. If the memory usage then reduces such that aggressive pruning is no longer required then we do not immediately reset the memory status to ok. However, if the job is closed and reopened, or if it switches nodes because the node it was running on leaves the cluster then the memory status is reset to ok if pruning is no longer required.

This discrepancy should be addressed. If aggressive pruning is no longer required then the job's memory status should be reset to ok without requiring a process restart.

[droberts195]

Just thinking about this a bit more, if we reset the memory_status to ok without recording the fact that it was in hard_limit or soft_limit then we have lost valuable troubleshooting information.

Therefore I propose that this issue is tackled simultaneously with #1174, and that in addition to adding peak_model_bytes we also add peak_memory_status, max_memory_status or worst_memory_status. This would store the worst memory status over the job's lifetime, while memory_status could reset to ok after pruning or an increase in the model_memory_limit.

[dimitris-athanasiou]

Just thinking about this a bit more, if we reset the memory_status to ok without recording the fact that it was in hard_limit or soft_limit then we have lost valuable troubleshooting information.

Another idea to help with this would be to audit warning messages when memory status changes. We might want to do this on top of any other solution we choose.

[tveasey]

Perhaps this should be an annotation.

[droberts195]

Another idea to help with this would be to audit warning messages when memory status changes

I think we already do this.

Perhaps this should be an annotation

In the future we could have an annotation as well as an audit message - one shows up on the config view and the other in the results view, so the change would then be visible in both.

This can be one of the new annotations to be added by the annotations project.

Given this I think it's OK to reset soft limit to OK without changing the stats object, so I'll go ahead and implement this as originally suggested. Scrub my first comment on this issue.