[ML] Should we have an option for annotations to be retained after job deletion?

[droberts195]

#33376 (comment) says:

We also decided that it would be useful to have persistent annotations, that don't get deleted if all the jobs they relate to get deleted. This would be useful for the case of repeatedly deleting and recreating a job to try out slightly different configurations.

So there should be an is_persistent boolean field in the annotations too.

However, the is_persistent field was never implemented and in fact, up to 7.9, all annotations persist after job deletion.

In 7.9 and above it will be completely unacceptable to retain all ML annotations forever: once model change annotations are added there could be thousands of them. Therefore #57976 will need to be addressed so that annotations get deleted alongside the jobs they relate to.

However, this reopens the question of whether it would be useful for a small handful of user-added annotations to persist past job deletion, for the use case where a user is iterating on job configuration repeatedly deleting and recreating a job and wants certain points on its timeline to remain annotated throughout this iteration.

As an aside, if we do introduce the idea of persistent annotations, retain would probably be a better field name than is_persistent as we have already used it for model snapshots.

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[sophiec20]

Completely agree we should delete the system created annotations as these relate to the model. The model is being deleted, so we should delete its associated annotations.

Wrt user-created annotations - this is more nuanced. It's valuable information in the case of deleting/recreating the same job. For example, if someone marked up 20 unusual events and then re-ran the job with an additional influencer, then it's useful to retain these annotations - it would be horrible to have to manually recreate these and we have no easy way for someone to save them. But this workflow is still not ideal (as the user has to know to use the same job name) and I suspect this only occurs for a small number of users and jobs.

If we were to think about adding user controls for retaining annotations, then I think we have more to discuss.. there is more to the UX than simply tagging the annotation when it is created.

• should we default to "retain" when the annotation is created or not?
  If we default to retain:true, then this is pretty similar to not deleting the user-annotations.
  If we default to retain:false, then this is pretty similar to deleting user-annotations
  Would user-annotations created before 7.9 be retained or not?
  A user might not know they want this annotation retained when the create it, and they may change their mind when they delete the job.
• should we ask the user if they want to retain their user-annotations when they delete a job?
  At this point, we'd need API changes to allow conditional deletion and potentially UI workflow to allow users to review their annotations so they could decide if they wanted to keep them or not. (these annotations may have been created by someone else)..
• should we allow user-annotations to be carried across when cloning a job
  Maybe
• should we allow user-annotations to be applicable to multiple jobs
  Probably

Retaining user-annotations upon job deletion has some value and changing this is a change to current functionality (albeit believed to be rarely used functionality).

If we were to think about narrowest scope - then perhaps we should only delete system-annotations upon job deletion and retain user-annotations. This keeps existing functionality as is, whilst cleaning up the notifications. We can then consider a roadmap item to design a better experience for user-annotations.

[droberts195]

I suspect this only occurs for a small number of users and jobs

It's just you @sophiec20 🤣

[droberts195]

If we were to think about narrowest scope - then perhaps we should only delete system-annotations upon job deletion and retain user-annotations. This keeps existing functionality as is, whilst cleaning up the notifications. We can then consider a roadmap item to design a better experience for user-annotations.

That narrowest scope is what we did (in #58169 in 7.9).

#74310 has been opened since then to add an option to delete user annotations. Adding that extra option would be a solution to this issue, so closing.