Clean up job for old annotations.

[sanchitraizada]

Alert execution engine is one of the useful functionality that would potentially be used extensively. The "events" are persisted as annotations. This can potentially grow larger as the adoption increases. Do we see any potential issues with regards to retention/maintenance etc?

Just opening a placeholder issues to discuss and brainstorm on this.

[torkelo]

yes, there needs to be some retention config & cleanup scheduling

[bergquist]

This should be implemented the same way as #9671
But instead of keeping X versions we should delete annotations older than X days.

But I don't think its as easy to find a good default value. So I suggest
this feature should be disabled by default and admins can choose to enable it.

[torkelo]

But instead of keeping X versions we should delete annotations older than X days.

That sounds like a good plan. Tricky with a good default value

[bergquist]

Thinking a little bit more about this issue. We need to solve two problems.

1. Cleaning up alert annotations. Which can be solved the same way as we solved dashboard versions.
2. Cleaning up other annotations from the annotation table. Which can be solved be deleting all annotations Where type = '' and older than X days.

Both should be triggered from https://github.com/grafana/grafana/blob/master/pkg/services/cleanup/cleanup.go#L48

[FUSAKLA]

This would be extremely helpful since I just found out that my annotation table in MySQL has 160k entries.

most of them are those second mentioned without type

+-------+----------+
| type  | count(*) |
+-------+----------+
|       |   159692 |
| alert |     7682 |
+-------+----------+

Thank's for looking into this!

[kylebrandt]

Notes on what I think we need for complete annotations cleanup:

Annotations Cleanup Feature:

• AND/OR
  • Cleanup by End Time (age)
  • Cleanup by Total Annotation Count (limit)
• Cleanup by Type (Human, Alert, API?)
• Cleanup of Annotation tags?

Schedule Trigger (with job lock):
    Loop Annotation Types:
        If count of older than EndTime OR count greater than N;
            Loop:
                Delete up to batch size
                If oldest newer than EndTime AND count less than N;
                    break;
                sleep;
                if LoopCount > 1 {
                    log if count grows faster than batch size
                }

Job getting counts for cleanup can also record counts as metrics.

Other notes:

• Too many total annotations should not delete Human created annotations without administrative configuration to do so.

[bergquist]

@kylebrandt Good, write up.

I wonder if we can slim down the requirements for this feature to only delete any annotations if they are older the X (off by default). I can see many different configurable ways to delete annotations

• Delete All
• Delete annotations created by humans
• Delete all annotations not created by humans
• Delete alerts annotations
• Delete all none alert annotations
• Delete all annotations matching certain tags

I'm not sure we want to provide configuration for all those options. I know I contradict my previous comments but thinking more about this we need so many different ways to decide what to delete to make it a good feature.

Delete jobs like this is very scriptable in case someone have special requirements.