Alert marker fills up with deleted alerts' states

[siavashs]

Alert marker is shared between multiple components, and this results in the marker to fill up with states of alerts which are already deleted.

This probably happens in multiple scenarios but one confirmed case is:

1. Alert is marked active
2. Alert is dispatched
3. Alert is resolved
   • Alert is deleted from memory provider(s)
   • Alert is deleted from marker
4. Notifier fails to send the notification dispatched form step 2
   • Notifier retries
   • Alert is added with active status to the marker again and will never be deleted

This is visible on alertmanager_marked_alerts metric, the marker holds hundreds of thousands of states until Alertmanager is restarted.