Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alerts continue to dispatch actions even after they've been disabled or deleted #3159

Closed
rbeckman-nextgen opened this issue May 11, 2020 · 2 comments
Milestone

Comments

@rbeckman-nextgen
Copy link
Collaborator

@rbeckman-nextgen rbeckman-nextgen commented May 11, 2020

When an alert action is triggered, the worker just creates a new action task and submits it to the executor. Once it gets submitted, the task will run, regardless of whether the alert gets disabled or deleted in the meantime (unless of course the executor gets shutdown, which never happens in this case). Instead, each action task should have a way to check whether the alert still exists (and is enabled) before actually doing anything.

Or alternatively, the user should have some way to "invalidate" the action task queue, like a "Reset Alert" task (and perhaps that just happens automatically when an alert is disabled/deleted). Maybe when an action task is created, it stores a creation timestamp. Then when the user resets the alert, the alert worker stores an invalidation timestamp for that alert. Then in the action task nothing is done if the invalidation timestamp is greater than the creation timestamp. We wouldn't want to shutdown the executor or anything, because then it would cancel tasks for all alerts, not just a specific one.

On the one hand, the argument could be made that "the alert was enabled at the time of the action trigger, so that action should execute". However take this example:

Say you have a channel that for whatever reason started throwing a TON of errors overnight, such that a TON of alert actions got submitted, faster than the actions themselves could complete (so now there are a bunch of action tasks waiting in line at the executor). Now the user wakes up and sees that he's receiving a bunch of e-mails. He fixes the problem in the channel and redeploys, and everything is working fine, yet he's still receiving those alert e-mails! He clears the trigger count on the Alerts view, and sees that it's zero (and stays at zero). He disables the alert: no good, the e-mails keep coming. He deletes the alert altogether: no good, the e-mails keep coming ("how is that possible, the alert no longer even exists!").

In the real world, I think people will care less about the "at-the-time-of correctness", and more about "when I disable an alert, I shouldn't still be getting alert e-mails".

Imported Issue. Original Details:
Jira Issue Key: MIRTH-3248
Reporter: narupley
Created: 2014-04-24T08:43:00.000-0700

@rbeckman-nextgen rbeckman-nextgen added this to the 3.0.3 milestone May 11, 2020
@rbeckman-nextgen
Copy link
Collaborator Author

@rbeckman-nextgen rbeckman-nextgen commented May 11, 2020

When an alert is disabled, all actions that were queued up for that alert will no longer send anything to their recipients. Even if the alert is re-enabled, all actions that were queued before the alert was enabled will not do anything. This prevents a large backlog of queued actions from continuing to dispatch even after the alert is disabled.

Imported Comment. Original Details:
Author: wayneh
Created: 2014-05-09T15:21:28.000-0700

@rbeckman-nextgen
Copy link
Collaborator Author

@rbeckman-nextgen rbeckman-nextgen commented May 11, 2020

Tested by spamming a slow alert listener channel with thousands of alerts. Verified that disabling the alert does nothing in 3.0.2, but in 3.0.3 it now stops them.

Imported Comment. Original Details:
Author: brentm
Created: 2014-05-14T11:52:57.000-0700

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.