Notifications stuck in queue, no way to purge.

[showingtime-devops]

ISSUE TYPE

• Bug Report

SUMMARY

I have 200 notifications stuck in queue with no way to remove them and I cannot delete the notification template.

ENVIRONMENT

• AWX version: 15.0.1
• AWX install method: Amazon EKS
• Ansible version: 2.9.14
• Operating System: Linux
• Web Browser: Firefox, Chrome

STEPS TO REPRODUCE

N/A

EXPECTED RESULTS

Expect to be able to resend or purge old notifications.

ACTUAL RESULTS

No way to resend or purge old notifications.

ADDITIONAL INFORMATION

awx notifications list
{
     "next": "/api/v2/notifications/?page=2",
     "count": 199,
     "previous": null,
     "results": [
          {
               "related": {
                    "notification_template": "/api/v2/notification_templates/2/"
               },
               "summary_fields": {
                    "notification_template": {
                         "id": 2,
                         "name": "Slack devops-notify",
                         "description": ""
                    }
               },
               "url": "/api/v2/notifications/576/",
               "status": "pending",

Error Message when attempting to delete Notification Template in AWX Web UI

Error!
ERROR: Delete not allowed while there are pending notifications

[ryanpetrello]

Hey @showingtime-devops,

There might be some hints in your logs that could explain why these notifications are stuck in pending - I expect it's likely related to some sort of bug or side effect of a service restart.

If anybody wants to take a stab at improving this, there's a few things we could do to address this in terms of cleanup.

1. Allow you to resend "old" pending notifications (probably a medium amount of work)
2. Make the deletion logic for notification templates not block you if the pending notifications are older than some reasonable time limit (a few hours, maybe?)

Does a patch like this work for you? If so, we could turn it into a PR:

diff --git a/awx/api/views/__init__.py b/awx/api/views/__init__.py
index df138b8a6d..87a12a7d51 100644
--- a/awx/api/views/__init__.py
+++ b/awx/api/views/__init__.py
@@ -4257,7 +4257,9 @@ class NotificationTemplateDetail(RetrieveUpdateDestroyAPIView):
         obj = self.get_object()
         if not request.user.can_access(self.model, 'delete', obj):
             return Response(status=status.HTTP_404_NOT_FOUND)
-        if obj.notifications.filter(status='pending').exists():
+
+        recent = now() - dateutil.relativedelta.relativedelta(hours=8)
+        if obj.notifications.filter(status='pending', created__gt=recent).exists():
             return Response({"error": _("Delete not allowed while there are pending notifications")},
                             status=status.HTTP_405_METHOD_NOT_ALLOWED)
         return super(NotificationTemplateDetail, self).delete(request, *args, **kwargs)

[showingtime-devops]

Hi @ryanpetrello ,

Option 2 works for me. I really don't need a job notification from last year 😄

[danielsouzasp]

I could fix it by deleting the pending notification via psql CLI:
awx=# DELETE FROM main_notification WHERE status='pending';
DELETE 1