st2.notifiers.work queue is growing.

[lakshmi-kannan]

Reported by: @emptywee

From slack:

it was st2.notifiers.work queue


[5:18] 
with over 747k messages


[5:18] 
purged it


[5:21] 
I see no consumers for that queue


[5:21] 
can anyone here point me in the right direction with this? :slightly_smiling_face:

[lakshmi-kannan]

I think this is happening after we switched notifier to scan execution queue as opposed to liveactions queue. There is no consumer for liveactions queue.

[lakshmi-kannan]

Notifiers now consume "ActionExecution" RabbitMQ exchange with queue name "st2.notifiers.execution.work". Notifier used to scan "LiveAction" exchange with
"st2.notifiers.work" queue name. When you upgrade from older versions, make sure
"st2.notifiers.work" queue size is 0. If you upgrade when it's non-empty, you might
miss notifications. Post upgrade, please delete queue "st2.notifiers.work" queue manually
using

rabbitmqadmin purge queue name=st2.notifiers.work

If not, this queue will grow indefinitely and rabbitmq would eat up a lot of disk space.

[lakshmi-kannan]

Added info in Upgrade Notes StackStorm/st2docs#564

[armab]

"As seen on all production servers."

expecting to see that issue in our fleet as well