New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

st2.notifiers.work queue is growing. #3622

Closed
lakshmi-kannan opened this Issue Jul 27, 2017 · 4 comments

Comments

Projects
None yet
2 participants
@lakshmi-kannan
Contributor

lakshmi-kannan commented Jul 27, 2017

Reported by: @emptywee

From slack:

it was st2.notifiers.work queue


[5:18] 
with over 747k messages


[5:18] 
purged it


[5:21] 
I see no consumers for that queue


[5:21] 
can anyone here point me in the right direction with this? :slightly_smiling_face:

@lakshmi-kannan lakshmi-kannan added this to the 2.4.0 milestone Jul 27, 2017

@lakshmi-kannan

This comment has been minimized.

Contributor

lakshmi-kannan commented Jul 27, 2017

I think this is happening after we switched notifier to scan execution queue as opposed to liveactions queue. There is no consumer for liveactions queue.

@lakshmi-kannan lakshmi-kannan self-assigned this Jul 27, 2017

@lakshmi-kannan

This comment has been minimized.

Contributor

lakshmi-kannan commented Jul 27, 2017

Notifiers now consume "ActionExecution" RabbitMQ exchange with queue name "st2.notifiers.execution.work". Notifier used to scan "LiveAction" exchange with
"st2.notifiers.work" queue name. When you upgrade from older versions, make sure
"st2.notifiers.work" queue size is 0. If you upgrade when it's non-empty, you might
miss notifications. Post upgrade, please delete queue "st2.notifiers.work" queue manually
using

rabbitmqadmin purge queue name=st2.notifiers.work

If not, this queue will grow indefinitely and rabbitmq would eat up a lot of disk space.

@lakshmi-kannan

This comment has been minimized.

Contributor

lakshmi-kannan commented Jul 27, 2017

Added info in Upgrade Notes StackStorm/st2docs#564

@armab

This comment has been minimized.

Member

armab commented Jul 28, 2017

"As seen on all production servers."

expecting to see that issue in our fleet as well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment