Temporary queue filling up

[salapat11]

MQ - v9.1
mq-golang - v3.0.1
Go - 1.11

Temporary queue is created when Prometheus service is started and filling up in no time. Why is a temporary queue created?

[ibmmqmet]

The statistics are published to a queue created as part of the subscription, from where the monitor programs read them. I'd expect there to be messages built up on the queue until the next poll interval from Prometheus, and then the queue is emptied ready for the next batch of publications from the queue manager.

How many queues are you monitoring? There will be several publications associated wih each queue. Thje queue manager will normally publish the resource statistics every 10 seconds.

The queue would only fill if the monitor program cannot keep up with the publication rate.

[salapat11]

Thanks for the quick response. Is "SYSTEM.DEFAULT.MODEL.QUEUE" not used for reading the statistics?
There are 96 queues being monitored on this queue manager and the max queue depth is 5000 for the temporary queue. So ~3000 messages are produced for each batch.

[ibmmqmet]

SYSTEM.DEFAULT.MODEL.QUEUE is a template from which a dynamic queue is created when the model is opened. That's how Model queues are used.

There is a configuration option for the monitor to choose a different replyQ name which could point to a predefined Local queue or a different Model queue, perhaps with a different MAXDEPTH.

[salapat11]

Thanks again. This issue only happens on couple of servers which are in DMZ and have large number of queues. Will try with a different model queue.

[salapat11]

Prometheus server actively scrapes endpoints on the MQ servers. If there is a disaster on the Prometheus server, the temporary queue gets filled up with the queue manager publications. Is there anyway to stop the statistics from publishing? Queue manager PROMETHEUS service can be disabled, but this has to be done on each queue manager and server.

[ibmmqmet]

There's no way to stop stats from being published, other than stopping the monitor program. If the queue does fills up, then the qmgr will continue to attempt to publish to it (and fail). Once the monitor gets called again from a recovered Prometheus, then it will report on the data that had been available up to the point of the queue filling, and then the next set of publications from the qmgr will get processed normally. So you end up missing the numbers generated between the queue filling and the next scrape, but it will recover after that. You could of course automate things around notification of the queue filling up (MQ can generate an event message in those circumstances) but then you need a different kind of monitor process - it's not the kind of thing that Prometheus is aimed at - to recognise the event and take some action.

[salapat11]

Thanks for the explanation. That leads to have an Active/Active architecture.

[ibmmqmet]

Setting CAPEXPRY on the model queue definition used for the replyq works with the latest updates to the monitor packages