Pending sub-job messages in ActiveMQ queues are not processed until a new job is created

[jrobble]

This bug has been observed in environments where a very high number of jobs are processed. It does not happen consistently.

This bug may be related to this issue, this comment or this issue. Setting the cache level to CACHE_NONE and/or disabling producerFlowControl may help.

This bug seems related to https://dzone.com/articles/starving-jms-consumers-in-activemq.

[jrobble]

From the user:

• Set producerFlowControl=false (/opt/apache-activemq-5.13.0/conf/activemq.xml)
• Imposed memory limits for request queues (/opt/apache-activemq-5.13.0/conf/activemq.xml)
• Set prefetch for request queues to 0 (/opt/mpf/data/nodeManagerConfig.xml)
• Increased maximum connections to 5000 (/opt/apache-activemq-5.13.0/conf/activemq.xml)
• Increased the max concurrent consumers for MPF.COMPLETED_DETECTIONS to 60 (/opt/apache-tomcat/webapps/workflow-manager/WEB-INF/classes/applicationContext-jms.xml)
• Increased ActiveMQ’s memory usage (/opt/apache-activemq-5.13.0/bin/linux-x86-64/wrapper.conf and /opt/apache-activemq-5.13.0/conf/activemq.xml)
• Turned on persistence (/opt/apache-activemq-5.13.0/conf/activemq.xml)

These are captured in:

• Add support for ActiveMQ profiles #887
• Remove queue prefetching from component service queues #749
• Increase ActiveMQ maxConcurrentConsumers for MPF.COMPLETED_DETECTIONS #888