using ThreadPooTaskExecuter in DefaultMessageListenerContainer results in wrong number of activeConsumers [SPR-7455]

[spring-projects-issues]

Tovi Sofer opened SPR-7455 and commented

I created standalone Spring application to consume messages from JMS queue. For that I used DefaultMessageListenerContainer. In order for the container to use thread-pool for the created invokers (and not create a new thread each time) I defined taskExecuter to be org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.class.

This works OK, except from number of active consumers: I defined concurrentConsumers to be 5 (max is 10), but when I look at jconsole I see activeConsumers=1. (concurrentConsumers is shown as 5).
I didn't change any of ThreadPoolTaskExecuter default definitions - meaning corePoolSize is 1, etc.
When I remove taskExecuter attribute from DefaultMessageListenerContainer definition (and therefore SimpleTaskExecuter is used) I see in jconsole 5 active consumers as expected.

Spring XML file:

<bean id="listenerContainer"
	class="org.springframework.jms.listener.DefaultMessageListenerContainer">
	<property name="concurrentConsumers" value="5" />
	<property name="maxConcurrentConsumers" value="10/>
	<property name="connectionFactory" ref="connectionFactory" />
	<property name="destinationName" value="${server.jms.queue.jndiName}" />
	<property name="destinationResolver" ref="destinationResolver" />
	<property name="messageListener" ref="callbackClientMDP" />
	<property name="sessionTransacted" value="true"/>
	<property name="taskExecutor" ref="taskExecutor"/>
	<property name="idleConsumerLimit" value="${client.jms.listener.idleConsumerLimit}" />
	<property name="idleTaskExecutionLimit" value="${client.jms.listener.idleTaskExecutionLimit}" />
	<property name="receiveTimeout" value="${client.jms.listener.receiveTimeout}" />
</bean>

<!-- Task executer for excuting listeners-invokers.--> 
<bean id="taskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor"/>

Please let me know if more info is needed.
I think this can be easily restored.

Thanks.

---

Reference URL: http://forum.springsource.org/showthread.php?t=93068

[spring-projects-issues]

Juergen Hoeller commented

I'm afraid this is a misunderstanding: If you refer to an external thread pool there, that thread pool needs to have at least as many threads configured as the DefaultMessageListenerContainer intends to use. Note that a single ThreadPoolTaskExecutor may be shared by any number of DefaultMessageListenerContainers!

That said, if all you're concerned about is reusing threads, simply stick with a standard DefaultMessageListenerContainer setup: This will create and hold a number of threads according to the 'concurrentConsumers' setting, for the liftime of the DefaultMessageListenerContainer instance. This is usually very efficient.

In other words, the benefit of specifying an external ThreadPoolTaskExecutor is exactly that option to share it with other DefaultMessageListenerContainers (or even with any other TaskExecutor client). If all you want is thread reuse, then let DefaultMessageListenerContainer manage its threads internally.

Juergen

[spring-projects-issues]

Tovi Sofer commented

Thanks for the clarification.
I didn't fully undesrtand your explanation - if I want to have dynamic number of consumers using concurrentConsumers and maxConcurrenctConsumers, and I want the thread reuse for these dynamic consumers - Then I don't need to change the container taskExecuter?
If I don't change it, When dynamically increasing number of consumers - from where will the threads be taken? (SimpleTaskExcuter creates new thread for each task)

[spring-projects-issues]

Juergen Hoeller commented

Point taken, if you're using dynamic scheduling, there will be new threads created and destroyed every now and then. However, you may configure DefaultMessageListenerContainer to hold on to its threads for as long as possible: see 'idleTaskExecutionLimit' and related properties. As long as messages are incoming, invokers will be kept active and threads will be reused; also, the basic 'concurrentConsumers' number of threads will always be reused. Just the extra threads up to 'maxConcurrentConsumers' will be destroyed again if not enough messages come in.

Note that the key point of dynamic scaling is to increase and decrease the number of threads; otherwise you could simply set 'maxConcurrentConsumers' to 10, which would reuse 10 threads for the lifetime of the application. Whether DefaultMessageListenerContainer does the scaling itself or an external thread pool does it instead, doesn't make much difference; in both cases, extra threads will be created and destroyed as justified by the current load. As I outlined, the primary advantage of an external thread pool is that it may be reused by multiple DefaultMessageListenerContainers: You may for example configure a thread pool size of 30, with five DefaultMessageListenerContainers pointing to the same thread pool, and each of those allowed to scale up to 10 maxConcurrentConsumers. This allows for reusing threads even if the load between different message destinations is very uneven.

So if you would nevertheless like to use a ThreadPoolTaskExecutor, consider setting its 'corePoolSize' to 10 and switching its 'allowCoreThreadTimeOut' flag to "true". The ThreadPoolTaskExecutor will then be allowed to serve up to 10 threads on demand, and also to destroy them when they are not needed. Note that the 'maxPoolSize' setting is a bit different there: Those 'max' threads will only be used once the thread pool's queue is full. That's why I'd recommend a higher 'corePoolSize' with 'allowCoreThreadTimeOut' instead.

Juergen