An incorrectly configured rendezvous channel & poller causes a memory leak. [INT-2835]

[spring-operator]

Tom Greasley opened INT-2835 and commented

By naively configuring a rendezvous channel it's very easy to create a serious memory leak.

A rendezvous queue blocks message receivers until there is a message to process. However, due to the way pollers work, polling tasks will continue to be created and queued in the task executer according to schedule. The task executor threads will be stalled as they are blocked for messages from the rendezvous channel.

An in-balance between sender and poller frequency will mean that the task queue could grow uncontrolled.

A solution is to ensure a poll frequency that matches the frequency of arriving messages or to configure the poller's task executor with a suitable queue capacity.

It might be nice to have a brief mention of this in the documentation.

---

Affects: 2.1.3

Attachments:

• INT-2835.zip (9.09 kB)

1 votes, 3 watchers

[spring-operator]

Gary Russell commented

I can only reproduce this by setting the poller's receive-timeout to -1 (infinity).

With a zero, or positive timeout, the thread returns (the default timeout is 1 second).

Can you confirm that you are using -1 for the timeout? Is there any particular reason why?

[spring-operator]

Tom Greasley commented

No, I'm not using a timeout of -1. I'm just putting together a simple test app.

[spring-operator]

Tom Greasley commented

Or rather I'm not specifically setting a timeout.

[spring-operator]

Tom Greasley commented

Simple application reproducing the problem.

[spring-operator]

Tom Greasley commented

A sample log from the application, note the task queue length:

2012-11-27 15:18:14,769 [task-scheduler-1] INFO com.neoworks.int2835.TaskExecutorMonitor - Task queue length: 1.
2012-11-27 15:18:14,769 [task-scheduler-2] INFO com.neoworks.int2835.Sender - Sending: 0.
2012-11-27 15:18:14,830 [receivingServiceExecutor-1] INFO com.neoworks.int2835.Receiver - Received: 0
2012-11-27 15:18:15,768 [task-scheduler-8] INFO com.neoworks.int2835.TaskExecutorMonitor - Task queue length: 11.
2012-11-27 15:18:15,821 [task-scheduler-1] INFO com.neoworks.int2835.Sender - Sending: 1.
2012-11-27 15:18:15,821 [receivingServiceExecutor-6] INFO com.neoworks.int2835.Receiver - Received: 1
2012-11-27 15:18:16,768 [task-scheduler-8] INFO com.neoworks.int2835.TaskExecutorMonitor - Task queue length: 21.
2012-11-27 15:18:16,822 [task-scheduler-7] INFO com.neoworks.int2835.Sender - Sending: 2.
2012-11-27 15:18:16,822 [receivingServiceExecutor-6] INFO com.neoworks.int2835.Receiver - Received: 2
2012-11-27 15:18:17,768 [task-scheduler-10] INFO com.neoworks.int2835.TaskExecutorMonitor - Task queue length: 31.
2012-11-27 15:18:17,823 [task-scheduler-2] INFO com.neoworks.int2835.Sender - Sending: 3.
2012-11-27 15:18:17,823 [receivingServiceExecutor-6] INFO com.neoworks.int2835.Receiver - Received: 3
2012-11-27 15:18:18,768 [task-scheduler-5] INFO com.neoworks.int2835.TaskExecutorMonitor - Task queue length: 41.
2012-11-27 15:18:18,824 [task-scheduler-10] INFO com.neoworks.int2835.Sender - Sending: 4.
2012-11-27 15:18:18,824 [receivingServiceExecutor-6] INFO com.neoworks.int2835.Receiver - Received: 4
2012-11-27 15:18:19,768 [task-scheduler-4] INFO com.neoworks.int2835.TaskExecutorMonitor - Task queue length: 51.
2012-11-27 15:18:19,825 [task-scheduler-5] INFO com.neoworks.int2835.Sender - Sending: 5.
2012-11-27 15:18:19,825 [receivingServiceExecutor-6] INFO com.neoworks.int2835.Receiver - Received: 5
2012-11-27 15:18:20,769 [task-scheduler-6] INFO com.neoworks.int2835.TaskExecutorMonitor - Task queue length: 61.
2012-11-27 15:18:20,827 [task-scheduler-2] INFO com.neoworks.int2835.Sender - Sending: 6.
2012-11-27 15:18:20,827 [receivingServiceExecutor-6] INFO com.neoworks.int2835.Receiver - Received: 6
2012-11-27 15:18:21,769 [task-scheduler-6] INFO com.neoworks.int2835.TaskExecutorMonitor - Task queue length: 71.
2012-11-27 15:18:21,828 [task-scheduler-1] INFO com.neoworks.int2835.Sender - Sending: 7.
2012-11-27 15:18:21,828 [receivingServiceExecutor-6] INFO com.neoworks.int2835.Receiver - Received: 7

[spring-operator]

Tom Greasley commented

Appologies, there is a slight error in my sample application.

The monitor method of the TaskExecutorMonitor should read:

[spring-operator]

Tom Greasley commented

Appologies, there is a slight error in my sample application.

The monitor method of the TaskExecutorMonitor should read:

[spring-operator]

Tom Greasley commented

@Scheduled(fixedRate = 1000)
public void monitor()
{
final int size = executor.getThreadPoolExecutor().getQueue().size();
LOG.info("Task queue length: {}.", size);
}

[spring-operator]

Tom Greasley commented

Also, setting an explicit and short receive-timeout does indeed prevent the queue growing.

[spring-operator]

Tom Greasley commented

Appologies for the stream of consciousness. I've just figured out what I was missing here. I was underestimating the importance of setting the receive timeout when polling queued channels. I assumed that a poll of an empty queue would return by default rather than block and wait for a message. Using a timeout longer than the rate/delay can only make the queue grow. This applies to all of the queued channel implementations.

[spring-operator]

Tom Greasley commented

The documentation states:

[spring-operator]

Tom Greasley commented

"For example, consider two options that seem similar on the surface but are actually quite different: the first has an interval trigger of 5 seconds and a receive timeout of 50 milliseconds while the second has an interval trigger of 50 milliseconds and a receive timeout of 5 seconds."

In the second case the executor queue will grow uncontrolled unless it is also configured with a capacity and rejection policy.

[spring-operator]

Gary Russell commented

The RendevousChannel is rather specialized; when run with a timeout, you might not get what you expected because there likely is not another thread there waiting for the handoff and the sending thread will pause until the next poll.

You may find that the ExecutorChannel is a better bet because, with it, you get an immediate handoff to the task executor thread (if one is available).

On the other hand, the sending thread won't be paused if there is no thread available.

[spring-operator]

Tom Greasley commented

Thanks for your reply. A blocked sender was actually the behaviour I was after in this instance. However the problem I was seeing would also apply to a standard queued channel.

I think the problem stems from me not appreciating the need to tune the pollers when dealing with queues. Consider a simple queued channel and poller, where the poller has a trigger rate that is less than the receive timeout. If the inbound message rate is less than the trigger rate then the task queue will grow with pollers waiting for messages that don't come. Conversely if the inbound message rate is greater than the trigger rate then the channel queue will grow with messages waiting for a poller to pick them up. It's a balancing act involving message rate, trigger rate, receive timeout and thread pool size.

The core issue I had was a low message rate and a trigger rate that was less than the default receive timeout. Under these circumstances, the executor queue can only grow.

[spring-operator]

Artem Bilan commented

It is not clear what else to do here. There is pretty comprehensive doc on the matter: https://docs.spring.io/spring-integration/docs/current/reference/html/endpoint.html#endpoint-pollingconsumer.

Plus it is just a direct fact with any executors that task queue is getting full if there is not enough working threads. The CallerBlocksPolicy can be used to avoid an overhead with producer.