Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Doc: Producing JMS messages with JmsTemplate can cause problems with Artemis cluster [SPR-17554] #22086

Open
spring-projects-issues opened this issue Dec 1, 2018 · 1 comment
Open

Doc: Producing JMS messages with JmsTemplate can cause problems with Artemis cluster [SPR-17554] #22086

spring-projects-issues opened this issue Dec 1, 2018 · 1 comment
Labels
in: messaging type: task
Milestone
General Backlog

Comments

@spring-projects-issues
Copy link
Collaborator

@spring-projects-issues spring-projects-issues commented Dec 1, 2018

Alexander Kosarev opened SPR-17554 and commented

There is a service with spring-boot-starter-activemq in dependencies. Producing JMS messages with JmsTemplate can cause problems with an Artemis cluster if the service connected to the cluster using ActiveMQ failover transport.

Message redistribution fails on one of the cluster nodes at some point, and then messages start to stick in a service queue with name like $.artemis.internal.sf.CLUSTER_NAME.NODE_NAME. The failed node can only receive messages from other cluster nodes after that but not send. Stuck messages can be delivered only after node's restart.

Typical stack trace on the failed node looks like:

2018-12-01 15:50:15,152 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: removing consumer which did not handle a message, consumer=ClusterConnectionBridge@c7532ab [name=$.artemis.internal.sf.amq-cluster.da830009-f4bb-11e8-9942-96a451134f87, queue=QueueImpl[name=$.artemis.internal.sf.amq-cluster.da830009-f4bb-11e8-9942-96a451134f87, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=ff7d54e0-f4bd-11e8-bb28-2cd05a8d61d6], temp=false]@35abb427 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@c7532ab [name=$.artemis.internal.sf.amq-cluster.da830009-f4bb-11e8-9942-96a451134f87, queue=QueueImpl[name=$.artemis.internal.sf.amq-cluster.da830009-f4bb-11e8-9942-96a451134f87, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=ff7d54e0-f4bd-11e8-bb28-2cd05a8d61d6], temp=false]@35abb427 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=amq-cluster-node01-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=192-168-1-107], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1815556070[nodeUUID=ff7d54e0-f4bd-11e8-bb28-2cd05a8d61d6, connector=TransportConfiguration(name=amq-cluster-node02-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61716&host=192-168-1-107, address=, server=ActiveMQServerImpl::serverUUID=ff7d54e0-f4bd-11e8-bb28-2cd05a8d61d6])) [initialConnectors=[TransportConfiguration(name=amq-cluster-node01-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=192-168-1-107], discoveryGroupConfiguration=null]], message=Reference[363508]:NON-RELIABLE:CoreMessage[messageID=363508,durable=false,userID=null,priority=0, timestamp=0,expiration=0, durable=false, address=ActiveMQ.Advisory.TempQueue,size=1081,properties=TypedProperties[__HDR_BROKER_IN_TIME=1543661415147,_AMQ_ROUTING_TYPE=0,__HDR_GROUP_SEQUENCE=0,__HDR_COMMAND_ID=0,__HDR_DATASTRUCTURE=[0000 0066 0800 0000 0000 0178 0100 2849 443A 6173 6B2D 6C61 7074 6F70 2D33  ...  3061 2D38 3837 392D 6139 6536 6565 3664 3564 6331 0100 0000 0000 0000 0000),_AMQ_ROUTE_TO$.artemis.internal.sf.amq-cluster.da830009-f4bb-11e8-9942-96a451134f87=[0000 0000 0006 036B),bytesAsLongs(394091],_AMQ_DUPL_ID=ID:ask-laptop-46697-1543660919081-1:1:0:0:12817,__HDR_MESSAGE_ID=[0000 004C 6E00 017B 0100 2549 443A 6173 6B2D 6C61 7074 6F70 2D34 3636 3937  ...  0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 3211 0000 0000 0000 0000),__HDR_DROPPABLE=false,__HDR_ARRIVAL=0,__HDR_PRODUCER_ID=[0000 0039 7B01 0025 4944 3A61 736B 2D6C 6170 746F 702D 3436 3639 372D 3135  ... 33 3636 3039 3139 3038 312D 313A 3100 0000 0000 0000 0000 0000 0000 0000 00),JMSType=Advisory]]@614876900: java.lang.IndexOutOfBoundsException: writerIndex: 4 (expected: readerIndex(0) <= writerIndex <= capacity(0))
	at io.netty.buffer.AbstractByteBuf.writerIndex(AbstractByteBuf.java:118) [netty-all-4.1.24.Final.jar:4.1.24.Final]
	at io.netty.buffer.WrappedByteBuf.writerIndex(WrappedByteBuf.java:129) [netty-all-4.1.24.Final.jar:4.1.24.Final]
	at org.apache.activemq.artemis.core.buffers.impl.ResetLimitWrappedActiveMQBuffer.writerIndex(ResetLimitWrappedActiveMQBuffer.java:128) [artemis-core-client-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.core.buffers.impl.ResetLimitWrappedActiveMQBuffer.<init>(ResetLimitWrappedActiveMQBuffer.java:60) [artemis-core-client-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.core.message.impl.CoreMessage.internalWritableBuffer(CoreMessage.java:360) [artemis-core-client-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.core.message.impl.CoreMessage.getBodyBuffer(CoreMessage.java:353) [artemis-core-client-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:242) [artemis-core-client-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:129) [artemis-core-client-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl.deliverStandardMessage(BridgeImpl.java:743) [artemis-server-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl.handle(BridgeImpl.java:619) [artemis-server-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.core.server.impl.QueueImpl.handle(QueueImpl.java:2983) [artemis-server-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2334) [artemis-server-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.core.server.impl.QueueImpl.access$2000(QueueImpl.java:107) [artemis-server-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:3209) [artemis-server-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.6.3.jar:2.6.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_191]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_191]
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.6.3.jar:2.6.3]2018-12-01 15:50:15,158 WARN  [org.apache.activemq.artemis.core.server.impl.QueueImpl] null: java.util.NoSuchElementException
	at org.apache.activemq.artemis.utils.collections.PriorityLinkedListImpl$PriorityLinkedListIterator.repeat(PriorityLinkedListImpl.java:172) [artemis-commons-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2353) [artemis-server-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.core.server.impl.QueueImpl.access$2000(QueueImpl.java:107) [artemis-server-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:3209) [artemis-server-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.6.3.jar:2.6.3]
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.6.3.jar:2.6.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_191]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_191]
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.6.3.jar:2.6.3]

This problem doesn't occur with Spring Boot 1.5.18+, 2.0.7+ and 2.1.0+ if default properties are used. But it occurs if older versions of dependencies are used or JMS caching is disabled or plain ActiveMQConnectionFactory used instead of CachingConnectionFactory or PooledConnectionFactory.

Example project and configuration of cluster with 2 nodes are attached.

 

Tested on Artemis 2.6.3 and relative JBoss AMQ 7

Affects: 4.3.21, 5.1.3

Attachments:
@spring-projects-issues
Copy link
Collaborator Author

@spring-projects-issues spring-projects-issues commented Dec 3, 2018

Alexander Kosarev commented

It seems like the issue is more related to Artemis, and it is solved in newer versions of Spring Boot by JMS caching. But developers may not be aware about such problems, so it would be quite good to have a note in docs about it.

@spring-projects-issues spring-projects-issues added in: messaging type: task labels Jan 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
in: messaging type: task
Projects
None yet
Milestone
General Backlog
Development

No branches or pull requests
1 participant
@spring-projects-issues