Setup of JMS message listener invoker failed for destination 'queue-name' - trying to recover. Cause: Local transactions cannot span multiple top-level entities such as queue or topic. TrackingId:some_tracking_id, SystemTracker:gateway5, Timestamp:2024-02-29T21:57:11 [condition = amqp:not-allowed]

[Ahmetanakol]

Library name and version

spring-cloud-azure-starter-servicebus-jms 5.9.1

Query/Question

Hello Team,

We are using JmsListener to consume messages from servicebus queues. We have multiple queues on servicebus namespace.

This is how we annotate each listener method.
@JmsListener(destination = "ServicebusQueueName", containerFactory = "jmsListenerContainerFactory")

JmsTemplate is used to enqueue messages

      jmsTemplate.send(
          queueName,
          session -> {
            var message = jmsTemplate.getMessageConverter().toMessage(content, session);
            AmqpJmsMessageFacade facade = (AmqpJmsMessageFacade) ((JmsTextMessage) message).getFacade();

            facade.setTracingAnnotation(
                "x-opt-scheduled-enqueue-time",
                Date.from(OffsetDateTime.now()
                    .plusSeconds(5)
                    .toInstant()));
            return message;
          });

Whenever a message is received, we processed it and then enqueue a new message to one of our queues. What we realized is that we see a lot of warnings in our application logs.

Open of resource:(JmsConsumerInfo: { ID:some_id destination = ServicebusQueueName }) failed: Local transactions cannot span multiple top-level entities such as queue or topic. TrackingId:some_tracking_id, SystemTracker:gateway5, Timestamp:2024-03-01T10:23:23 [condition = amqp:not-allowed]

We are able to receive and process the incoming message and then we send it to a different or same destination with updated data. The reported warning doesn't prevent us from receiving and sending messages, however I would like to understand why it is happening. Any help would be appreciated.

Thank you

Environment

Azure Servicebus - Queues hosted on Azure cloud platform

Project Dependencies:
Java 17
spring-boot-starter-parent 3.2.3
spring-jms 6.1.4
jakarta.jms-api 3.1.0
spring-cloud-azure-starter-servicebus-jms 5.9.1

Spring Boot Servicebus JMS Configuration:

spring:
jms:
servicebus:
connection-string: SERVICEBUS_CONNECTION_STRING
pricing-tier: 'premium'
idle-timeout: 1800000
pool:
enabled: true

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @chenrujun @moarychan @Netyyyy @saragluna.

[Netyyyy]

Hi @Ahmetanakol , Thank you for reporting this issue. We have received your submission and will take a look. Could you help provide a minimal project and complete logs?
Please feel free to provide any additional information or context that you think may be helpful. We'll keep you updated on the progress of our review.

[joshfree]

Possibly related issue: #29881

[Ahmetanakol]

Hello @Netyyyy , I created a sample project in order to simulate the behaviour we are seeing in our production application. However I am not sure what triggers this behaviour. Meanwhile, I can share some of the logs that contains tracking ids I believe they can help you with the investigation.

Log level: WARN

Logger Class: org.apache.qpid.jms.provider.amqp.builders.AmqpResourceBuilder

Message: Open of resource:(JmsConsumerInfo: { ID:d7f8d045-3753-4eb3-b676-cce0b0f8cc71:14:12:63982, destination = fueling-process-initiated }) failed: Local transactions cannot span multiple top-level entities such as queue or topic. TrackingId:f2e784d0b104442092dfeed33a988b35_G5S2, SystemTracker:gateway5, Timestamp:2024-03-04T14:10:35 [condition = amqp:not-allowed]

I am attaching the sample project too. You can see how we send messages to the queues and consume those messages via JmsListeners.

demo.zip

[Netyyyy]

Hi @Ahmetanakol , thanks for reply, but there seems to be some problem opening your demo, could you help check about that?

[Ahmetanakol]

Hi @Netyyyy I am able to download the zip file. I am not sure why you are having trouble. I uploaded it again. Additionally, you can also find it here https://github.com/Ahmetanakol/demo/tree/main/demo