Error Setup of JMS message listener invoker failed for destination <subscription> - trying to recover. Cause: The Session is closed

[JinuJoseph79]

Describe the bug
We are using Azure service bus with topics and subscription. The messages in the subscription are processed by our springboot application deployed on Azure Web App. We use spring-cloud-azure-starter-servicebus-jms for sending and receiving messages. We have issue that web app stops listening to Service bus and messages are not processed. In the application insight logs we see "Setup of JMS message listener invoker failed for destination - trying to recover. Cause: The Session is closed" and ","Setup of JMS message listener invoker failed for destination '' - trying to recover. Cause: Unknown error from remote' for each of the subscription configured for listening.

Exception or Stack Trace

3/5/2024, 8:03:09.867 PM,"Setup of JMS message listener invoker failed for destination '' - trying to recover. Cause: Unknown error from remote peer",2,trace,"{""LoggerName"":""org.springframework.jms.listener.DefaultMessageListenerContainer"",""SourceType"":""Logger"",""ThreadName"":""org.springframework.jms.JmsListenerEndpointContainer#0-18""}"

2/23/2024, 8:01:00.788 PM,"Setup of JMS message listener invoker failed for destination '' - trying to recover. Cause: The Session is closed",2,trace,"{""LoggerName"":""org.springframework.jms.listener.DefaultMessageListenerContainer"",""SourceType"":""Logger"",""ThreadName"":""org.springframework.jms.JmsListenerEndpointContainer#2-17""}"

To Reproduce
Steps to reproduce the behavior: Its occurring randomly. In the past 3 months it has occurred around 4 or 5 times in Production

Code Snippet
We have followed Microsoft documentation for configuring our Service bus.
Use JMS in Spring to access Azure Service Bus - Java on Azure | Microsoft Learn

For listening we use

@JmsListener(destination = TOPIC_NAME, containerFactory = "topicJmsListenerContainerFactory",
        subscription = SUBSCRIPTION_NAME)
 
For Sending we use 
jmsTemplate.send(
          topic,
          new MessageCreator() {
            @Override
            public Message createMessage(Session session) throws JMSException {
              BytesMessage byteMessage = session.createBytesMessage();
              byteMessage.writeBytes(message.getBytes());
              return setHeaderProperties(byteMessage, type);
            }
          });

application.properties:
spring.jms.servicebus.connection-string=
spring.jms.servicebus.pricing-tier=Premium
spring.jms.servicebus.prefetch-policy.all=5
spring.jms.servicebus.prefetch-policy.durable-topic-prefetch=5
spring.jms.servicebus.prefetch-policy.topic-prefetch=5
spring.servicebus.concurrency=5-40

Expected behavior
Messages should be processed continuously without any disruption

Screenshots
If applicable, add screenshots to help explain your problem.

Setup (please complete the following information):

• OS: Azure Web App Linux
• IDE: Error only on Cloud
• Library/Libraries: spring-cloud-azure-starter-servicebus-jms, 5.8.0
• Java version: 17
• App Server/Environment: Web App
• Frameworks: Springboot 3.2.3

If you suspect a dependency version mismatch (e.g. you see NoClassDefFoundError, NoSuchMethodError or similar), please check out Troubleshoot dependency version conflict article first. If it doesn't provide solution for the problem, please provide:

• verbose dependency tree (mvn dependency:tree -Dverbose)
• exception message, full stack trace, and any available logs

Additional context
Add any other context about the problem here.

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• [x ] Bug Description Added
• Repro Steps Added
• Setup information Added

[joshfree]

@yiliuTo could you please take a look

[Netyyyy]

Hi @JinuJoseph79 ,Thank you for reporting this issue. We have received your submission and will take a look. We appreciate your input and will review this matter as soon as possible.
Please feel free to provide any additional information or context that you think may be helpful. We'll keep you updated on the progress of our review.

[Netyyyy]

Unable to get details about the error, suspect it is related to service bus server side, let's wait for their fix first

[github-actions]

Hi @JinuJoseph79. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[joshfree]

Adding needs-author-feedback as we're waiting for a reply from @JinuJoseph79

[github-actions]

Hi @JinuJoseph79, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[JinuJoseph79]

Hi, Is the fix from service bus rolled out? We still have the problem that sometime our backend stops listening to messages from servicebus. We see "Cause: Unknown error from remote peer" in the logs

[Netyyyy]

Hi @JinuJoseph79 , sorry for the late reply, could u help provide the namespace and region to check if its already deployed or not?

[JinuJoseph79]

Hi @Netyyyy ,
Namespace: sb-landingzone-ntdev
Location: West Europe

[Netyyyy]

Hi @Netyyyy , Namespace: sb-landingzone-ntdev Location: West Europe
@vinaysurya please take a look to check whether this namespace and region deployed or not?

[joshfree]

pinging @vinaysurya

[Netyyyy]

Hi @JinuJoseph79 , ServiceBus server side has a fix recently for premium tier.
So if you still meet this error in premium tier, please add environment property PN_TRACE_FRM=1 and provide the log info here, thanks.