[BUG] JMSTemplate is invalid state after 10 minutes idle #31966
Comments
ghost
added
the
needs-triage
stliu
added
bug
ghost
removed
the
needs-triage
|
Accoring to this doc https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-amqp-troubleshoot#connection-is-closed an AMQP Link will be closed after 10 minutes idle
an AMQP Connection will be closed after all links are closed
And the last one
To summary
So, for a long-running application which is very busy at daytime ( 1000 message per second ) and very low traffic at night ( 1 message per hour ), what is the strategy here? first and foremost, whenever that message comes, so, it leaves us with two options:
|
AMQP protocol tracing analysis
Full AMQP tracing can be found here 8e31f0b3a82e20f391312a622083d9d5 |
|
After some internal discussion, @shankarsama would consider to adding an AMQP property for making the link timeout configurable, which we can leverage to fix this issue. And @shankarsama , would you help to update here when that configuration is released in the service side? |
|
But @yiliuTo, your proposal will NOT FIX the issue. In microsoft/azure-spring-boot#817 (comment) you've clearly stated that here you're gonna provide a fix, not a workaround. While the option to configure EFFECTIVE timeout (which ServiceBus will respect for a change), is a nice feature, prolonging or disabling timeout WILL NOT FIX the underlying issue that JmsTemplate provided by your starter cannot recover from obviously recoverable exceptions. Suggestion: leave the timeouts be, just make the JmsTemplate able to recover internally when a known situation like this happens. I will repeat myself, but your other library |
@zoladkow , thanks for your suggestion, we will consider about it. |
|
any update on this issue? |
No updates at the moment. Rest assured, we are actively try to resolve it and will keep you informed if there are any developments. |
|
@Netyyyy @saragluna can you update this issue? |
Hi @neffsvg , we are still working on this issue. |
|
@Netyyyy @vinaysurya we are having a go live deadline in a view weeks. And QA needs to check everything first. So high pressure on us. Is there a pr or anything we can track the progress? Given that the last updates were May 26 and 2 days ago. @zoladkow are you still using this? Maybe we can join forces with them to solve this. |
@neffsvg Not anymore, we spotted the issue very early (and long ago) and decided instead go with plain azure-servicebus library. Granted, it did not provide all the @JmsListener or JmsTemplate Spring stuff, but that's absolutely not a problem. The most important thing was, that Queue/TopicClient would not go into invalid state after idle disconnect, just handle this internally like common sense would dictate. Oh, and this issue is by far not new. You can check the previous one here: microsoft/azure-spring-boot#817 Given that only real solution would require to redo this starter on top of azure-servicebus (or whatever they changed the name too - yeah, naming convention changes, top priority, eh...), which would require to actually provide adapters to work with Spring JMS, i can see why they are so reluctant - the gain seems awfully too small to justify such an effort. Volunteer effort especially. |
|
This issue should be addressed for now. We have made a change on ServiceBus service side to not enforce the expiring of idle links aggressively for JMS customers that come through azure-servicebus-jms libraries (which is the dependent library used by spring-cloud-azure-starter-servicebus-jms, version 4.x ). This is only available for premium messaging namespaces. There are still active quota enforcements on the number of active links that can be had for a namespace at any given point in time. The real long term fix will likely need a fix involving Qpid JMS library, where a producer object on client is not immediately considered close on receiving a link close. Instead when using the jmsproducer object, the qpid jms client library has to check if the underlying link is in closed state and if so re-create the underneath amqp link again. |
|
Please note that the fix on the ServiceBus service side is in the process of deployment. We expect the fix to be deployed in about 3 weeks from today across all ServiceBus clusters. |
|
I had the same issue and I fixed it by enabling pool in spring boot More information |
Describe the bug
When using
JMSTemplateto send message to Service Bus, it throws exception if the connection is idle more than 10 minutesanother issue. is the performance, it takes 5 seconds to send the first message, see the log below for more detials
Exception or Stack Trace
To Reproduce
Add the code snippet that causes the issue.
it can be reproducecd by this code
I'm using
Expected behavior
There are two problems with this issue:
the underlying connection is not kept alive
with tracing log enabled, we can see, there is a
IdleTimeoutCheckis running every 60 seconds to try to keep the connection alive by sending an Empty Frame, but seems this doesn't keep the connection alive, so how to keep the connection alive? is it related to Azure/azure-sdk-for-python#10209?org.springframework.jms.connection.SingleConnectionFactory#reconnectOnExceptionis not honoured.Spring Cloud Azure uses
CachingConnectionFactoryby default, according to its javadocFrom the current behavior of this exception, it doesn't auto recovery from the underlying connection.
I have the full log attached below
first message delivered!
why it take 4 seconds to send a single message???
can we do something to speed up this? warmup during bootstrap process?
--- seems Server send this message every 3 minutes
---> looks like this IdleTimeoutCheck doesn't work, maybe Azuer Service bus doesn't expect an empty frame?
--> Now in LINK_FINAL state, takes 10 minutes, is it a hard limit from Service Bus? is there a doc saying that? Azure/azure-sdk-for-python#10127
Azure/azure-sdk-for-python#10209
--> About to send a new message, the first one was sent in 2022-11-06 10:57:01.630, and we have a 15 minutes scheduler
The text was updated successfully, but these errors were encountered: