Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stuck threads in Hazlecast 3.6.2 Client for Session Replication #9060

Closed
SomannaDC opened this issue Oct 6, 2016 · 6 comments

Comments

Projects
None yet
3 participants
@SomannaDC
Copy link

commented Oct 6, 2016

Hi,

We are seeing stuck threads in Hazlecast 3.6.2 Client for Session Replication even with the Hazelcast Server is UP and running. Please find below the Exception:

[STUCK] ExecuteThread: '26' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007fd6040b8800 nid=0x7217 in Object.wait() [0x00007fd5c9741000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000071d42a598> (a com.hazelcast.client.spi.impl.ClientInvocationFuture)
        at com.hazelcast.client.spi.impl.ClientInvocationFuture.get(ClientInvocationFuture.java:88)
        - locked <0x000000071d42a598> (a com.hazelcast.client.spi.impl.ClientInvocationFuture)
        at com.hazelcast.client.spi.impl.ClientInvocationFuture.get(ClientInvocationFuture.java:74)
        at com.hazelcast.client.spi.impl.ClientInvocationFuture.get(ClientInvocationFuture.java:37)
        at com.hazelcast.client.spi.ClientProxy.invokeOnPartition(ClientProxy.java:126)
        at com.hazelcast.client.spi.ClientProxy.invoke(ClientProxy.java:120)
        at com.hazelcast.client.proxy.ClientMapProxy.executeOnKeyInternal(ClientMapProxy.java:1200)
        at com.hazelcast.client.proxy.NearCachedClientMapProxy.executeOnKeyInternal(NearCachedClientMapProxy.java:275)
        at com.hazelcast.client.proxy.ClientMapProxy.executeOnKey(ClientMapProxy.java:1195)
        at com.hazelcast.web.ClusteredSessionService.executeOnKey(ClusteredSessionService.java:172)
        at com.hazelcast.web.ClusteredSessionService.getAttribute(ClusteredSessionService.java:212)
        at com.hazelcast.web.HazelcastHttpSession.getAttribute(HazelcastHttpSession.java:116)
        at com.att.ebiz.framework.metrics.MetricsManager.getAttribute(MetricsManager.java:241)
        at com.att.ebiz.framework.metrics.MetricsManager.initTxInfo(MetricsManager.java:214)
        at com.att.ebiz.framework.metrics.MetricsManager.startEBizTx(MetricsManager.java:65)
        at com.att.ebiz.framework.metrics.MetricsManager.startEBizSpringTx(MetricsManager.java:43)
        at com.att.ebiz.framework.logging.client.filter.LoggingExtensionFilter.doFilter(LoggingExtensionFilter.java:82)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at com.hazelcast.web.WebFilter.doFilter(WebFilter.java:377)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
        at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
        at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
        at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
@mesutcelik

This comment has been minimized.

Copy link
Contributor

commented Oct 10, 2016

Hi @SomannaDC ,

Have you tried it with the latest patch release 3.6.5? Can you please also explain how this problem can be reproduced?

@pveentjer

This comment has been minimized.

Copy link
Member

commented Oct 10, 2016

What makes you conclude is actually stuck? What you are seeing here is part of the logic for a remote call. So what makes you conclude a call is stuck? Did you measure the response times?

@SomannaDC

This comment has been minimized.

Copy link
Author

commented Oct 10, 2016

@pveentjer Please find the complete stack stack trace below, lot of threads are stuck in weblogic for more than 10 minutes. Check for the following text in the weblogic log below: " configured time (StuckThreadMaxTime) of "600" seconds"

Weblogic Logs:
<Oct 5, 2016 2:47:54 PM CDT> <[STUCK] ExecuteThread: '128' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "607" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@175e5ab2[
POST /ebiz/registration/sessionCheck HTTP/1.1
Content-Length: 0
Accept: application/json, text/plain, /
Origin: http://zltv8064.vci.att.com:8118
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36
Referer: http://zltv8064.vci.att.com:8118/ebiz/emaintenance/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
Cookie: CSRF_token=d9f56dnzN9vMzj8H; encryLoginCookie=ZdjOZ4hb022LWoHB8BtUM4wCfyUQIbnvoskn0z4k1FY=; sessionMaxAliveCookie=alive; sessionCheckCookie=alive; TLTSID=52C27989A643E453403ED22ABE1FA4BF; AMCV_55633F7A534535110A490D44%40AdobeOrg=793872103%7CMCIDTS%7C17080%7CMCMID%7C69094915685670424927194160073540679194%7CMCAID%7CNONE%7CMCAAMLH-1476299321%7C7%7CMCAAMB-1476299321%7Chmk_Lq6TPIBMW925SPhw3Q; EbizCustomerInfo=RandomID:HZ7623713FFA7E408CBC310A0285F69C58&BCUserID:1602avpntm&GDBOrgObjectID:93406597; ECCUser=t0tQUmrB0jLMdsLG:t0I8wEIWxrDs2bxF:1607avpntmus; mbox=session#1475694520534-478581#1475697301|PC#1475694520534-478581.28_75#1476905041; s_cc=true; CommonHubID=HZ7623713FFA7E408CBC310A0285F69C58
X-BlueCoat-Via: d2a1ebb06ec69782
Connection: Keep-Alive
WL-Proxy-SSL: false
WL-Proxy-Client-IP: 135.28.59.140
Proxy-Client-IP: 135.28.59.140
X-Forwarded-For: 135.28.59.140
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: 1219768511

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.lang.Object.wait(Native Method)
com.hazelcast.client.spi.impl.ClientInvocationFuture.get(ClientInvocationFuture.java:88)
com.hazelcast.client.spi.impl.ClientInvocationFuture.get(ClientInvocationFuture.java:74)
com.hazelcast.client.spi.impl.ClientInvocationFuture.get(ClientInvocationFuture.java:37)
com.hazelcast.client.spi.ClientProxy.invokeOnPartition(ClientProxy.java:126)
com.hazelcast.client.spi.ClientProxy.invoke(ClientProxy.java:120)
com.hazelcast.client.proxy.ClientMapProxy.executeOnKeyInternal(ClientMapProxy.java:1200)
com.hazelcast.client.proxy.NearCachedClientMapProxy.executeOnKeyInternal(NearCachedClientMapProxy.java:275)
com.hazelcast.client.proxy.ClientMapProxy.executeOnKey(ClientMapProxy.java:1195)
com.hazelcast.web.ClusteredSessionService.executeOnKey(ClusteredSessionService.java:172)
com.hazelcast.web.ClusteredSessionService.getAttribute(ClusteredSessionService.java:212)
com.hazelcast.web.HazelcastHttpSession.getAttribute(HazelcastHttpSession.java:116)
com.att.ebiz.framework.metrics.MetricsManager.getAttribute(MetricsManager.java:241)
com.att.ebiz.framework.metrics.MetricsManager.initTxInfo(MetricsManager.java:170)
com.att.ebiz.framework.metrics.MetricsManager.startEBizTx(MetricsManager.java:65)
com.att.ebiz.framework.metrics.MetricsManager.startEBizSpringTx(MetricsManager.java:43)
com.att.ebiz.framework.logging.client.filter.LoggingExtensionFilter.doFilter(LoggingExtensionFilter.java:82)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
com.hazelcast.web.WebFilter.doFilter(WebFilter.java:377)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:178)

@SomannaDC

This comment has been minimized.

Copy link
Author

commented Oct 10, 2016

@mesutcelik We are currently using 3.6.2 and have not tried 3.6.5 since we are the just the client and our team which hosts the Hazelcast Server has enterprise license for 3.6.2.

@mesutcelik

This comment has been minimized.

Copy link
Contributor

commented Dec 13, 2016

Hi @SomannaDC ,

Are you still experiencing the problem?

@mesutcelik mesutcelik added this to the 3.7.5 milestone Dec 13, 2016

@mesutcelik

This comment has been minimized.

Copy link
Contributor

commented Dec 27, 2016

Hi @SomannaDC ,
We haven't heard from you and It could be that the issue is resolved on your side. I am closing it right now but please feel free to reopen if you are still experiencing the problem.

@mesutcelik mesutcelik closed this Dec 27, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.