New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

motan ThreadProtectedRequestRouter reject request #353

Open
yelllowhat opened this Issue Feb 9, 2017 · 2 comments

Comments

Projects
None yet
2 participants
@yelllowhat

yelllowhat commented Feb 9, 2017

motan 0.2.2 线上运行一段时间一直稳定,最近集中报出 多种错误最终导致 oom, 调查了几天也没有重现,这些问题在这里记录一下,看看大家的见解

  1. ThreadProtectedRequestRouter reject request
    2017-02-09 17:53:37,452 ERROR error:87 - RefererInvocationHandler invoke Error: uri=motan://10.0.251.14:0/com.csf.service.user.service.IPlatformService requestId=1558848751479747540 interface=com.csf.service.user.service.IPlatformService method=checkUserApiPrivilege(java.lang.String,java.lang.String)
    com.weibo.api.motan.exception.MotanServiceException: error_message: ThreadProtectedRequestRouter reject request: request_counter=22 total_counter=29 max_thread=32, status: 503, error_code: 10002,r=null

  2. NettyChannel failed to connect to server

2017-02-09 14:51:08,189 ERROR error:87 - NettyClient init pool create connect Error: url=motan://127.0.0.1:7003/com.csf.test.motan.IMotanMessageService
com.weibo.api.motan.exception.MotanServiceException: error_message: NettyChannel failed to connect to server, url: motan://127.0.0.1:7003/com.csf.test.motan.IMotanMessageService, result: true, success: false, connected: false, status: 503, error_code: 10001,r=null
at com.weibo.api.motan.transport.netty.NettyChannel.open(NettyChannel.java:150)
at com.weibo.api.motan.transport.netty.NettyChannelFactory.makeObject(NettyChannelFactory.java:53)
at org.apache.commons.pool.impl.GenericObjectPool.addObject(GenericObjectPool.java:1691)
at com.weibo.api.motan.transport.AbstractPoolClient.initPool(AbstractPoolClient.java:64)
at com.weibo.api.motan.transport.netty.NettyClient.open(NettyClient.java:217)
at com.weibo.api.motan.protocol.rpc.DefaultRpcProtocol$DefaultRpcReferer.doInit(DefaultRpcProtocol.java:196)
at com.weibo.api.motan.rpc.AbstractNode.init(AbstractNode.java:46)
at com.weibo.api.motan.protocol.AbstractProtocol.refer(AbstractProtocol.java:100)
at com.weibo.api.motan.protocol.support.ProtocolFilterDecorator.refer(ProtocolFilterDecorator.java:70)
at com.weibo.api.motan.cluster.support.ClusterSupport.notify(ClusterSupport.java:175)
at com.weibo.api.motan.cluster.support.ClusterSupport.init(ClusterSupport.java:82)
at com.weibo.api.motan.config.handler.SimpleConfigHandler.buildClusterSupport(SimpleConfigHandler.java:55)
at com.weibo.api.motan.config.RefererConfig.createClusterSupport(RefererConfig.java:184)
at com.weibo.api.motan.config.RefererConfig.initRef(RefererConfig.java:128)
at com.weibo.api.motan.config.RefererConfig.getRef(RefererConfig.java:85)
at com.weibo.api.motan.config.springsupport.RefererConfigBean.getObject(RefererConfigBean.java:44)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1517)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:251)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.autowireResource(CommonAnnotationBeanPostProcessor.java:461)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.getResource(CommonAnnotationBeanPostProcessor.java:435)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor$ResourceElement.getResourceToInject(CommonAnnotationBeanPostProcessor.java:559)
at org.springframework.beans.factory.annotation.InjectionMetadata$InjectedElement.inject(InjectionMetadata.java:169)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessPropertyValues(CommonAnnotationBeanPostProcessor.java:305)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1210)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:755)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)
at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:83)
at com.csf.RestMotanMain.main(RestMotanMain.java:17)

排查时,每隔一定的时间 NettyClient init pool create connect 初始连接服务出错

@rayzhang0603

This comment has been minimized.

Show comment
Hide comment
@rayzhang0603

rayzhang0603 Feb 9, 2017

Collaborator

@yelllowhat 第一个问题是server端业务处理线程数较小,导致线程池reject。建议生产环境配置maxWorkerThread="800" minWorkerThread="200"。
第二个问题是client与server端建立链接时失败,可以重点排查网络状况或路由限制之类。也有可能跟第一个问题相关

Collaborator

rayzhang0603 commented Feb 9, 2017

@yelllowhat 第一个问题是server端业务处理线程数较小,导致线程池reject。建议生产环境配置maxWorkerThread="800" minWorkerThread="200"。
第二个问题是client与server端建立链接时失败,可以重点排查网络状况或路由限制之类。也有可能跟第一个问题相关

@yelllowhat

This comment has been minimized.

Show comment
Hide comment
@yelllowhat

yelllowhat Feb 9, 2017

谢谢 @rayzhang0603

第一个问题,我已经修改了线程数,参照 URLParamType enum类,重点是第二个问题,怀疑频率检测连接并关闭重连,我尝试调整 minClientConnection 与 maxClientConnection 相同,发现还是有服务连接失败的问题,这也可能跟 netty 相关

yelllowhat commented Feb 9, 2017

谢谢 @rayzhang0603

第一个问题,我已经修改了线程数,参照 URLParamType enum类,重点是第二个问题,怀疑频率检测连接并关闭重连,我尝试调整 minClientConnection 与 maxClientConnection 相同,发现还是有服务连接失败的问题,这也可能跟 netty 相关

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment