motan rpc framework 连接自动断开重连

[yelllowhat]

描述：

在 motan 0.2.1 时多服务间会随着频繁请求产生连接阻塞问题，最终导致 out of heap, 升级到 motan 0.2.2 后，连接正常了，运行也不会出连接阻塞问题， 观察日志发现，每隔几分钟就出现自动断开重连的情况

日志：

2017-01-05 18:14:33,772 INFO info:55 - NettyChannelHandler channelConnected: remote=/127.0.0.1:7003 local=/127.0.0.1:62899 event=UpstreamChannelStateEvent
2017-01-05 18:14:33,825 INFO info:55 - NettyChannelHandler channelDisconnected: remote=/127.0.0.1:7003 local=/127.0.0.1:56965 event=UpstreamChannelStateEvent
2017-01-05 18:14:33,825 INFO info:55 - NettyChannelFactory_127.0.0.1_7003 client disconnect Success: motan://127.0.0.1:7003/com.csf.service.user.service.IPlatformService
2017-01-05 18:14:33,826 INFO info:55 - NettyChannelHandler channelDisconnected: remote=/127.0.0.1:7003 local=/127.0.0.1:56964 event=UpstreamChannelStateEvent
2017-01-05 18:14:33,826 INFO info:55 - NettyChannelFactory_127.0.0.1_7003 client disconnect Success: motan://127.0.0.1:7003/com.csf.service.user.service.IPlatformService
2017-01-05 18:14:33,826 INFO info:55 - NettyChannelHandler channelDisconnected: remote=/127.0.0.1:7003 local=/127.0.0.1:57432 event=UpstreamChannelStateEvent
2017-01-05 18:14:33,826 INFO info:55 - NettyChannelFactory_127.0.0.1_7003 client disconnect Success: motan://127.0.0.1:7003/com.csf.service.user.service.IPlatformService
2017-01-05 18:14:33,826 INFO info:55 - NettyChannelHandler channelConnected: remote=/127.0.0.1:7003 local=/127.0.0.1:62900 event=UpstreamChannelStateEvent
2017-01-05 18:14:33,826 INFO info:55 - NettyChannelHandler channelConnected: remote=/127.0.0.1:7003 local=/127.0.0.1:62901 event=UpstreamChannelStateEvent
2017-01-05 18:14:33,827 INFO info:55 - NettyChannelHandler channelConnected: remote=/127.0.0.1:7003 local=/127.0.0.1:62902 event=UpstreamChannelStateEvent
2017-01-05 18:14:33,914 INFO info:55 - NettyChannelHandler channelDisconnected: remote=/127.0.0.1:7003 local=/127.0.0.1:56967 event=UpstreamChannelStateEvent
2017-01-05 18:14:33,914 INFO info:55 - NettyChannelFactory_127.0.0.1_7003 client disconnect Success: motan://127.0.0.1:7003/com.csf.service.user.service.IStockService
2017-01-05 18:14:33,914 INFO info:55 - NettyChannelHandler channelDisconnected: remote=/127.0.0.1:7003 local=/127.0.0.1:56968 event=UpstreamChannelStateEvent
2017-01-05 18:14:33,915 INFO info:55 - NettyChannelFactory_127.0.0.1_7003 client disconnect Success: motan://127.0.0.1:7003/com.csf.service.user.service.IStockService
2017-01-05 18:14:33,915 INFO info:55 - NettyChannelHandler channelDisconnected: remote=/127.0.0.1:7003 local=/127.0.0.1:57435 event=UpstreamChannelStateEvent
2017-01-05 18:14:33,915 INFO info:55 - NettyChannelFactory_127.0.0.1_7003 client disconnect Success: motan://127.0.0.1:7003/com.csf.service.user.service.IStockService
2017-01-05 18:14:33,915 INFO info:55 - NettyChannelHandler channelConnected: remote=/127.0.0.1:7003 local=/127.0.0.1:62903 event=UpstreamChannelStateEvent
2017-01-05 18:14:33,915 INFO info:55 - NettyChannelHandler channelConnected: remote=/127.0.0.1:7003 local=/127.0.0.1:62904 event=UpstreamChannelStateEvent
2017-01-05 18:14:33,916 INFO info:55 - NettyChannelHandler channelConnected: remote=/127.0.0.1:7003 local=/127.0.0.1:62905 event=UpstreamChannelStateEvent

配置说明：

说明：此配置为 local 模式，没有采用 zk/consul 的方式做一致性管理

<motan:registry regProtocol="local" name="consul-registry" check="false" />
<motan:protocol id="motan-protocol" default="true" name="motan">
<motan:referer id="combinationBaseDao" directUrl="${consul.registry.host}:${service.client.default.port}"
interface="com.csf.service.user.dao.mongo.ICombinationBaseDao" />

分析：
@author maijunsheng

public void handleUpstream(ChannelHandlerContext ctx, ChannelEvent e) throws Exception {
if (e instanceof MessageEvent) {
messageReceived(ctx, (MessageEvent) e);
} else if (e instanceof WriteCompletionEvent) {
WriteCompletionEvent evt = (WriteCompletionEvent) e;
writeComplete(ctx, evt);
} else if (e instanceof ChildChannelStateEvent) {
ChildChannelStateEvent evt = (ChildChannelStateEvent) e;
if (evt.getChildChannel().isOpen()) {
childChannelOpen(ctx, evt);
} else {
childChannelClosed(ctx, evt);
}
} else if (e instanceof ChannelStateEvent) {
ChannelStateEvent evt = (ChannelStateEvent) e;
switch (evt.getState()) {
// 针对已经建立的连接进行取值判断
case CONNECTED:
if (evt.getValue() != null) {
channelConnected(ctx, evt);
} else {
channelDisconnected(ctx, evt);
}
..........................................
}
}

跟踪代码发现是自动检测引起的的重连接，但我们的应用是正常的，NettyChannelHandler , ChannelStateEvent 为什么取判断值是 null ，导致自动关闭，然后在重新连接。

请作者帮助分析剖解一下，谢谢

[rayzhang0603]

正常的逻辑中有两个地方会断开链接
1、reqeust发送异常（包括发送超时）会销毁链接，见NettyClient类line 176
2、链接池维护进程进行空闲链接回收，当空闲链接大于minClientConnection时进行。链接池创建见AbstractPoolClient类 line 43，initPool方法

ChannelStateEvent为null 应该是上述两种机制调用了netty channel的close方法触发的disconnect事件。

[yelllowhat]

好的，谢谢 rayzhang0603 , 经调查是 第二种情况 "当空闲链接大于minClientConnection时进行的,空闲链接回收" , 在配置项中可否通过 motan:protocol 指定空闲时间，如果没有只能通根据业务情况，适当的增大 minClientConnection size 。

[rayzhang0603]

目前无法配置空闲时间，可以考虑适当加大minClientConnection