Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

进程跑一天左右会异常僵死和无异常 #36

Closed
Mothpro opened this issue May 29, 2018 · 13 comments
Closed

进程跑一天左右会异常僵死和无异常 #36

Mothpro opened this issue May 29, 2018 · 13 comments

Comments

@Mothpro
Copy link

Mothpro commented May 29, 2018

版本:predixy predixy-1.0.5-pref
配置参数:
Name PredixyTrade
Bind 0.0.0.0:22152
WorkerThreads 4
MaxMemory 0
ClientTimeout 60
Log /data/srv/store/logs/predixy.log
LogRotate 1d
LogVerbSample 0
LogDebugSample 0
LogInfoSample 10000
LogNoticeSample 1
LogWarnSample 1
LogErrorSample 1
Include auth.conf
Include try.conf
Include latency.conf

auth.conf:
Authority {
Auth {
Mode write
}
Auth "#a complex password#" {
Mode admin
}
}

try.conf:
SentinelServerPool {
Databases 16
Hash crc16
Distribution modula
MasterReadPriority 0
StaticSlaveReadPriority 100
DynamicSlaveReadPriority 100
KeepAlive 120
Sentinels {
+ 0.0.0.1:17021
}
Group configs {
+ 0.0.0.2:22151
+ 0.0.0.3:22151
}
}

刚使用的第一天凌晨0点左右有报错宕机
qq20180529-101621

picture

第二日尝试编写shell脚本让其进程while tre自动运行,凌晨3点宕机无错误日志,重启进程后正常
qq111

@fortrue
Copy link
Member

fortrue commented May 29, 2018

应该和 #32 是同一个问题,你这边能复现吗?一直找不到复现的方式,或者出现这种情况的时候,用pstack看一下进程运行栈发出来看看。还有把日志发出来

@Mothpro
Copy link
Author

Mothpro commented May 29, 2018

目前为止每日都会出现,应该是可以复现的,当出现这个问题时我pstack一下看看
@fortrue

@fortrue
Copy link
Member

fortrue commented May 29, 2018

如果你是用的master分支的话,可以先换到issue32分支https://github.com/joyieldInc/predixy/tree/issue32

@Mothpro
Copy link
Author

Mothpro commented May 29, 2018

谢谢,已经切换issue32分支重新编译后重启,试探过pstack,有如下返回(目前是正常运行状况,尚无宕机)
222
不知这个是否会有影响?

@Mothpro
Copy link
Author

Mothpro commented May 30, 2018

predixy.log

@fortrue
今日一共崩溃3次,最后一次捕捉到完整的错误日志

运行环境是centOS 7.3

@fortrue
Copy link
Member

fortrue commented May 30, 2018

这个倒是和你之前贴的一样,以及和 #33 也一样。一天崩溃三次,看来像是某个特定的请求触发了bug,不知道你这边主要是些什么请求?

@Mothpro
Copy link
Author

Mothpro commented May 31, 2018

主要请求是hget、hset、hdel、hgetall、expire,其余请求没有

@Mothpro
Copy link
Author

Mothpro commented May 31, 2018

@fortrue
今天又挂了4次,一下是具体挂的时间点
23

目前看来崩溃还是挺频繁的。。。。

@fortrue
Copy link
Member

fortrue commented May 31, 2018

你这个请求量大吗?如果不大的话你看把日志都打开,记录一下完整日志我看看,也就是把LogDebugSample、LogVerbSample、LogInfoSample都设置为1

@Mothpro
Copy link
Author

Mothpro commented Jun 4, 2018

@fortrue 除了这个记录完整日志还有什么别的办法么。。我打开了所有日志以后,按每分钟来切割日志文件,每分钟的文件大小都1.5G左右。。。 :(

@fortrue
Copy link
Member

fortrue commented Jun 5, 2018

那看来你的请求量还比较大,要不你启动predixy前先把生成core文件设置打开,这样出现问题的时候让它生成core文件,然后把core发给我看看

@Mothpro
Copy link
Author

Mothpro commented Jun 8, 2018

@fortrue 发现一个现象,每次崩溃时,predixy的链接数都在1.5W以上,会不会是这个问题导致的?

@fortrue
Copy link
Member

fortrue commented Jun 8, 2018

1.5w连接倒也不算高,要是有更丰富的监控就好了,你可以看看下载CacheEasy试用,看看其监控指标,然后把你的predixy实例都加进来

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants