Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Binlog files preventing searchd from starting #275
Manticore Search version: Manticore 3.1.0
OS version: Debian 9.11
Build version: 445e806e@190716
Sometimes, after Manticore crashes for some reason, it is impossible to restart it because of an error related to binlog files. After removing the binlog files (binlog.meta and binlog.lock), Manticore starts normally. The issue here is that a manual intervention (removing the files) is needed before restarting searchd. This is a problem on a production environment where we want the daemon to be automatically restarted after a crash. A Google search suggests that the issue has been around for years.
Steps to reproduce: I don’t know how to trigger this bug.
Messages from log files: When the daemon refuses to start, the error message is:
Yes. We recently had one crash with normal binlog replay.
I did some more investigations.
About the searchd crashes. It seems that our recent crashes were related to a lack of disk space. They were always occurring at the time a diskspace-consuming task was triggered by a cronjob. What’s more, the stack traces are all pointing to UpdateAttributes and the SphinxAPI request dumps (in base64) are always showing a request about updating an MVA attribute (lists_id).
While I havn’t been able to reproduce the crash, my analysis is that since UpdateAttributes increases the size of binlog.001, searchd somehow crashes because of the lack of free disk space and/or some other conditions. While searchd normally recovers from such crash and replays the binlog on the next restart, in some situations, the binlog.001 file is not present (presumably because of the lack of disk space) and searchd refuses to start because of that.
So I was able to reproduce the bug by just removing the binlog.001 file before a normal restart. You can try to start searchd with this binlog to reproduce the issue locally.