Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Repair Search Index eat all memory and crash riak #465

Open
tomascharvat opened this Issue Jan 7, 2014 · 1 comment

Comments

Projects
None yet
2 participants

Riak v1.4.6
Search Index (not 2i) got corrupted for unknown reason. It may be related to presence of siblings of referral keys.
When corruption occur, noise traffic happen between nodes. In time noise seems to raise. Then I followed guide to fix Search Index.
I started repair on 1 server. In few mins, size of riak search index's rose by 50% or so, riak eat all RAM, all swap and then crashed. Coredump included in link.
Node did startup again however consumed diskspace were never freed.
I tried to repair another server. Server repaired without problem, however another server run out of RAM and swap. But this time riak didnt crash.
Coredump
https://www.virusfree.cz/report.tar.bz2

Contributor

jaredmorrow commented Mar 24, 2014

/cc @rzezeski since he is the expert on search index corruption. I was going to link to his guide on repairing, but it looks like you already tried that.

@jaredmorrow jaredmorrow added this to the 1.4.9 milestone Mar 24, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment