Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Guard against reading corrupt/empty ring files #190

Closed
ghost opened this Issue Jun 13, 2012 · 4 comments

Comments

Projects
None yet
4 participants
@ghost

ghost commented Jun 13, 2012

Right now, we currently attempt to read the latest ring file, regardless if it is corrupt, empty, et cetera. This has been seen several times, and typically we delete the most recent ring file, and use the previous one.

Riak itself could attempt to implement this functionality.

See also: https://issues.basho.com/show_bug.cgi?id=1078

Contributor

dizzyd commented Jun 13, 2012

There are a few questions that arise here:

  1. Do we use the last-known-good ring file if the latest is corrupt? This might lead to strange issues if the LKG was very old or in a previous configuration.
  2. Do we delete corrupt ring files? We can't parse them, so they're basically useless.

I do agree we can do better. As a interim fix, we could update riaknostic to check for this problem so that it's at least easier to diagnose.

I think 1).

Actually this can be repeatable problem if disk became full unexpectedly:

1)you can`t write data, but fs seems to have allocated some free space in advance so it can create inode and edit directory to have new file.
2) you move/delete some files
3) restart riak, riak crashes because of corrupted ring file
4) you delete ring files
5) compaction or something else occurs - disk is full again =)
6) empty ring files are there again, riak crashes, you delete ring files, etc.
7) repeat X times =)

very entertaining)

Contributor

jonmeredith commented Jun 22, 2012

The pull request above should help the out of disk space or crash during writing case. It won't fix filesystem corruption, but then you're probably dealing with a whole world of hurt.

@ghost ghost assigned slfritchie Nov 16, 2012

Contributor

slfritchie commented Nov 17, 2012

Resolved via #195

@slfritchie slfritchie closed this Nov 17, 2012

chardan pushed a commit that referenced this issue Apr 21, 2013

Merge pull request #190 from basho/az666-2i-unencoded-values-master
Multipart responses now encode 2i values.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment