Make switching from RDB to AOF (or the contrary) safer in case of OP errors. #837

Open
antirez opened this Issue Dec 19, 2012 · 7 comments

Projects

None yet

7 participants

@antirez
Owner

Here is an OP error pattern that can result in data loss:

  • Edit redis.conf to switch from RDB to AOF (enabling AOF, still taking RDB on).
  • Forget to actually turn AOF on in the currently running Redis instance using CONFIG SET (error!)
  • Restart the server because of a server upgrade or everything like that.

After the restart the following happens:

  • There is an RDB file that contains the most recent version of data. However it is not loaded as AOF has precedence in that case (and AOF is enabled because it was turned on in redis.conf).
  • A new AOF file is created. The old one is missing because AOF was not activated in the running instance.
  • The DB inside Redis memory is empty because AOF file was non existing.
  • After some time Redis may persist, generating an RDB file overwriting the existing RDB file, that is the only file where a copy of data exists.
  • Data loss happened!

After discussing this problem in the Google Group the idea is to do the following:

  • If AOF is on, RDB is on, but RDB is more recent than AOF (with non existing AOF considered infinitely old), the current RDB file is renamed to a different location, and a warning is logged about the event.
  • OR alternatively, the server refuses to start.

However the consequences from the point of view of service availability should be considered before implementing this feature in Redis 2.8. Especially:

  • Is it possible that during normal operations RDB is more recent than AOF file (from the point of view of modification time)?
  • Should the time difference between the two (RDB and AOF) be greater than a given value?
@charsyam

I think if RDB is newer than AOF, it means AOF is wrong.
at that time, I think just delete AOF is better.

@antirez
Owner

Unfortunately it is not always so simple.

Example: AOF enabled, everything on, but no write commands for 5 minutes.
However in the middle of this 5 minutes the user issues the BGSAVE command.

Now RDB is newer than AOF, but the AOF is perfectly valid as well.

Probably this feature is simply much more dangerous than the current setup...

@damnabit

I recently experienced a related issue. I created a dataset before switching both RDB and AOF on. I had a valid dump.rdb but no AOF file. Thus Redis refused to load the data.

I think the appropriate behavior is to load the RDB first, regardless. Then apply AOF, if AOF is empty nothing more happens.

I'm not entirely sure why AOF should take precedence. The entire purpose of AOF (in this case) should be to have the most current data, which by default should contain the delta of the RDB.

If AOF is empty or non-existent the logic shouldn't be to a) refuse to load the RDB if it exists, or b) delete anything.

Lastly, if there is RDB and an empty AOF backup both files and use RDB data to generate a new AOF. Thus saving you from any data loss.

This would be the same as loading the RDB with AOF off, turning AOF on and saving.

@lisa188

I am new to the development process (sorry if I ask a dumb question) and was reading up on Redis persistence when I got stuck at a concept somewhat related to what damnabit said.

I am trying to make sure that when I recover data (i.e. upon Redis restart), Redis will load RDB first and then apply AOF where AOF contains the key changes since the last time RDB has been saved.

My approach was to basically make a cron job that makes backup copies of RDB every X minutes and AOF every Y seconds. So if something were to happen, Redis could recover from RDB first and then apply any AOF changes after that save (which after reading this... I'm guessing is not what Redis does?)

However something was brought to my attention:
1) I need to make sure that during the time RDB is being rewritten and the immediate next key change that is stored in AOF will follow right after the RDB

I understand (I think) that AOF will rewrite from memory in a temp file first meanwhile new key changes are still being stored in the old AOF and in mem buffer. After which the temporary file will replace the old AOF but keep the new changes.

Which brings me to my question:
Does the AOF file ever start from scratch? Because otherwise it could get extremely large over time...

(And another question related to that, I think I am misunderstanding what 'rewriting of the shortest AOF' means - is it just the difference between the disk and the current writes?)

Thanks! And I apologize for the painfully long ... post/question. Hopefully I'm not posting in the wrong spot.

@abcarroll

I just want to add, please someone fix this, or at least add a big, strongly worded warning on the Redis Persistence documentation page. I did this and wiped 2.1GB of very important data.

Sure, it's not really anyone's fault but my own, but I just assumed it was a safe operation, and that AOF was more or less a binary log file that layered over the rdb dump (similar to InnoDB's binary log file) which it clearly has substantial differences.

You'll note that I assumed a lot for this occur, and broke tons of golden rules along the way, but I'm just noting for the record, that at least one person definitely killed a very important data set because of this "issue".

P. S. Using the redis-rdb-tools, looks like I am able to recover at least 5,000 hash buckets that were absolutely vital, the rest can be forgotten or recovered later, but I ended up immediately shutting the box down and manually reading the entire disk for disk offsets with the specific keys I need, reading them with a hacked up version of redis-rdb-tool's parse.py. (Edit: grammar, typos)

@yihuang

I experienced a related issue, i configed an online redis server to aof mode using config set, and config rewrite to persistent it, then i restart redis server, i lost data, because log rewrite haven't happended. so the aof log is not complete data.

@tmm1

Was this improved in the 2.8 release?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment