Rewrite AOF: Invalid cross-device link #305

Closed
janoberst opened this Issue Jan 19, 2012 · 8 comments

Comments

Projects
None yet
7 participants
Contributor

janoberst commented Jan 19, 2012

I get the following error when attempting an AOF rewrite:

Error trying to rename the temporary AOF: Invalid cross-device link

The reason: I have the base directory and AOF file on different disks. Since redis just tries to rename() the file atomically, the process fails if the two paths are on different disks.

I have done this to avoid I/O contention: I have all my redis instances write to the same AOF partition. In case I add a new slave, a BGSAVE is started, which uses a lot of I/O. Therefore I thought I could just put the base dir on a different disk, so that AOF fsyncs don't get horrible slow when a dump is happening.

I'm not sure how easy it would be to support AOF and base dir on different disks. If this is unwanted or too hard to implement, Redis should at least check on startup if base dir and AOF log are on the same disk, maybe by trying a rename() and stopping if it fails.

Contributor

pietern commented Jan 25, 2012

We absolutely need to address this.

What I propose is the following solution. Instead of using only the base directory to determine a temporary file name for the AOF, or RDB snapshot, while it is being written, we can use the full name of the final filename and pad it with a string to make it unique. This would enable using different paths (and thereby disks) for AOF and RDB.

@antirez: What do you think?

Same issue here. Bump!

sreeix commented Nov 14, 2012

+1 We have the same issue, and are now using the workaround, but nice to have a fix.

Just realized this one's affecting me too. Looked into the logs and it's just dumping this (small excerpt) all day long: http://d.pr/n/v0m7

Edit: I seem to have "fixed" it by running CONFIG SET dir <aof_dir> where aof_dir is the path to the (other) disk where I've configured redis to write the AOF file.

keimoon commented Dec 6, 2012

@brunodecarvalho You are my savior. Having unrewritable 14GB aof file and 100MB/s IO rate.

Owner

antirez commented Dec 6, 2012

The file rewritten by the child process must be swapped atomically with the old one, I don't think there is an option to support cross-device AOF file rewrite for us. However it is bad that people don't notice what is happening, we should probably document this better or completely avoid absolute paths as the AOF file name (it just works because of lack of checks, not because it should).

Basically all Redis persistency should happen in the directory configured via "dir".

Basically all Redis persistency should happen in the directory configured via "dir".

Sounds perfectly reasonable. Hard part will be deciding:

  1. Whether Redis should fail on boot if dir is not the target path for the aof file (I'd probably go with this one);
  2. Whether Redis should change dir to the target path for the aof file, silently "fixing" this issue (doesn't follow the PLA)
  3. Assuming .2, what happens when both aof and rdb are enabled and their target paths are different (another reason to go with .1).

Whatever you do, just don't opt with forcing the aof to go on dir. Folks running Redis on EC2 have (or should!) their aof on a different, larger EBS, whereas redis is typically installed on the root one, which is only 8GB.

Keep up the awesome work!

@vperron vperron referenced this issue in fabtools/fabtools May 6, 2013

Merged

Fix redis module DB persistency issue #113

Owner

antirez commented May 8, 2013

Nowadays Redis never tries to persist an RDB and an AOF at the same time, so basically this should no longer be an issue. However the remaining issue is that Redis should force RDB / AOF file name to be just file names and not paths, so I'm opening a new issue about this. Closing this one.

@antirez antirez closed this May 8, 2013

mjbommar pushed a commit to mjbommar/fabtools that referenced this issue Nov 3, 2013

Fix dir bug that caused redis to fail every write
`dir` config var should be set, and all data is relative to it.

* CF http://redis.io/topics/quickstart
* CF antirez/redis#305
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment