[BUG] Complete data loss when enabling AOF through config

[threema-danilo]

Describe the bug

Given a Redis 7 server that was started with defualt parameters, when stopping it and starting it with --appendonly yes, all data is gone.

To reproduce

Start Server:

docker run --rm -ti -v /tmp/redis:/data -p 6379:6379 docker.io/redis:7.0.12-alpine

Connect to server and write data:

$ redis-cli
127.0.0.1:6379> keys *
(empty array)
127.0.0.1:6379> set hello "world"
OK

Stop server, and restart it with --appendonly yes:

docker run --rm -ti -v /tmp/redis:/data -p 6379:6379 docker.io/redis:7.0.12-alpine --appendonly yes --appendfsync everysec

Reconnect to server, look at existing keys:

$ redis-cli
127.0.0.1:6379> keys *
(empty array)

The data is now gone. It remains gone when restarting the server again without --appendonly yes.

Expected behavior

From reading through https://redis.io/docs/management/persistence/, my understanding of AOF+RDB is that the AOF is like a "journal" that contains writes since the last RDB snapshot. So when enabling AOF, the old RDB file should still be read, and no data should be lost (or that the data is being automatically converted). That understanding is probably wrong. However, the documentation does not seem clear enough about this.

In the documentation section "How I can switch to AOF, if I'm currently using dump.rdb snapshots?", there are instructions on how to switch a live instance to AOF. However, I understood this as a way to upgrade the database without any downtime. I did not see an indication that stopping the server and changing the config would lose the data.

In the same section, there is this phrase:

IMPORTANT: remember to edit your redis.conf to turn on the AOF, otherwise when you restart the server the configuration changes will be lost and the server will start again with the old configuration.

The warning is that the configuration changes will be lost. There is no word about data loss.

I would expect that either Redis can handle configuration changes from RDB to RDB+AOF without data loss, or that there is a clear warning in the documentation that hints at this behavior.

(Potentially related: #972)

[AmosTsay]

Wish you can get a response from the official

[sundb]

@threema-danilo Please see the comments in the redis.conf

# If the AOF is enabled on startup Redis will load the AOF, that is the file
# with the better durability guarantees.

If you want to change from RDB to RDB+AOF, you should first use config set appendaof true when Redis is running, not restarting.

[threema-danilo]

That comment tells me that the AOF file will be loaded. Since the name of the mode to be used is "RDB+AOF", my expectation was that the AOF file will be loaded in addition to the RDB file.

Furthermore, the comment says that the AOF file will be loaded, not that the RDB file will be truncated (leading in data loss even when switching back to AOF=off).

In case this is the expected and desired behavior: Would a documentation PR be welcome, which adds some warning notes about it?

[sundb]

@threema-danilo Converting from rdb to rdb+aof already exists in the link How I can switch to AOF, if I'm currently using dump.rdb snapshots?, not sure if we need to repeat it in the conf.

[oranagra]

@threema-danilo any documentation PR (considering it's short, clear and improves the current state) is welcome.

[threema-danilo]

The two documentation PRs are here:

• Docs: Improve RDB to AOF conversion docs redis-doc#2521
• Config: redis.conf: Add data loss warning to "appendonly" #12506

I think these notes would already help a lot. Having some kind of support for detecting the previous persistence type after a restart and automatically converting the data would be better of course (from a user point of view), but now at least it's clearly documented.